You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@drill.apache.org by ts...@apache.org on 2015/01/15 06:05:25 UTC
[1/6] drill git commit: Added Drill docs
Repository: drill
Updated Branches:
refs/heads/gh-pages c37bc59fe -> 84b7b36d9
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/tutorial/001-install-sandbox.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/tutorial/001-install-sandbox.md b/_docs/drill-docs/tutorial/001-install-sandbox.md
new file mode 100644
index 0000000..e63ddd4
--- /dev/null
+++ b/_docs/drill-docs/tutorial/001-install-sandbox.md
@@ -0,0 +1,56 @@
+---
+title: "Installing the Apache Drill Sandbox"
+parent: "Apache Drill Tutorial"
+---
+This tutorial uses the MapR Sandbox, which is a Hadoop environment pre-configured with Apache Drill.
+
+To complete the tutorial on the MapR Sandbox with Apache Drill, work through
+the following pages in order:
+
+ * [Installing the Apache Drill Sandbox](/confluence/display/DRILL/Installing+the+Apache+Drill+Sandbox)
+ * [Getting to Know the Drill Setup](/confluence/display/DRILL/Getting+to+Know+the+Drill+Setup)
+ * [Lesson 1: Learn About the Data Set](/confluence/display/DRILL/Lesson+1%3A+Learn+About+the+Data+Set)
+ * [Lesson 2: Run Queries with ANSI SQL](/confluence/display/DRILL/Lesson+2%3A+Run+Queries+with+ANSI+SQL)
+ * [Lesson 3: Run Queries on Complex Data Types](/confluence/display/DRILL/Lesson+3%3A+Run+Queries+on+Complex+Data+Types)
+ * [Summary](/confluence/display/DRILL/Summary)
+
+# About Apache Drill
+
+Drill is an Apache open-source SQL query engine for Big Data exploration.
+Drill is designed from the ground up to support high-performance analysis on
+the semi-structured and rapidly evolving data coming from modern Big Data
+applications, while still providing the familiarity and ecosystem of ANSI SQL,
+the industry-standard query language. Drill provides plug-and-play integration
+with existing Apache Hive and Apache HBase deployments.Apache Drill 0.5 offers
+the following key features:
+
+ * Low-latency SQL queries
+
+ * Dynamic queries on self-describing data in files (such as JSON, Parquet, text) and MapR-DB/HBase tables, without requiring metadata definitions in the Hive metastore.
+
+ * ANSI SQL
+
+ * Nested data support
+
+ * Integration with Apache Hive (queries on Hive tables and views, support for all Hive file formats and Hive UDFs)
+
+ * BI/SQL tool integration using standard JDBC/ODBC drivers
+
+# MapR Sandbox with Apache Drill
+
+MapR includes Apache Drill as part of the Hadoop distribution. The MapR
+Sandbox with Apache Drill is a fully functional single-node cluster that can
+be used to get an overview on Apache Drill in a Hadoop environment. Business
+and technical analysts, product managers, and developers can use the sandbox
+environment to get a feel for the power and capabilities of Apache Drill by
+performing various types of queries. Once you get a flavor for the technology,
+refer to the [Apache Drill web site](http://incubator.apache.org/drill/) and
+[Apache Drill documentation
+](https://cwiki.apache.org/confluence/display/DRILL/Apache+Drill+Wiki)for more
+details.
+
+Note that Hadoop is not a prerequisite for Drill and users can start ramping
+up with Drill by running SQL queries directly on the local file system. Refer
+to [Apache Drill in 10 minutes](https://cwiki.apache.org/confluence/display/DR
+ILL/Apache+Drill+in+10+Minutes) for an introduction to using Drill in local
+(embedded) mode.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/tutorial/002-get2kno-sb.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/tutorial/002-get2kno-sb.md b/_docs/drill-docs/tutorial/002-get2kno-sb.md
new file mode 100644
index 0000000..e7b24a8
--- /dev/null
+++ b/_docs/drill-docs/tutorial/002-get2kno-sb.md
@@ -0,0 +1,235 @@
+---
+title: "Getting to Know the Drill Sandbox"
+parent: "Apache Drill Tutorial"
+---
+This section describes the configuration of the Apache Drill system that you
+have installed and introduces the overall use case for the tutorial.
+
+# Storage Plugins Overview
+
+The Hadoop cluster within the sandbox is set up with MapR-FS, MapR-DB, and
+Hive, which all serve as data sources for Drill in this tutorial. Before you
+can run queries against these data sources, Drill requires each one to be
+configured as a storage plugin. A storage plugin defines the abstraction on
+the data sources for Drill to talk to and provides interfaces to read/write
+and get metadata from the data source. Each storage plugin also exposes
+optimization rules for Drill to leverage for efficient query execution.
+
+Take a look at the pre-configured storage plugins by opening the Drill Web UI.
+
+Feel free to skip this section and jump directly to the queries: [Lesson 1:
+Learn About the Data
+Set](/confluence/display/DRILL/Lesson+1%3A+Learn+About+the+Data+Set)
+
+ * Launch a web browser and go to: `http://<IP address of the sandbox>:8047`
+ * Go to the Storage tab
+ * Open the configured storage plugins one at a time by clicking Update
+ * You will see the following plugins configured.
+
+## maprdb
+
+A storage plugin configuration for MapR-DB in the sandbox. Drill uses a single
+storage plugin for connecting to HBase as well as MapR-DB, which is an
+enterprise grade in-Hadoop NoSQL database. See the [Apache Drill
+Wiki](https://cwiki.apache.org/confluence/display/DRILL/Registering+HBase) for
+information on how to configure Drill to query HBase.
+
+ {
+ "type" : "hbase",
+ "enabled" : true,
+ "config" : {
+ "hbase.table.namespace.mappings" : "*:/tables"
+ }
+ }
+
+## dfs
+
+This is a storage plugin configuration for the MapR file system (MapR-FS) in
+the sandbox. The connection attribute indicates the type of distributed file
+system: in this case, MapR-FS. Drill can work with any distributed system,
+including HDFS, S3, and so on.
+
+The configuration also includes a set of workspaces; each one represents a
+location in MapR-FS:
+
+ * root: access to the root file system location
+ * clicks: access to nested JSON log data
+ * logs: access to flat (non-nested) JSON log data in the logs directory and its subdirectories
+ * views: a workspace for creating views
+
+A workspace in Drill is a location where users can easily access a specific
+set of data and collaborate with each other by sharing artifacts. Users can
+create as many workspaces as they need within Drill.
+
+Each workspace can also be configured as “writable” or not, which indicates
+whether users can write data to this location and defines the storage format
+in which the data will be written (parquet, csv, json). These attributes
+become relevant when you explore Drill SQL commands, especially CREATE TABLE
+AS (CTAS) and CREATE VIEW.
+
+Drill can query files and directories directly and can detect the file formats
+based on the file extension or the first few bits of data within the file.
+However, additional information around formats is required for Drill, such as
+delimiters for text files, which are specified in the “formats” section below.
+
+ {
+ "type": "file",
+ "enabled": true,
+ "connection": "maprfs:///",
+ "workspaces": {
+ "root": {
+ "location": "/mapr/demo.mapr.com/data",
+ "writable": false,
+ "storageformat": null
+ },
+ "clicks": {
+ "location": "/mapr/demo.mapr.com/data/nested",
+ "writable": true,
+ "storageformat": "parquet"
+ },
+ "logs": {
+ "location": "/mapr/demo.mapr.com/data/flat",
+ "writable": true,
+ "storageformat": "parquet"
+ },
+ "views": {
+ "location": "/mapr/demo.mapr.com/data/views",
+ "writable": true,
+ "storageformat": "parquet"
+ },
+ "formats": {
+ "psv": {
+ "type": "text",
+ "extensions": [
+ "tbl"
+ ],
+ "delimiter": "|"
+ },
+ "csv": {
+ "type": "text",
+ "extensions": [
+ "csv"
+ ],
+ "delimiter": ","
+ },
+ "tsv": {
+ "type": "text",
+ "extensions": [
+ "tsv"
+ ],
+ "delimiter": "\t"
+ },
+ "parquet": {
+ "type": "parquet"
+ },
+ "json": {
+ "type": "json"
+ }
+ }}
+
+## hive
+
+A storage plugin configuration for a Hive data warehouse within the sandbox.
+Drill connects to the Hive metastore by using the configured metastore thrift
+URI. Metadata for Hive tables is automatically available for users to query.
+
+ {
+ "type": "hive",
+ "enabled": true,
+ "configProps": {
+ "hive.metastore.uris": "thrift://localhost:9083",
+ "hive.metastore.sasl.enabled": "false"
+ }
+ }
+
+# Client Application Interfaces
+
+Drill also provides additional application interfaces for the client tools to
+connect and access from Drill. The interfaces include the following.
+
+### ODBC/JDBC drivers
+
+Drill provides ODBC/JDBC drivers to connect from BI tools such as Tableau,
+MicroStrategy, SQUirrel, and Jaspersoft; refer to [Using ODBC to Access Apache
+Drill from BI Tools](http://doc.mapr.com/display/MapR/Using+ODBC+to+Access+Apa
+che+Drill+from+BI+Tools) and [Using JDBC to Access Apache Drill](http://doc.ma
+pr.com/display/MapR/Using+JDBC+to+Access+Apache+Drill+from+SQuirreL) to learn
+more.
+
+### SQLLine
+
+SQLLine is a JDBC application that comes packaged with Drill. In order to
+start working with it, you can use the command line on the demo cluster to log
+in as root, then enter `sqlline`. Use `mapr` as the login password. For
+example:
+
+ $ ssh root@localhost -p 2222
+ Password:
+ Last login: Mon Sep 15 13:46:08 2014 from 10.250.0.28
+ Welcome to your Mapr Demo virtual machine.
+ [root@maprdemo ~]# sqlline
+ sqlline version 1.1.6
+ 0: jdbc:drill:>
+
+### Drill Web UI
+
+The Drill Web UI is a simple user interface for configuring and manage Apache
+Drill. This UI can be launched from any of the nodes in the Drill cluster. The
+configuration for Drill includes setting up storage plugins that represent the
+data sources on which Drill performs queries. The sandbox comes with storage
+plugins configured for the Hive, HBase, MapR file system, and local file
+system.
+
+Users and developers can get the necessary information for tuning and
+performing diagnostics on queries, such as the list of queries executed in a
+session and detailed query plan profiles for each.
+
+Detailed configuration and management of Drill is out of scope for this
+tutorial.
+
+The Web interface for Apache Drill also provides a query UI where users can
+submit queries to Drill and observe results. Here is a screen shot of the Web
+UI for Apache Drill:
+
+
+
+### REST API
+
+Drill provides a simple REST API for the users to query data as well as manage
+the system. The Web UI leverages the REST API to talk to Drill.
+
+This tutorial introduces sample queries that you can run by using SQLLine.
+Note that you can run the queries just as easily by launching the Drill Web
+UI. No additional installation or configuration is required.
+
+# Use Case Overview
+
+As you run through the queries in this tutorial, put yourself in the shoes of
+an analyst with basic SQL skills. Let us imagine that the analyst works for an
+emerging online retail business that accepts purchases from its customers
+through both an established web-based interface and a new mobile application.
+
+The analyst is data-driven and operates mostly on the business side with
+little or no interaction with the IT department. Recently the central IT team
+has implemented a Hadoop-based infrastructure to reduce the cost of the legacy
+database system, and most of the DWH/ETL workload is now handled by
+Hadoop/Hive. The master customer profile information and product catalog are
+managed in MapR-DB, which is a NoSQL database. The IT team has also started
+acquiring clickstream data that comes from web and mobile applications. This
+data is stored in Hadoop as JSON files.
+
+The analyst has a number of data sources that he could explore, but exploring
+them in isolation is not the way to go. There are some potentially very
+interesting analytical connections between these data sources. For example, it
+would be good to be able to analyze customer records in the clickstream data
+and tie them to the master customer data in MapR DB.
+
+The analyst decides to explore various data sources and he chooses to do that
+by using Apache Drill. Think about the flexibility and analytic capability of
+Apache Drill as you work through the tutorial.
+
+# What's Next
+
+Start running queries by going to [Lesson 1: Learn About the Data
+Set](/confluence/display/DRILL/Lesson+1%3A+Learn+About+the+Data+Set).
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/tutorial/003-lesson1.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/tutorial/003-lesson1.md b/_docs/drill-docs/tutorial/003-lesson1.md
new file mode 100644
index 0000000..8f3465f
--- /dev/null
+++ b/_docs/drill-docs/tutorial/003-lesson1.md
@@ -0,0 +1,423 @@
+---
+title: "Lession 1: Learn about the Data Set"
+parent: "Apache Drill Tutorial"
+---
+## Goal
+
+This lesson is simply about discovering what data is available, in what
+format, using simple SQL SELECT statements. Drill is capable of analyzing data
+without prior knowledge or definition of its schema. This means that you can
+start querying data immediately (and even as it changes), regardless of its
+format.
+
+The data set for the tutorial consists of:
+
+ * Transactional data: stored as a Hive table
+
+ * Product catalog and master customer data: stored as MapR-DB tables
+
+ * Clickstream and logs data: stored in the MapR file system as JSON files
+
+## Queries in This Lesson
+
+This lesson consists of select * queries on each data source.
+
+## Before You Begin
+
+### Start sqlline
+
+If sqlline is not already started, use a Terminal or Command window to log
+into the demo VM as root, then enter `sqlline`:
+
+ $ ssh root@10.250.0.6
+ Password:
+ Last login: Mon Sep 15 13:46:08 2014 from 10.250.0.28
+ Welcome to your Mapr Demo virtual machine.
+ [root@maprdemo ~]# sqlline
+ sqlline version 1.1.6
+ 0: jdbc:drill:>
+
+You can run queries from this prompt to complete the tutorial. To exit from
+`sqlline`, type:
+
+ 0: jdbc:drill:> !quit
+
+Note that though this tutorial demonstrates the queries using SQLLine, you can
+also execute queries using the Drill Web UI.
+
+### List the available workspaces and databases:
+
+ 0: jdbc:drill:> show databases;
+ +-------------+
+ | SCHEMA_NAME |
+ +-------------+
+ | hive.default |
+ | dfs.default |
+ | dfs.logs |
+ | dfs.root |
+ | dfs.views |
+ | dfs.clicks |
+ | dfs.data |
+ | dfs.tmp |
+ | sys |
+ | maprdb |
+ | cp.default |
+ | INFORMATION_SCHEMA |
+ +-------------+
+ 12 rows selected
+
+Note that this command exposes all the metadata available from the storage
+plugins configured with Drill as a set of schemas. This includes the Hive and
+MapR-DB databases as well as the workspaces configured in the file system. As
+you run queries in the tutorial, you will switch among these schemas by
+submitting the USE command. This behavior resembles the ability to use
+different database schemas (namespaces) in a relational database system.
+
+## Query Hive Tables
+
+The orders table is a six-column Hive table defined in the Hive metastore.
+This is a Hive external table pointing to the data stored in flat files on the
+MapR file system. The orders table contains 122,000 rows.
+
+### Set the schema to hive:
+
+ 0: jdbc:drill:> use hive;
+ +------------+------------+
+ | ok | summary |
+ +------------+------------+
+ | true | Default schema changed to 'hive' |
+ +------------+------------+
+
+You will run the USE command throughout this tutorial. The USE command sets
+the schema for the current session.
+
+### Describe the table:
+
+You can use the DESCRIBE command to show the columns and data types for a Hive
+table:
+
+ 0: jdbc:drill:> describe orders;
+ +-------------+------------+-------------+
+ | COLUMN_NAME | DATA_TYPE | IS_NULLABLE |
+ +-------------+------------+-------------+
+ | order_id | BIGINT | YES |
+ | month | VARCHAR | YES |
+ | cust_id | BIGINT | YES |
+ | state | VARCHAR | YES |
+ | prod_id | BIGINT | YES |
+ | order_total | INTEGER | YES |
+ +-------------+------------+-------------+
+
+The DESCRIBE command returns complete schema information for Hive tables based
+on the metadata available in the Hive metastore.
+
+### Select 5 rows from the orders table:
+
+ 0: jdbc:drill:> select * from orders limit 5;
+ +------------+------------+------------+------------+------------+-------------+
+ | order_id | month | cust_id | state | prod_id | order_total |
+ +------------+------------+------------+------------+------------+-------------+
+ | 67212 | June | 10001 | ca | 909 | 13 |
+ | 70302 | June | 10004 | ga | 420 | 11 |
+ | 69090 | June | 10011 | fl | 44 | 76 |
+ | 68834 | June | 10012 | ar | 0 | 81 |
+ | 71220 | June | 10018 | az | 411 | 24 |
+ +------------+------------+------------+------------+------------+-------------+
+
+Because orders is a Hive table, you can query the data in the same way that
+you would query the columns in a relational database table. Note the use of
+the standard LIMIT clause, which limits the result set to the specified number
+of rows. You can use LIMIT with or without an ORDER BY clause.
+
+Drill provides seamless integration with Hive by allowing queries on Hive
+tables defined in the metastore with no extra configuration. Note that Hive is
+not a prerequisite for Drill, but simply serves as a storage plugin or data
+source for Drill. Drill also lets users query all Hive file formats (including
+custom serdes). Additionally, any UDFs defined in Hive can be leveraged as
+part of Drill queries.
+
+Because Drill has its own low-latency SQL query execution engine, you can
+query Hive tables with high performance and support for interactive and ad-hoc
+data exploration.
+
+## Query MapR-DB and HBase Tables
+
+The customers and products tables are MapR-DB tables. MapR-DB is an enterprise
+in-Hadoop NoSQL database. It exposes the HBase API to support application
+development. Every MapR-DB table has a row_key, in addition to one or more
+column families. Each column family contains one or more specific columns. The
+row_key value is a primary key that uniquely identifies each row.
+
+Drill allows direct queries on MapR-DB and HBase tables. Unlike other SQL on
+Hadoop options, Drill requires no overlay schema definitions in Hive to work
+with this data. Think about a MapR-DB or HBase table with thousands of
+columns, such as a time-series database, and the pain of having to manage
+duplicate schemas for it in Hive!
+
+### Products Table
+
+The products table has two column families.
+
+Column Family|Columns
+
+---|---
+
+details
+
+|
+
+name
+
+category
+
+pricing
+
+|
+
+price
+
+The products table contains 965 rows.
+
+### Customers Table
+
+The Customers table has three column families.
+
+Column Family|Columns
+-------------|-------
+ address | state
+ loyalty | agg_rev
+ | membership
+ personal | age
+ | gender
+
+The customers table contains 993 rows.
+
+### Set the workspace to maprdb:
+
+ 0: jdbc:drill:> use maprdb;
+ +------------+------------+
+ | ok | summary |
+ +------------+------------+
+ | true | Default schema changed to 'maprdb' |
+ +------------+------------+
+
+### Describe the tables:
+
+ 0: jdbc:drill:> describe customers;
+ +-------------+------------+-------------+
+ | COLUMN_NAME | DATA_TYPE | IS_NULLABLE |
+ +-------------+------------+-------------+
+ | row_key | ANY | NO |
+ | address | (VARCHAR(1), ANY) MAP | NO |
+ | loyalty | (VARCHAR(1), ANY) MAP | NO |
+ | personal | (VARCHAR(1), ANY) MAP | NO |
+ +-------------+------------+-------------+
+
+ 0: jdbc:drill:> describe products;
+ +-------------+------------+-------------+
+ | COLUMN_NAME | DATA_TYPE | IS_NULLABLE |
+ +-------------+------------+-------------+
+ | row_key | ANY | NO |
+ | details | (VARCHAR(1), ANY) MAP | NO |
+ | pricing | (VARCHAR(1), ANY) MAP | NO |
+ +-------------+------------+-------------+
+
+Unlike the Hive example, the DESCRIBE command does not return the full schema
+up to the column level. Wide-column NoSQL databases such as MapR-DB and HBase
+can be schema-less by design; every row has its own set of column name-value
+pairs in a given column family, and the column value can be of any data type,
+as determined by the application inserting the data.
+
+A “MAP” complex type in Drill represents this variable column name-value
+structure, and “ANY” represents the fact that the column value can be of any
+data type. Observe the row_key, which is also simply bytes and has the type
+ANY.
+
+### Select 5 rows from the products table:
+
+ 0: jdbc:drill:> select * from products limit 5;
+ +------------+------------+------------+
+ | row_key | details | pricing |
+ +------------+------------+------------+
+ | [B@a1a3e25 | {"category":"bGFwdG9w","name":"IlNvbnkgbm90ZWJvb2si"} | {"price":"OTU5"} |
+ | [B@103a43af | {"category":"RW52ZWxvcGVz","name":"IzEwLTQgMS84IHggOSAxLzIgUHJlbWl1bSBEaWFnb25hbCBTZWFtIEVudmVsb3Blcw=="} | {"price":"MT |
+ | [B@61319e7b | {"category":"U3RvcmFnZSAmIE9yZ2FuaXphdGlvbg==","name":"MjQgQ2FwYWNpdHkgTWF4aSBEYXRhIEJpbmRlciBSYWNrc1BlYXJs"} | {"price" |
+ | [B@9bcf17 | {"category":"TGFiZWxz","name":"QXZlcnkgNDk4"} | {"price":"Mw=="} |
+ | [B@7538ef50 | {"category":"TGFiZWxz","name":"QXZlcnkgNDk="} | {"price":"Mw=="} |
+
+Given that Drill requires no up front schema definitions indicating data
+types, the query returns the raw byte arrays for column values, just as they
+are stored in MapR-DB (or HBase). Observe that the column families (details
+and pricing) have the map data type and appear as JSON strings.
+
+In Lesson 2, you will use CAST functions to return typed data for each column.
+
+### Select 5 rows from the customers table:
+
+
+ +0: jdbc:drill:> select * from customers limit 5;
+ +------------+------------+------------+------------+
+ | row_key | address | loyalty | personal |
+ +------------+------------+------------+------------+
+ | [B@284bae62 | {"state":"Imt5Ig=="} | {"agg_rev":"IjEwMDEtMzAwMCI=","membership":"ImJhc2ljIg=="} | {"age":"IjI2LTM1Ig==","gender":"Ik1B |
+ | [B@7ffa4523 | {"state":"ImNhIg=="} | {"agg_rev":"IjAtMTAwIg==","membership":"ImdvbGQi"} | {"age":"IjI2LTM1Ig==","gender":"IkZFTUFMRSI= |
+ | [B@7d13e79 | {"state":"Im9rIg=="} | {"agg_rev":"IjUwMS0xMDAwIg==","membership":"InNpbHZlciI="} | {"age":"IjI2LTM1Ig==","gender":"IkZFT |
+ | [B@3a5c7df1 | {"state":"ImtzIg=="} | {"agg_rev":"IjMwMDEtMTAwMDAwIg==","membership":"ImdvbGQi"} | {"age":"IjUxLTEwMCI=","gender":"IkZF |
+ | [B@e507726 | {"state":"Im5qIg=="} | {"agg_rev":"IjAtMTAwIg==","membership":"ImJhc2ljIg=="} | {"age":"IjIxLTI1Ig==","gender":"Ik1BTEUi" |
+ +------------+------------+------------+------------+
+
+Again the table returns byte data that needs to be cast to readable data
+types.
+
+## Query the File System
+
+Along with querying a data source with full schemas (such as Hive) and partial
+schemas (such as MapR-DB and HBase), Drill offers the unique capability to
+perform SQL queries directly on file system. The file system could be a local
+file system, or a distributed file system such as MapR-FS, HDFS, or S3.
+
+In the context of Drill, a file or a directory is considered as synonymous to
+a relational database “table.” Therefore, you can perform SQL operations
+directly on files and directories without the need for up-front schema
+definitions or schema management for any model changes. The schema is
+discovered on the fly based on the query. Drill supports queries on a variety
+of file formats including text, CSV, Parquet, and JSON in the 0.5 release.
+
+In this example, the clickstream data coming from the mobile/web applications
+is in JSON format. The JSON files have the following structure:
+
+ {"trans_id":31920,"date":"2014-04-26","time":"12:17:12","user_info":{"cust_id":22526,"device":"IOS5","state":"il"},"trans_info":{"prod_id":[174,2],"purch_flag":"false"}}
+ {"trans_id":31026,"date":"2014-04-20","time":"13:50:29","user_info":{"cust_id":16368,"device":"AOS4.2","state":"nc"},"trans_info":{"prod_id":[],"purch_flag":"false"}}
+ {"trans_id":33848,"date":"2014-04-10","time":"04:44:42","user_info":{"cust_id":21449,"device":"IOS6","state":"oh"},"trans_info":{"prod_id":[582],"purch_flag":"false"}}
+
+
+The clicks.json and clicks.campaign.json files contain metadata as part of the
+data itself (referred to as “self-describing” data). Also note that the data
+elements are complex, or nested. The initial queries below do not show how to
+unpack the nested data, but they show that easy access to the data requires no
+setup beyond the definition of a workspace.
+
+### Query nested clickstream data
+
+#### Set the workspace to dfs.clicks:
+
+ 0: jdbc:drill:> use dfs.clicks;
+ +------------+------------+
+ | ok | summary |
+ +------------+------------+
+ | true | Default schema changed to 'dfs.clicks' |
+ +------------+------------+
+
+In this case, setting the workspace is a mechanism for making queries easier
+to write. When you specify a file system workspace, you can shorten references
+to files in the FROM clause of your queries. Instead of having to provide the
+complete path to a file, you can provide the path relative to a directory
+location specified in the workspace. For example:
+
+ "location": "/mapr/demo.mapr.com/data/nested"
+
+Any file or directory that you want to query in this path can be referenced
+relative to this path. The clicks directory referred to in the following query
+is directly below the nested directory.
+
+#### Select 2 rows from the clicks.json file:
+
+ 0: jdbc:drill:> select * from `clicks/clicks.json` limit 2;
+ +------------+------------+------------+------------+------------+
+ | trans_id | date | time | user_info | trans_info |
+ +------------+------------+------------+------------+------------+
+ | 31920 | 2014-04-26 | 12:17:12 | {"cust_id":22526,"device":"IOS5","state":"il"} | {"prod_id":[174,2],"purch_flag":"false"} |
+ | 31026 | 2014-04-20 | 13:50:29 | {"cust_id":16368,"device":"AOS4.2","state":"nc"} | {"prod_id":[],"purch_flag":"false"} |
+ +------------+------------+------------+------------+------------+
+ 2 rows selected
+
+Note that the FROM clause reference points to a specific file. Drill expands
+the traditional concept of a “table reference” in a standard SQL FROM clause
+to refer to a file in a local or distributed file system.
+
+The only special requirement is the use of back ticks to enclose the file
+path. This is necessary whenever the file path contains Drill reserved words
+or characters.
+
+#### Select 2 rows from the campaign.json file:
+
+ 0: jdbc:drill:> select * from `clicks/clicks.campaign.json` limit 2;
+ +------------+------------+------------+------------+------------+------------+
+ | trans_id | date | time | user_info | ad_info | trans_info |
+ +------------+------------+------------+------------+------------+------------+
+ | 35232 | 2014-05-10 | 00:13:03 | {"cust_id":18520,"device":"AOS4.3","state":"tx"} | {"camp_id":"null"} | {"prod_id":[7,7],"purch_flag":"true"} |
+ | 31995 | 2014-05-22 | 16:06:38 | {"cust_id":17182,"device":"IOS6","state":"fl"} | {"camp_id":"null"} | {"prod_id":[],"purch_flag":"false"} |
+ +------------+------------+------------+------------+------------+------------+
+ 2 rows selected
+
+Notice that with a select * query, any complex data types such as maps and
+arrays return as JSON strings. You will see how to unpack this data using
+various SQL functions and operators in the next lesson.
+
+## Query Logs Data
+
+Unlike the previous example where we performed queries against clicks data in
+one file, logs data is stored as partitioned directories on the file system.
+The logs directory has three subdirectories:
+
+ * 2012
+
+ * 2013
+
+ * 2014
+
+Each of these year directories fans out to a set of numbered month
+directories, and each month directory contains a JSON file with log records
+for that month. The total number of records in all log files is 48000.
+
+The files in the logs directory and its subdirectories are JSON files. There
+are many of these files, but you can use Drill to query them all as a single
+data source, or to query a subset of the files.
+
+#### Set the workspace to dfs.logs:
+
+ 0: jdbc:drill:> use dfs.logs;
+ +------------+------------+
+ | ok | summary |
+ +------------+------------+
+ | true | Default schema changed to 'dfs.logs' |
+ +------------+------------+
+
+#### Select 2 rows from the logs directory:
+
+ 0: jdbc:drill:> select * from logs limit 2;
+ +------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+----------+
+ | dir0 | dir1 | trans_id | date | time | cust_id | device | state | camp_id | keywords | prod_id | purch_fl |
+ +------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+----------+
+ | 2014 | 8 | 24181 | 08/02/2014 | 09:23:52 | 0 | IOS5 | il | 2 | wait | 128 | false |
+ | 2014 | 8 | 24195 | 08/02/2014 | 07:58:19 | 243 | IOS5 | mo | 6 | hmm | 107 | false |
+ +------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+----------+
+
+Note that this is flat JSON data. The dfs.clicks workspace location property
+points to a directory that contains the logs directory, making the FROM clause
+reference for this query very simple. You do not have to refer to the complete
+directory path on the file system.
+
+The column names dir0 and dir1 are special Drill variables that identify
+subdirectories below the logs directory. In Lesson 3, you will do more complex
+queries that leverage these dynamic variables.
+
+#### Find the total number of rows in the logs directory (all files):
+
+ 0: jdbc:drill:> select count(*) from logs;
+ +------------+
+ | EXPR$0 |
+ +------------+
+ | 48000 |
+ +------------+
+
+This query traverses all of the files in the logs directory and its
+subdirectories to return the total number of rows in those files.
+
+# What's Next
+
+Go to [Lesson 2: Run Queries with ANSI
+SQL](/confluence/display/DRILL/Lesson+2%3A+Run+Queries+with+ANSI+SQL).
+
+
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/tutorial/004-lesson2.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/tutorial/004-lesson2.md b/_docs/drill-docs/tutorial/004-lesson2.md
new file mode 100644
index 0000000..d9c68d5
--- /dev/null
+++ b/_docs/drill-docs/tutorial/004-lesson2.md
@@ -0,0 +1,392 @@
+---
+title: "Lession 2: Run Queries with ANSI SQL"
+parent: "Apache Drill Tutorial"
+---
+## Goal
+
+This lesson shows how to do some standard SQL analysis in Apache Drill: for
+example, summarizing data by using simple aggregate functions and connecting
+data sources by using joins. Note that Apache Drill provides ANSI SQL support,
+not a “SQL-like” interface.
+
+## Queries in This Lesson
+
+Now that you know what the data sources look like in their raw form, using
+select * queries, try running some simple but more useful queries on each data
+source. These queries demonstrate how Drill supports ANSI SQL constructs and
+also how you can combine data from different data sources in a single SELECT
+statement.
+
+ * Show an aggregate query on a single file or table. Use GROUP BY, WHERE, HAVING, and ORDER BY clauses.
+
+ * Perform joins between Hive, MapR-DB, and file system data sources.
+
+ * Use table and column aliases.
+
+ * Create a Drill view.
+
+## Aggregation
+
+
+### Set the schema to hive:
+
+ 0: jdbc:drill:> use hive;
+ +------------+------------+
+ | ok | summary |
+ +------------+------------+
+ | true | Default schema changed to 'hive' |
+ +------------+------------+
+ 1 row selected
+
+### Return sales totals by month:
+
+ 0: jdbc:drill:> select `month`, sum(order_total)
+ from orders group by `month` order by 2 desc;
+ +------------+------------+
+ | month | EXPR$1 |
+ +------------+------------+
+ | June | 950481 |
+ | May | 947796 |
+ | March | 836809 |
+ | April | 807291 |
+ | July | 757395 |
+ | October | 676236 |
+ | August | 572269 |
+ | February | 532901 |
+ | September | 373100 |
+ | January | 346536 |
+ +------------+------------+
+
+Drill supports SQL aggregate functions such as SUM, MAX, AVG, and MIN.
+Standard SQL clauses work in the same way in Drill queries as in relational
+database queries.
+
+Note that back ticks are required for the “month” column only because “month”
+is a reserved word in SQL.
+
+### Return the top 20 sales totals by month and state:
+
+ 0: jdbc:drill:> select `month`, state, sum(order_total) as sales from orders group by `month`, state
+ order by 3 desc limit 20;
+ +------------+------------+------------+
+ | month | state | sales |
+ +------------+------------+------------+
+ | May | ca | 119586 |
+ | June | ca | 116322 |
+ | April | ca | 101363 |
+ | March | ca | 99540 |
+ | July | ca | 90285 |
+ | October | ca | 80090 |
+ | June | tx | 78363 |
+ | May | tx | 77247 |
+ | March | tx | 73815 |
+ | August | ca | 71255 |
+ | April | tx | 68385 |
+ | July | tx | 63858 |
+ | February | ca | 63527 |
+ | June | fl | 62199 |
+ | June | ny | 62052 |
+ | May | fl | 61651 |
+ | May | ny | 59369 |
+ | October | tx | 55076 |
+ | March | fl | 54867 |
+ | March | ny | 52101 |
+ +------------+------------+------------+
+ 20 rows selected
+
+Note the alias for the result of the SUM function. Drill supports column
+aliases and table aliases.
+
+## HAVING Clause
+
+This query uses the HAVING clause to constrain an aggregate result.
+
+### Set the workspace to dfs.clicks
+
+ 0: jdbc:drill:> use dfs.clicks;
+ +------------+------------+
+ | ok | summary |
+ +------------+------------+
+ | true | Default schema changed to 'dfs.clicks' |
+ +------------+------------+
+ 1 row selected
+
+### Return total number of clicks for devices that indicate high click-throughs:
+
+ 0: jdbc:drill:> select t.user_info.device, count(*) from `clicks/clicks.json` t
+ group by t.user_info.device
+ having count(*) > 1000;
+ +------------+------------+
+ | EXPR$0 | EXPR$1 |
+ +------------+------------+
+ | IOS5 | 11814 |
+ | AOS4.2 | 5986 |
+ | IOS6 | 4464 |
+ | IOS7 | 3135 |
+ | AOS4.4 | 1562 |
+ | AOS4.3 | 3039 |
+ +------------+------------+
+
+The aggregate is a count of the records for each different mobile device in
+the clickstream data. Only the activity for the devices that registered more
+than 1000 transactions qualify for the result set.
+
+## UNION Operator
+
+Use the same workspace as before (dfs.clicks).
+
+### Combine clicks activity from before and after the marketing campaign
+
+ 0: jdbc:drill:> select t.trans_id transaction, t.user_info.cust_id customer from `clicks/clicks.campaign.json` t
+ union all
+ select u.trans_id, u.user_info.cust_id from `clicks/clicks.json` u limit 5;
+ +-------------+------------+
+ | transaction | customer |
+ +-------------+------------+
+ | 35232 | 18520 |
+ | 31995 | 17182 |
+ | 35760 | 18228 |
+ | 37090 | 17015 |
+ | 37838 | 18737 |
+ +-------------+------------+
+
+This UNION ALL query returns rows that exist in two files (and includes any
+duplicate rows from those files): `clicks.campaign.json` and `clicks.json`.
+
+## Subqueries
+
+### Set the workspace to hive:
+
+ 0: jdbc:drill:> use hive;
+ +------------+------------+
+ | ok | summary |
+ +------------+------------+
+ | true | Default schema changed to 'hive' |
+ +------------+------------+
+
+### Compare order totals across states:
+
+ 0: jdbc:drill:> select o1.cust_id, sum(o1.order_total) as ny_sales,
+ (select sum(o2.order_total) from hive.orders o2
+ where o1.cust_id=o2.cust_id and state='ca') as ca_sales
+ from hive.orders o1 where o1.state='ny' group by o1.cust_id
+ order by cust_id limit 20;
+ +------------+------------+------------+
+ | cust_id | ny_sales | ca_sales |
+ +------------+------------+------------+
+ | 1001 | 72 | 47 |
+ | 1002 | 108 | 198 |
+ | 1003 | 83 | null |
+ | 1004 | 86 | 210 |
+ | 1005 | 168 | 153 |
+ | 1006 | 29 | 326 |
+ | 1008 | 105 | 168 |
+ | 1009 | 443 | 127 |
+ | 1010 | 75 | 18 |
+ | 1012 | 110 | null |
+ | 1013 | 19 | null |
+ | 1014 | 106 | 162 |
+ | 1015 | 220 | 153 |
+ | 1016 | 85 | 159 |
+ | 1017 | 82 | 56 |
+ | 1019 | 37 | 196 |
+ | 1020 | 193 | 165 |
+ | 1022 | 124 | null |
+ | 1023 | 166 | 149 |
+ | 1024 | 233 | null |
+ +------------+------------+------------+
+
+This example demonstrates Drill support for correlated subqueries. This query
+uses a subquery in the select list and correlates the result of the subquery
+with the outer query, using the cust_id column reference. The subquery returns
+the sum of order totals for California, and the outer query returns the
+equivalent sum, for the same cust_id, for New York.
+
+The result set is sorted by the cust_id and presents the sales totals side by
+side for easy comparison. Null values indicate customer IDs that did not
+register any sales in that state.
+
+## CAST Function
+
+### Use the maprdb workspace:
+
+ 0: jdbc:drill:> use maprdb;
+ +------------+------------+
+ | ok | summary |
+ +------------+------------+
+ | true | Default schema changed to 'maprdb' |
+ +------------+------------+
+ 1 row selected
+
+### Return customer data with appropriate data types
+
+ 0: jdbc:drill:> select cast(row_key as int) as cust_id, cast(t.personal.name as varchar(20)) as name,
+ cast(t.personal.gender as varchar(10)) as gender, cast(t.personal.age as varchar(10)) as age,
+ cast(t.address.state as varchar(4)) as state, cast(t.loyalty.agg_rev as dec(7,2)) as agg_rev,
+ cast(t.loyalty.membership as varchar(20)) as membership
+ from customers t limit 5;
+ +------------+------------+------------+------------+------------ +------------+------------+
+ | cust_id | name | gender | age | state | agg_rev | membership |
+ +------------+------------+------------+------------+------------+------------+------------+
+ | 10001 | "Corrine Mecham" | "FEMALE" | "15-20" | "va" | 197.00 | "silver" |
+ | 10005 | "Brittany Park" | "MALE" | "26-35" | "in" | 230.00 | "silver" |
+ | 10006 | "Rose Lokey" | "MALE" | "26-35" | "ca" | 250.00 | "silver" |
+ | 10007 | "James Fowler" | "FEMALE" | "51-100" | "me" | 263.00 | "silver" |
+ | 10010 | "Guillermo Koehler" | "OTHER" | "51-100" | "mn" | 202.00 | "silver" |
+ +------------+------------+------------+------------+------------+------------+------------+
+ 5 rows selected
+
+Note the following features of this query:
+
+ * The CAST function is required for every column in the table. This function returns the MapR-DB/HBase binary data as readable integers and strings. Alternatively, you can use CONVERT_TO/CONVERT_FROM functions to decode the columns. CONVERT_TO and CONVERT_FROM are more efficient than CAST in most cases.
+ * The row_key column functions as the primary key of the table (a customer ID in this case).
+ * The table alias t is required; otherwise the column family names would be parsed as table names and the query would return an error.
+
+### Remove the quotes from the strings:
+
+You can use the regexp_replace function to remove the quotes around the
+strings in the query results. For example, to return a state name va instead
+of “va”:
+
+ 0: jdbc:drill:> select cast(row_key as int), regexp_replace(cast(t.address.state as varchar(10)),'"','')
+ from customers t limit 1;
+ +------------+------------+
+ | EXPR$0 | EXPR$1 |
+ +------------+------------+
+ | 10001 | va |
+ +------------+------------+
+ 1 row selected
+
+## CREATE VIEW Command
+
+ 0: jdbc:drill:> use dfs.views;
+ +------------+------------+
+ | ok | summary |
+ +------------+------------+
+ | true | Default schema changed to 'dfs.views' |
+ +------------+------------+
+
+### Use a mutable workspace:
+
+A mutable (or writable) workspace is a workspace that is enabled for “write”
+operations. This attribute is part of the storage plugin configuration. You
+can create Drill views and tables in mutable workspaces.
+
+### Create a view on a MapR-DB table
+
+ 0: jdbc:drill:> create or replace view custview as select cast(row_key as int) as cust_id,
+ cast(t.personal.name as varchar(20)) as name,
+ cast(t.personal.gender as varchar(10)) as gender,
+ cast(t.personal.age as varchar(10)) as age,
+ cast(t.address.state as varchar(4)) as state,
+ cast(t.loyalty.agg_rev as dec(7,2)) as agg_rev,
+ cast(t.loyalty.membership as varchar(20)) as membership
+ from maprdb.customers t;
+ +------------+------------+
+ | ok | summary |
+ +------------+------------+
+ | true | View 'custview' replaced successfully in 'dfs.views' schema |
+ +------------+------------+
+ 1 row selected
+
+Drill provides CREATE OR REPLACE VIEW syntax similar to relational databases
+to create views. Use the OR REPLACE option to make it easier to update the
+view later without having to remove it first. Note that the FROM clause in
+this example must refer to maprdb.customers. The MapR-DB tables are not
+directly visible to the dfs.views workspace.
+
+Unlike a traditional database where views typically are DBA/developer-driven
+operations, file system-based views in Drill are very lightweight. A view is
+simply a special file with a specific extension (.drill). You can store views
+even in your local file system or point to a specific workspace. You can
+specify any query against any Drill data source in the body of the CREATE VIEW
+statement.
+
+Drill provides a decentralized metadata model. Drill is able to query metadata
+defined in data sources such as Hive, HBase, and the file system. Drill also
+supports the creation of metadata in the file system.
+
+### Query data from the view:
+
+ 0: jdbc:drill:> select * from custview limit 1;
+ +------------+------------+------------+------------+------------+------------+------------+
+ | cust_id | name | gender | age | state | agg_rev | membership |
+ +------------+------------+------------+------------+------------+------------+------------+
+ | 10001 | "Corrine Mecham" | "FEMALE" | "15-20" | "va" | 197.00 | "silver" |
+ +------------+------------+------------+------------+------------+------------+------------+
+
+Once the users get an idea on what data is available by exploring it directly
+from file system , views can be used as a way to take the data in downstream
+tools like Tableau, Microstrategy etc for downstream analysis and
+visualization. For these tools, a view appears simply as a “table” with
+selectable “columns” in it.
+
+## Query Across Data Sources
+
+Continue using dfs.views for this query.
+
+### Join the customers view and the orders table:
+
+ 0: jdbc:drill:> select membership, sum(order_total) as sales from hive.orders, custview
+ where orders.cust_id=custview.cust_id
+ group by membership order by 2;
+ +------------+------------+
+ | membership | sales |
+ +------------+------------+
+ | "basic" | 380665 |
+ | "silver" | 708438 |
+ | "gold" | 2787682 |
+ +------------+------------+
+ 3 rows selected
+
+In this query, we are reading data from a MapR-DB table (represented by
+custview) and combining it with the order information in Hive. When doing
+cross data source queries such as this, you need to use fully qualified
+table/view names. For example, the orders table is prefixed by “hive,” which
+is the storage plugin name registered with Drill. We are not using any prefix
+for “custview” because we explicitly switched the dfs.views workspace where
+custview is stored.
+
+Note: If the results of any of your queries appear to be truncated because the
+rows are wide, set the maximum width of the display to 10000:
+
+Do not use a semicolon for this SET command.
+
+### Join the customers, orders, and clickstream data:
+
+ 0: jdbc:drill:> select custview.membership, sum(orders.order_total) as sales from hive.orders, custview,
+ dfs.`/mapr/demo.mapr.com/data/nested/clicks/clicks.json` c
+ where orders.cust_id=custview.cust_id and orders.cust_id=c.user_info.cust_id
+ group by custview.membership order by 2;
+ +------------+------------+
+ | membership | sales |
+ +------------+------------+
+ | "basic" | 372866 |
+ | "silver" | 728424 |
+ | "gold" | 7050198 |
+ +------------+------------+
+ 3 rows selected
+
+This three-way join selects from three different data sources in one query:
+
+ * hive.orders table
+ * custview (a view of the HBase customers table)
+ * clicks.json file
+
+The join column for both sets of join conditions is the cust_id column. The
+views workspace is used for this query so that custview can be accessed. The
+hive.orders table is also visible to the query.
+
+However, note that the JSON file is not directly visible from the views
+workspace, so the query specifies the full path to the file:
+
+ dfs.`/mapr/demo.mapr.com/data/nested/clicks/clicks.json`
+
+
+# What's Next
+
+Go to [Lesson 3: Run Queries on Complex Data Types](/confluence/display/DRILL/
+Lesson+3%3A+Run+Queries+on+Complex+Data+Types).
+
+
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/tutorial/005-lesson3.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/tutorial/005-lesson3.md b/_docs/drill-docs/tutorial/005-lesson3.md
new file mode 100644
index 0000000..d9b362a
--- /dev/null
+++ b/_docs/drill-docs/tutorial/005-lesson3.md
@@ -0,0 +1,379 @@
+---
+title: "Lession 3: Run Queries on Complex Data Types"
+parent: "Apache Drill Tutorial"
+---
+## Goal
+
+This lesson focuses on queries that exercise functions and operators on self-
+describing data and complex data types. Drill offers intuitive SQL extensions
+to work with such data and offers high query performance with an architecture
+built from the ground up for complex data.
+
+## Queries in This Lesson
+
+Now that you have run ANSI SQL queries against different tables and files with
+relational data, you can try some examples including complex types.
+
+ * Access directories and subdirectories of files in a single SELECT statement.
+ * Demonstrate simple ways to access complex data in JSON files.
+ * Demonstrate the repeated_count function to aggregate values in an array.
+
+## Query Partitioned Directories
+
+You can use special variables in Drill to refer to subdirectories in your
+workspace path:
+
+ * dir0
+ * dir1
+ * …
+
+Note that these variables are dynamically determined based on the partitioning
+of the file system. No up-front definitions are required on what partitions
+exist. Here is a visual example of how this works:
+
+
+
+### Set workspace to dfs.logs:
+
+ 0: jdbc:drill:> use dfs.logs;
+ +------------+------------+
+ | ok | summary |
+ +------------+------------+
+ | true | Default schema changed to 'dfs.logs' |
+ +------------+------------+
+
+### Query logs data for a specific year:
+
+ 0: jdbc:drill:> select * from logs where dir0='2013' limit 10;
+ +------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+-----------+------------+
+ | dir0 | dir1 | trans_id | date | time | cust_id | device | state | camp_id | keywords | prod_id | purch_flag |
+ +------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+-----------+------------+
+ | 2013 | 11 | 12119 | 11/09/2013 | 02:24:51 | 262 | IOS5 | ny | 0 | chamber | 198 | false |
+ | 2013 | 11 | 12120 | 11/19/2013 | 09:37:43 | 0 | AOS4.4 | il | 2 | outside | 511 | false |
+ | 2013 | 11 | 12134 | 11/10/2013 | 23:42:47 | 60343 | IOS5 | ma | 4 | and | 421 | false |
+ | 2013 | 11 | 12135 | 11/16/2013 | 01:42:13 | 46762 | AOS4.3 | ca | 4 | here's | 349 | false |
+ | 2013 | 11 | 12165 | 11/26/2013 | 21:58:09 | 41987 | AOS4.2 | mn | 4 | he | 271 | false |
+ | 2013 | 11 | 12168 | 11/09/2013 | 23:41:48 | 8600 | IOS5 | in | 6 | i | 459 | false |
+ | 2013 | 11 | 12196 | 11/20/2013 | 02:23:06 | 15603 | IOS5 | tn | 1 | like | 324 | false |
+ | 2013 | 11 | 12203 | 11/25/2013 | 23:50:29 | 221 | IOS6 | tx | 10 | if | 323 | false |
+ | 2013 | 11 | 12206 | 11/09/2013 | 23:53:01 | 2488 | AOS4.2 | tx | 14 | unlike | 296 | false |
+ | 2013 | 11 | 12217 | 11/06/2013 | 23:51:56 | 0 | AOS4.2 | tx | 9 | can't | 54 | false |
+ +------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+
+
+
+This query constrains files inside the subdirectory named 2013. The variable
+dir0 refers to the first level down from logs, dir1 to the next level, and so
+on. So this query returned 10 of the rows for February 2013.
+
+### Further constrain the results using multiple predicates in the query:
+
+This query returns a list of customer IDs for people who made a purchase via
+an IOS5 device in August 2013.
+
+ 0: jdbc:drill:> select dir0 as yr, dir1 as mth, cust_id from logs
+ where dir0='2013' and dir1='8' and device='IOS5' and purch_flag='true'
+ order by `date`;
+ +------------+------------+------------+
+ | yr | mth | cust_id |
+ +------------+------------+------------+
+ | 2013 | 8 | 4 |
+ | 2013 | 8 | 521 |
+ | 2013 | 8 | 1 |
+ | 2013 | 8 | 2 |
+ | 2013 | 8 | 4 |
+ | 2013 | 8 | 549 |
+ | 2013 | 8 | 72827 |
+ | 2013 | 8 | 38127 |
+ ...
+
+### Return monthly counts per customer for a given year:
+
+ 0: jdbc:drill:> select cust_id, dir1 month_no, count(*) month_count from logs
+ where dir0=2014 group by cust_id, dir1 order by cust_id, month_no limit 10;
+ +------------+------------+-------------+
+ | cust_id | month_no | month_count |
+ +------------+------------+-------------+
+ | 0 | 1 | 143 |
+ | 0 | 2 | 118 |
+ | 0 | 3 | 117 |
+ | 0 | 4 | 115 |
+ | 0 | 5 | 137 |
+ | 0 | 6 | 117 |
+ | 0 | 7 | 142 |
+ | 0 | 8 | 19 |
+ | 1 | 1 | 66 |
+ | 1 | 2 | 59 |
+ +------------+------------+-------------+
+ 10 rows selected
+
+This query groups the aggregate function by customer ID and month for one
+year: 2014.
+
+## Query Complex Data
+
+Drill provides some specialized operators and functions that you can use to
+analyze nested data natively without transformation. If you are familiar with
+JavaScript notation, you will already know how some of these extensions work.
+
+### Set the workspace to dfs.clicks:
+
+ 0: jdbc:drill:> use dfs.clicks;
+ +------------+------------+
+ | ok | summary |
+ +------------+------------+
+ | true | Default schema changed to 'dfs.clicks' |
+ +------------+------------+
+
+### Explore clickstream data:
+
+Note that the user_info and trans_info columns contain nested data: arrays and
+arrays within arrays. The following queries show how to access this complex
+data.
+
+ 0: jdbc:drill:> select * from `clicks/clicks.json` limit 5;
+ +------------+------------+------------+------------+------------+
+ | trans_id | date | time | user_info | trans_info |
+ +------------+------------+------------+------------+------------+
+ | 31920 | 2014-04-26 | 12:17:12 | {"cust_id":22526,"device":"IOS5","state":"il"} | {"prod_id":[174,2],"purch_flag":"false"} |
+ | 31026 | 2014-04-20 | 13:50:29 | {"cust_id":16368,"device":"AOS4.2","state":"nc"} | {"prod_id":[],"purch_flag":"false"} |
+ | 33848 | 2014-04-10 | 04:44:42 | {"cust_id":21449,"device":"IOS6","state":"oh"} | {"prod_id":[582],"purch_flag":"false"} |
+ | 32383 | 2014-04-18 | 06:27:47 | {"cust_id":20323,"device":"IOS5","state":"oh"} | {"prod_id":[710,47],"purch_flag":"false"} |
+ | 32359 | 2014-04-19 | 23:13:25 | {"cust_id":15360,"device":"IOS5","state":"ca"} | {"prod_id": [0,8,170,173,1,124,46,764,30,711,0,3,25],"purch_flag":"true"} |
+ +------------+------------+------------+------------+------------+
+
+
+### Unpack the user_info column:
+
+ 0: jdbc:drill:> select t.user_info.cust_id as custid, t.user_info.device as device,
+ t.user_info.state as state
+ from `clicks/clicks.json` t limit 5;
+ +------------+------------+------------+
+ | custid | device | state |
+ +------------+------------+------------+
+ | 22526 | IOS5 | il |
+ | 16368 | AOS4.2 | nc |
+ | 21449 | IOS6 | oh |
+ | 20323 | IOS5 | oh |
+ | 15360 | IOS5 | ca |
+ +------------+------------+------------+
+
+This query uses a simple table.column.column notation to extract nested column
+data. For example:
+
+ t.user_info.cust_id
+
+where `t` is the table alias provided in the query, `user_info` is a top-level
+column name, and `cust_id` is a nested column name.
+
+The table alias is required; otherwise column names such as `user_info` are
+parsed as table names by the SQL parser.
+
+### Unpack the trans_info column:
+
+ 0: jdbc:drill:> select t.trans_info.prod_id as prodid, t.trans_info.purch_flag as
+ purchased
+ from `clicks/clicks.json` t limit 5;
+ +------------+------------+
+ | prodid | purchased |
+ +------------+------------+
+ | [174,2] | false |
+ | [] | false |
+ | [582] | false |
+ | [710,47] | false |
+ | [0,8,170,173,1,124,46,764,30,711,0,3,25] | true |
+ +------------+------------+
+ 5 rows selected
+
+Note that this result reveals that the prod_id column contains an array of IDs
+(one or more product ID values per row, separated by commas). The next step
+shows how you to access this kind of data.
+
+## Query Arrays
+
+Now use the [ n ] notation, where n is the position of the value in an array,
+starting from position 0 (not 1) for the first value. You can use this
+notation to write interesting queries against nested array data.
+
+For example:
+
+ trans_info.prod_id[0]
+
+refers to the first value in the nested prod_id column and
+
+ trans_info.prod_id[20]
+
+refers to the 21st value, assuming one exists.
+
+### Find the first product that is searched for in each transaction:
+
+ 0: jdbc:drill:> select t.trans_id, t.trans_info.prod_id[0] from `clicks/clicks.json` t limit 5;
+ +------------+------------+
+ | trans_id | EXPR$1 |
+ +------------+------------+
+ | 31920 | 174 |
+ | 31026 | null |
+ | 33848 | 582 |
+ | 32383 | 710 |
+ | 32359 | 0 |
+ +------------+------------+
+ 5 rows selected
+
+### For which transactions did customers search on at least 21 products?
+
+ 0: jdbc:drill:> select t.trans_id, t.trans_info.prod_id[20]
+ from `clicks/clicks.json` t
+ where t.trans_info.prod_id[20] is not null
+ order by trans_id limit 5;
+ +------------+------------+
+ | trans_id | EXPR$1 |
+ +------------+------------+
+ | 10328 | 0 |
+ | 10380 | 23 |
+ | 10701 | 1 |
+ | 11100 | 0 |
+ | 11219 | 46 |
+ +------------+------------+
+ 5 rows selected
+
+This query returns transaction IDs and product IDs for records that contain a
+non-null product ID at the 21st position in the array.
+
+### Return clicks for a specific product range:
+
+ 0: jdbc:drill:> select * from (select t.trans_id, t.trans_info.prod_id[0] as prodid,
+ t.trans_info.purch_flag as purchased
+ from `clicks/clicks.json` t) sq
+ where sq.prodid between 700 and 750 and sq.purchased='true'
+ order by sq.prodid;
+ +------------+------------+------------+
+ | trans_id | prodid | purchased |
+ +------------+------------+------------+
+ | 21886 | 704 | true |
+ | 20674 | 708 | true |
+ | 22158 | 709 | true |
+ | 34089 | 714 | true |
+ | 22545 | 714 | true |
+ | 37500 | 717 | true |
+ | 36595 | 718 | true |
+ ...
+
+This query assumes that there is some meaning to the array (that it is an
+ordered list of products purchased rather than a random list).
+
+## Perform Operations on Arrays
+
+### Rank successful click conversions and count product searches for each session:
+
+ 0: jdbc:drill:> select t.trans_id, t.`date` as session_date, t.user_info.cust_id as
+ cust_id, t.user_info.device as device, repeated_count(t.trans_info.prod_id) as
+ prod_count, t.trans_info.purch_flag as purch_flag
+ from `clicks/clicks.json` t
+ where t.trans_info.purch_flag = 'true' order by prod_count desc;
+ +------------+--------------+------------+------------+------------+------------+
+ | trans_id | session_date | cust_id | device | prod_count | purch_flag |
+ +------------+--------------+------------+------------+------------+------------+
+ | 37426 | 2014-04-06 | 18709 | IOS5 | 34 | true |
+ | 31589 | 2014-04-16 | 18576 | IOS6 | 31 | true |
+ | 11600 | 2014-04-07 | 4260 | AOS4.2 | 28 | true |
+ | 35074 | 2014-04-03 | 16697 | AOS4.3 | 27 | true |
+ | 17192 | 2014-04-22 | 2501 | AOS4.2 | 26 | true |
+ ...
+
+This query uses a Drill SQL extension, the repeated_count function, to get an
+aggregated count of the array values. The query returns the number of products
+searched for each session that converted into a purchase and ranks the counts
+in descending order. Only clicks that have resulted in a purchase are counted.
+
+## Store a Result Set in a Table for Reuse and Analysis
+
+Finally, run another correlated subquery that returns a fairly large result
+set. To facilitate additional analysis on this result set, you can easily and
+quickly create a Drill table from the results of the query.
+
+### Continue to use the dfs.clicks workspace
+
+ 0: jdbc:drill:> use dfs.clicks;
+ +------------+------------+
+ | ok | summary |
+ +------------+------------+
+ | true | Default schema changed to 'dfs.clicks' |
+ +------------+------------+
+
+### Return product searches for high-value customers:
+
+ 0: jdbc:drill:> select o.cust_id, o.order_total, t.trans_info.prod_id[0] as prod_id
+ from hive.orders as o, `clicks/clicks.json` t
+ where o.cust_id=t.user_info.cust_id
+ and o.order_total > (select avg(inord.order_total)
+ from hive.orders inord where inord.state = o.state);
+ +------------+-------------+------------+
+ | cust_id | order_total | prod_id |
+ +------------+-------------+------------+
+ ...
+ | 9650 | 69 | 16 |
+ | 9650 | 69 | 560 |
+ | 9650 | 69 | 959 |
+ | 9654 | 76 | 768 |
+ | 9656 | 76 | 32 |
+ | 9656 | 76 | 16 |
+ ...
+ +------------+-------------+------------+
+ 106,281 rows selected
+
+This query returns a list of products that are being searched for by customers
+who have made transactions that are above the average in their states.
+
+### Materialize the result of the previous query:
+
+ 0: jdbc:drill:> create table product_search as select o.cust_id, o.order_total, t.trans_info.prod_id[0] as prod_id
+ from hive.orders as o, `clicks/clicks.json` t
+ where o.cust_id=t.user_info.cust_id and o.order_total > (select avg(inord.order_total)
+ from hive.orders inord where inord.state = o.state);
+ +------------+---------------------------+
+ | Fragment | Number of records written |
+ +------------+---------------------------+
+ | 0_0 | 106281 |
+ +------------+---------------------------+
+ 1 row selected
+
+This example uses a CTAS statement to create a table based on a correlated
+subquery that you ran previously. This table contains all of the rows that the
+query returns (106,281) and stores them in the format specified by the storage
+plugin (Parquet format in this example). You can create tables that store data
+in csv, parquet, and json formats.
+
+### Query the new table to verify the row count:
+
+This example simply checks that the CTAS statement worked by verifying the
+number of rows in the table.
+
+ 0: jdbc:drill:> select count(*) from product_search;
+ +------------+
+ | EXPR$0 |
+ +------------+
+ | 106281 |
+ +------------+
+ 1 row selected
+
+### Find the storage file for the table:
+
+ [root@maprdemo product_search]# cd /mapr/demo.mapr.com/data/nested/product_search
+ [root@maprdemo product_search]# ls -la
+ total 451
+ drwxr-xr-x. 2 mapr mapr 1 Sep 15 13:41 .
+ drwxr-xr-x. 4 root root 2 Sep 15 13:41 ..
+ -rwxr-xr-x. 1 mapr mapr 460715 Sep 15 13:41 0_0_0.parquet
+
+Note that the table is stored in a file called `0_0_0.parquet`. This file is
+stored in the location defined by the dfs.clicks workspace:
+
+ "location": "http://demo.mapr.com/data/nested)"
+
+with a subdirectory that has the same name as the table you created.
+
+## What's Next
+
+Complete the tutorial with the [Summary](/confluence/display/DRILL/Summary).
+
+
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/tutorial/006-summary.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/tutorial/006-summary.md b/_docs/drill-docs/tutorial/006-summary.md
new file mode 100644
index 0000000..f210766
--- /dev/null
+++ b/_docs/drill-docs/tutorial/006-summary.md
@@ -0,0 +1,14 @@
+---
+title: "Summary"
+parent: "Apache Drill Tutorial"
+---
+This tutorial introduced Apache Drill and its ability to run ANSI SQL queries
+against various data sources, including Hive tables, MapR-DB/HBase tables, and
+file system directories. The tutorial also showed how to work with and
+manipulate complex and multi-structured data commonly found in Hadoop/NoSQL
+systems.
+
+Now that you are familiar with different ways to access the sample data with
+Drill, you can try writing your own queries against your own data sources.
+Refer to the [Apache Drill documentation](https://cwiki.apache.org/confluence/
+display/DRILL/Apache+Drill+Wiki) for more information.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/tutorial/install-sandbox/001-install-mapr-vm.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/tutorial/install-sandbox/001-install-mapr-vm.md b/_docs/drill-docs/tutorial/install-sandbox/001-install-mapr-vm.md
new file mode 100644
index 0000000..f3d8953
--- /dev/null
+++ b/_docs/drill-docs/tutorial/install-sandbox/001-install-mapr-vm.md
@@ -0,0 +1,55 @@
+---
+title: "Installing the MapR Sandbox with Apache Drill on VMware Player/VMware Fusion"
+parent: "Installing the Apache Drill Sandbox"
+---
+Complete the following steps to install the MapR Sandbox with Apache Drill on
+VMware Player or VMware Fusion:
+
+ 1. Download the MapR Sandbox with Drill file to a directory on your machine:
+<https://www.mapr.com/products/mapr-sandbox-hadoop/download-sandbox-drill>
+
+ 2. Open the virtual machine player, and select the **Open a Virtual Machine **option.
+
+ Tip for VMware Fusion
+
+If you are running VMware Fusion, select** Import**.
+
+
+
+ 3. Navigate to the directory where you downloaded the MapR Sandbox with Apache Drill file, and select `MapR-Sandbox-For-Apache-Drill-4.0.1_VM.ova`.
+
+
+
+The Import Virtual Machine dialog appears.
+
+ 4. Click **Import**. The virtual machine player imports the sandbox.
+
+
+
+ 5. Select `MapR-Sandbox-For-Apache-Drill-4.0.1_VM`, and click **Play virtual machine**. It takes a few minutes for the MapR services to start.
+After the MapR services start and installation completes, the following screen
+appears:
+
+
+
+Note the URL provided in the screen, which corresponds to the Web UI in Apache
+Drill.
+
+ 6. Verify that a DNS entry was created on the host machine for the virtual machine. If not, create the entry.
+
+ * For Linux and Mac, create the entry in `/etc/hosts`.
+
+ * For WIndows, create the entry in the `%WINDIR%\system32\drivers\etc\hosts` file.
+Example: `127.0.1.1 <vm_hostname>`
+
+ 7. You can navigate to the URL provided to experience Drill Web UI or you can login to the sandbox through the command line.
+
+ a. To navigate to the MapR Sandbox with Apache Drill, enter the provided URL in your browser's address bar.
+
+ b. To login to the virtual machine and access the command line, press Alt+F2 on Windows or Option+F5 on Mac. When prompted, enter `mapr` as the login and password.
+
+# What's Next
+
+After downloading and installing the sandbox, continue with the tutorial by
+[Getting to Know the Drill
+Setup](/confluence/display/DRILL/Getting+to+Know+the+Drill+Setup).
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/tutorial/install-sandbox/002-install-mapr-vb.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/tutorial/install-sandbox/002-install-mapr-vb.md b/_docs/drill-docs/tutorial/install-sandbox/002-install-mapr-vb.md
new file mode 100644
index 0000000..9ff26d5
--- /dev/null
+++ b/_docs/drill-docs/tutorial/install-sandbox/002-install-mapr-vb.md
@@ -0,0 +1,72 @@
+---
+title: "Installing the MapR Sandbox with Apache Drill on VirtualBox"
+parent: "Installing the Apache Drill Sandbox"
+---
+The MapR Sandbox for Apache Drill on VirtualBox comes with NAT port forwarding
+enabled, which allows you to access the sandbox using localhost as hostname.
+
+Complete the following steps to install the MapR Sandbox with Apache Drill on
+VirtualBox:
+
+ 1. Download the MapR Sandbox with Apache Drill file to a directory on your machine:
+<https://www.mapr.com/products/mapr-sandbox-hadoop/download-sandbox-drill>
+
+ 2. Open the virtual machine player.
+
+ 3. Select **File > Import Appliance**. The Import Virtual Appliance dialog appears.
+
+ 
+
+ 4. Navigate to the directory where you downloaded the MapR Sandbox with Apache Drill and click **Next**. The Appliance Settings window appears.
+
+ 
+
+ 5. Select the check box at the bottom of the screen: **Reinitialize the MAC address of all network cards**, then click **Import**. The Import Appliance imports the sandbox.
+
+ 6. When the import completes, select **File > Preferences**. The VirtualBox - Settings dialog appears.
+
+ 
+
+ 7. Select **Network**.
+
+ The correct setting depends on your network connectivity when you run the
+Sandbox. In general, if you are going to use a wired Ethernet connection,
+select **NAT Networks **and **vboxnet0**. If you are going to use a wireless
+network, select **Host-only Networks** and the **VirtualBox Host-Only Ethernet
+Adapter**. If no adapters appear, click the green** +** button to add the
+VirtualBox adapter.
+
+ 
+
+ 8. Click **OK **to continue.
+
+ 9. Click . The MapR-Sandbox-For-Apache-Drill-0.6.0-r2-4.0.1 - Settings dialog appears.
+
+ 
+
+ 10. Click **OK** to continue.
+
+ 11. Click **Start**. It takes a few minutes for the MapR services to start. After the MapR services start and installation completes, the following screen appears:
+
+ 
+
+ 12. The client must be able to resolve the actual hostname of the Drill node(s) with the IP(s). Verify that a DNS entry was created on the client machine for the Drill node(s).
+If a DNS entry does not exist, create the entry for the Drill node(s).
+
+ * For Windows, create the entry in the %WINDIR%\system32\drivers\etc\hosts file.
+
+ * For Linux and Mac, create the entry in /etc/hosts.
+<drill-machine-IP> <drill-machine-hostname>
+Example: `127.0.1.1 maprdemo`
+
+ 13. You can navigate to the URL provided or to [localhost:8047](http://localhost:8047) to experience the Drill Web UI, or you can log into the sandbox through the command line.
+
+ a. To navigate to the MapR Sandbox with Apache Drill, enter the provided URL in your browser's address bar.
+
+ b. To log into the virtual machine and access the command line, enter Alt+F2 on Windows or Option+F5 on Mac. When prompted, enter `mapr` as the login and password.
+
+# What's Next
+
+After downloading and installing the sandbox, continue with the tutorial by
+[Getting to Know the Drill
+Setup](/confluence/display/DRILL/Getting+to+Know+the+Drill+Setup).
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/11.png
----------------------------------------------------------------------
diff --git a/_docs/img/11.png b/_docs/img/11.png
new file mode 100644
index 0000000..32e977a
Binary files /dev/null and b/_docs/img/11.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/18.png
----------------------------------------------------------------------
diff --git a/_docs/img/18.png b/_docs/img/18.png
new file mode 100644
index 0000000..691b816
Binary files /dev/null and b/_docs/img/18.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/19.png
----------------------------------------------------------------------
diff --git a/_docs/img/19.png b/_docs/img/19.png
new file mode 100644
index 0000000..fb02151
Binary files /dev/null and b/_docs/img/19.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/21.png
----------------------------------------------------------------------
diff --git a/_docs/img/21.png b/_docs/img/21.png
new file mode 100644
index 0000000..9d9d121
Binary files /dev/null and b/_docs/img/21.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/30.png
----------------------------------------------------------------------
diff --git a/_docs/img/30.png b/_docs/img/30.png
new file mode 100644
index 0000000..78a1c57
Binary files /dev/null and b/_docs/img/30.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/4.png
----------------------------------------------------------------------
diff --git a/_docs/img/4.png b/_docs/img/4.png
new file mode 100644
index 0000000..39d1a19
Binary files /dev/null and b/_docs/img/4.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/40.png
----------------------------------------------------------------------
diff --git a/_docs/img/40.png b/_docs/img/40.png
new file mode 100644
index 0000000..1b13fc8
Binary files /dev/null and b/_docs/img/40.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/42.png
----------------------------------------------------------------------
diff --git a/_docs/img/42.png b/_docs/img/42.png
new file mode 100644
index 0000000..9891423
Binary files /dev/null and b/_docs/img/42.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/46.png
----------------------------------------------------------------------
diff --git a/_docs/img/46.png b/_docs/img/46.png
new file mode 100644
index 0000000..4ef7a30
Binary files /dev/null and b/_docs/img/46.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/51.png
----------------------------------------------------------------------
diff --git a/_docs/img/51.png b/_docs/img/51.png
new file mode 100644
index 0000000..fdb1a1f
Binary files /dev/null and b/_docs/img/51.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/52.png
----------------------------------------------------------------------
diff --git a/_docs/img/52.png b/_docs/img/52.png
new file mode 100644
index 0000000..5550953
Binary files /dev/null and b/_docs/img/52.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/53.png
----------------------------------------------------------------------
diff --git a/_docs/img/53.png b/_docs/img/53.png
new file mode 100644
index 0000000..c3691bc
Binary files /dev/null and b/_docs/img/53.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/54.png
----------------------------------------------------------------------
diff --git a/_docs/img/54.png b/_docs/img/54.png
new file mode 100644
index 0000000..229571e
Binary files /dev/null and b/_docs/img/54.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/7.png
----------------------------------------------------------------------
diff --git a/_docs/img/7.png b/_docs/img/7.png
new file mode 100644
index 0000000..b77c63e
Binary files /dev/null and b/_docs/img/7.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/DrillWebUI.png
----------------------------------------------------------------------
diff --git a/_docs/img/DrillWebUI.png b/_docs/img/DrillWebUI.png
new file mode 100644
index 0000000..2d5c16a
Binary files /dev/null and b/_docs/img/DrillWebUI.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/DrillbitModules.png
----------------------------------------------------------------------
diff --git a/_docs/img/DrillbitModules.png b/_docs/img/DrillbitModules.png
new file mode 100644
index 0000000..2eb9904
Binary files /dev/null and b/_docs/img/DrillbitModules.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/Overview.png
----------------------------------------------------------------------
diff --git a/_docs/img/Overview.png b/_docs/img/Overview.png
new file mode 100644
index 0000000..fc78213
Binary files /dev/null and b/_docs/img/Overview.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/StoragePluginConfig.png
----------------------------------------------------------------------
diff --git a/_docs/img/StoragePluginConfig.png b/_docs/img/StoragePluginConfig.png
new file mode 100644
index 0000000..e57fd38
Binary files /dev/null and b/_docs/img/StoragePluginConfig.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/drill-runtime.png
----------------------------------------------------------------------
diff --git a/_docs/img/drill-runtime.png b/_docs/img/drill-runtime.png
new file mode 100644
index 0000000..e551d73
Binary files /dev/null and b/_docs/img/drill-runtime.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/drill2.png
----------------------------------------------------------------------
diff --git a/_docs/img/drill2.png b/_docs/img/drill2.png
new file mode 100644
index 0000000..3fcbbf3
Binary files /dev/null and b/_docs/img/drill2.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/example_query.png
----------------------------------------------------------------------
diff --git a/_docs/img/example_query.png b/_docs/img/example_query.png
new file mode 100644
index 0000000..410c22b
Binary files /dev/null and b/_docs/img/example_query.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/loginSandBox.png
----------------------------------------------------------------------
diff --git a/_docs/img/loginSandBox.png b/_docs/img/loginSandBox.png
new file mode 100644
index 0000000..30f73b2
Binary files /dev/null and b/_docs/img/loginSandBox.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/queryFlow.png
----------------------------------------------------------------------
diff --git a/_docs/img/queryFlow.png b/_docs/img/queryFlow.png
new file mode 100644
index 0000000..38183ac
Binary files /dev/null and b/_docs/img/queryFlow.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/slide-15-638.png
----------------------------------------------------------------------
diff --git a/_docs/img/slide-15-638.png b/_docs/img/slide-15-638.png
new file mode 100644
index 0000000..ffea69f
Binary files /dev/null and b/_docs/img/slide-15-638.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/storageplugin.png
----------------------------------------------------------------------
diff --git a/_docs/img/storageplugin.png b/_docs/img/storageplugin.png
new file mode 100644
index 0000000..fa04517
Binary files /dev/null and b/_docs/img/storageplugin.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/value1.png
----------------------------------------------------------------------
diff --git a/_docs/img/value1.png b/_docs/img/value1.png
new file mode 100644
index 0000000..bd799d0
Binary files /dev/null and b/_docs/img/value1.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/value2.png
----------------------------------------------------------------------
diff --git a/_docs/img/value2.png b/_docs/img/value2.png
new file mode 100644
index 0000000..832a485
Binary files /dev/null and b/_docs/img/value2.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/value3.png
----------------------------------------------------------------------
diff --git a/_docs/img/value3.png b/_docs/img/value3.png
new file mode 100644
index 0000000..6fad0e6
Binary files /dev/null and b/_docs/img/value3.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/value4.png
----------------------------------------------------------------------
diff --git a/_docs/img/value4.png b/_docs/img/value4.png
new file mode 100644
index 0000000..e99de82
Binary files /dev/null and b/_docs/img/value4.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/value5.png
----------------------------------------------------------------------
diff --git a/_docs/img/value5.png b/_docs/img/value5.png
new file mode 100644
index 0000000..de6f8d3
Binary files /dev/null and b/_docs/img/value5.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/value6.png
----------------------------------------------------------------------
diff --git a/_docs/img/value6.png b/_docs/img/value6.png
new file mode 100644
index 0000000..127f4b1
Binary files /dev/null and b/_docs/img/value6.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/value7.png
----------------------------------------------------------------------
diff --git a/_docs/img/value7.png b/_docs/img/value7.png
new file mode 100644
index 0000000..8720a07
Binary files /dev/null and b/_docs/img/value7.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/vbApplSettings.png
----------------------------------------------------------------------
diff --git a/_docs/img/vbApplSettings.png b/_docs/img/vbApplSettings.png
new file mode 100644
index 0000000..2f7451b
Binary files /dev/null and b/_docs/img/vbApplSettings.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/vbEthernet.png
----------------------------------------------------------------------
diff --git a/_docs/img/vbEthernet.png b/_docs/img/vbEthernet.png
new file mode 100644
index 0000000..c5bf85c
Binary files /dev/null and b/_docs/img/vbEthernet.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/vbGenSettings.png
----------------------------------------------------------------------
diff --git a/_docs/img/vbGenSettings.png b/_docs/img/vbGenSettings.png
new file mode 100644
index 0000000..cae235f
Binary files /dev/null and b/_docs/img/vbGenSettings.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/vbImport.png
----------------------------------------------------------------------
diff --git a/_docs/img/vbImport.png b/_docs/img/vbImport.png
new file mode 100644
index 0000000..e2f6cfe
Binary files /dev/null and b/_docs/img/vbImport.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/vbMaprSetting.png
----------------------------------------------------------------------
diff --git a/_docs/img/vbMaprSetting.png b/_docs/img/vbMaprSetting.png
new file mode 100644
index 0000000..b7720e3
Binary files /dev/null and b/_docs/img/vbMaprSetting.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/vbNetwork.png
----------------------------------------------------------------------
diff --git a/_docs/img/vbNetwork.png b/_docs/img/vbNetwork.png
new file mode 100644
index 0000000..bbc1c7a
Binary files /dev/null and b/_docs/img/vbNetwork.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/vbloginSandBox.png
----------------------------------------------------------------------
diff --git a/_docs/img/vbloginSandBox.png b/_docs/img/vbloginSandBox.png
new file mode 100644
index 0000000..69c31ab
Binary files /dev/null and b/_docs/img/vbloginSandBox.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/vmLibrary.png
----------------------------------------------------------------------
diff --git a/_docs/img/vmLibrary.png b/_docs/img/vmLibrary.png
new file mode 100644
index 0000000..c0b97a3
Binary files /dev/null and b/_docs/img/vmLibrary.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/vmShare.png
----------------------------------------------------------------------
diff --git a/_docs/img/vmShare.png b/_docs/img/vmShare.png
new file mode 100644
index 0000000..16ef052
Binary files /dev/null and b/_docs/img/vmShare.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/img/vmWelcome.png
----------------------------------------------------------------------
diff --git a/_docs/img/vmWelcome.png b/_docs/img/vmWelcome.png
new file mode 100644
index 0000000..84aa4a4
Binary files /dev/null and b/_docs/img/vmWelcome.png differ
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/user-guide/001-views.md
----------------------------------------------------------------------
diff --git a/_docs/user-guide/001-views.md b/_docs/user-guide/001-views.md
deleted file mode 100644
index 6eda05e..0000000
--- a/_docs/user-guide/001-views.md
+++ /dev/null
@@ -1,5 +0,0 @@
----
-title: "Views"
-parent: "User Guide"
----
-Views!
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/user-guide/002-sql-syntax.md
----------------------------------------------------------------------
diff --git a/_docs/user-guide/002-sql-syntax.md b/_docs/user-guide/002-sql-syntax.md
deleted file mode 100644
index bb3b884..0000000
--- a/_docs/user-guide/002-sql-syntax.md
+++ /dev/null
@@ -1,5 +0,0 @@
----
-title: "SQL Syntax"
-parent: "User Guide"
----
-SQL Syntax!
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/user-guide/sql-syntax/001-ddd-ddd.md
----------------------------------------------------------------------
diff --git a/_docs/user-guide/sql-syntax/001-ddd-ddd.md b/_docs/user-guide/sql-syntax/001-ddd-ddd.md
deleted file mode 100644
index 6ecdb6c..0000000
--- a/_docs/user-guide/sql-syntax/001-ddd-ddd.md
+++ /dev/null
@@ -1,7 +0,0 @@
----
-title: "A page on Ds"
-parent: "SQL Syntax"
----
-This is a documentation page.
-
-It talks about Ds.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/user-guide/views/001-aaa-aaa.md
----------------------------------------------------------------------
diff --git a/_docs/user-guide/views/001-aaa-aaa.md b/_docs/user-guide/views/001-aaa-aaa.md
deleted file mode 100644
index f8438a8..0000000
--- a/_docs/user-guide/views/001-aaa-aaa.md
+++ /dev/null
@@ -1,7 +0,0 @@
----
-title: "This is Aaa Aaa"
-parent: "Views"
----
-This is a documentation page.
-
-It talks about As.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/user-guide/views/002-bbb-bbb.md
----------------------------------------------------------------------
diff --git a/_docs/user-guide/views/002-bbb-bbb.md b/_docs/user-guide/views/002-bbb-bbb.md
deleted file mode 100644
index 6baa2bc..0000000
--- a/_docs/user-guide/views/002-bbb-bbb.md
+++ /dev/null
@@ -1,7 +0,0 @@
----
-title: "Bbb Bbb"
-parent: "Views"
----
-This is a documentation page.
-
-It talks about Bs.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/user-guide/views/003-ccc-ccc.md
----------------------------------------------------------------------
diff --git a/_docs/user-guide/views/003-ccc-ccc.md b/_docs/user-guide/views/003-ccc-ccc.md
deleted file mode 100644
index 83643ab..0000000
--- a/_docs/user-guide/views/003-ccc-ccc.md
+++ /dev/null
@@ -1,7 +0,0 @@
----
-title: "Another page"
-parent: "Views"
----
-This is a documentation page.
-
-It talks about Cs.
\ No newline at end of file
[5/6] drill git commit: Added Drill docs
Posted by ts...@apache.org.
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/arch/001-core-mod.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/arch/001-core-mod.md b/_docs/drill-docs/arch/001-core-mod.md
new file mode 100644
index 0000000..81a8107
--- /dev/null
+++ b/_docs/drill-docs/arch/001-core-mod.md
@@ -0,0 +1,30 @@
+---
+title: "Core Modules within a Drillbit"
+parent: "Architectural Overview"
+---
+The following image represents components within each Drillbit:
+
+
+
+The following list describes the key components of a Drillbit:
+
+ * **RPC end point**: Drill exposes a low overhead protobuf-based RPC protocol to communicate with the clients. Additionally, a C++ and Java API layers are also available for the client applications to interact with Drill. Clients can communicate to a specific Drillbit directly or go through a ZooKeeper quorum to discover the available Drillbits before submitting queries. It is recommended that the clients always go through ZooKeeper to shield clients from the intricacies of cluster management, such as the addition or removal of nodes.
+
+ * **SQL parser**: Drill uses Optiq, the open source framework, to parse incoming queries. The output of the parser component is a language agnostic, computer-friendly logical plan that represents the query.
+ * **Storage plugin interfaces**: Drill serves as a query layer on top of several data sources. Storage plugins in Drill represent the abstractions that Drill uses to interact with the data sources. Storage plugins provide Drill with the following information:
+ * Metadata available in the source
+ * Interfaces for Drill to read from and write to data sources
+ * Location of data and a set of optimization rules to help with efficient and faster execution of Drill queries on a specific data source
+
+In the context of Hadoop, Drill provides storage plugins for files and
+HBase/M7. Drill also integrates with Hive as a storage plugin since Hive
+provides a metadata abstraction layer on top of files, HBase/M7, and provides
+libraries to read data and operate on these sources (Serdes and UDFs).
+
+When users query files and HBase/M7 with Drill, they can do it directly or go
+through Hive if they have metadata defined there. Drill integration with Hive
+is only for metadata. Drill does not invoke the Hive execution engine for any
+requests.
+
+ * **Distributed cache: **Drill uses a distributed cache to manage metadata (not the data) and configuration information across various nodes. Sample metadata information that is stored in the cache includes query plan fragments, intermediate state of the query execution, and statistics. Drill uses Infinispan as its cache technology.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/arch/002-arch-hilite.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/arch/002-arch-hilite.md b/_docs/drill-docs/arch/002-arch-hilite.md
new file mode 100644
index 0000000..8761d62
--- /dev/null
+++ b/_docs/drill-docs/arch/002-arch-hilite.md
@@ -0,0 +1,15 @@
+---
+title: "Architectural Highlights"
+parent: "Architectural Overview"
+---
+The goal for Drill is to bring the **SQL Ecosystem** and **Performance** of
+the relational systems to **Hadoop scale** data **WITHOUT** compromising on
+the **Flexibility** of Hadoop/NoSQL systems. There are several core
+architectural elements in Apache Drill that make it a highly flexible and
+efficient query engine.
+
+Click on either of the following links to read more about Drill's performance
+and flexibility:
+
+ * [Flexibility](/confluence/display/DRILL/Flexibility)
+ * [Performance](/confluence/display/DRILL/Performance)
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/arch/arch-hilite/001-flexibility.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/arch/arch-hilite/001-flexibility.md b/_docs/drill-docs/arch/arch-hilite/001-flexibility.md
new file mode 100644
index 0000000..a36bb9b
--- /dev/null
+++ b/_docs/drill-docs/arch/arch-hilite/001-flexibility.md
@@ -0,0 +1,79 @@
+---
+title: "Flexibility"
+parent: "Architectural Highlights"
+---
+The following features contribute to Drill's flexible architecture:
+
+**_Dynamic schema discovery _**
+
+Drill does not require schema or type specification for the data in order to
+start the query execution process. Instead, Drill starts processing the data
+in units called record-batches and discovers the schema on the fly during
+processing. Self-describing data formats such as Parquet, JSON, AVRO, and
+NoSQL databases have schema specified as part of the data itself, which Drill
+leverages dynamically at query time. Schema can change over the course of a
+Drill query, so all of the Drill operators are designed to reconfigure
+themselves when such schema changing events occur.
+
+**_Flexible data model_**
+
+Drill is purpose-built from the ground up for complex/multi-structured data
+commonly seen in Hadoop/NoSQL applications such as social/mobile, clickstream,
+logs, and sensor equipped IOT. From a user point of view, Drill allows access
+to nested data attributes, just like SQL columns, and provides intuitive
+extensions to easily operate on them. From an architectural point of view,
+Drill provides a flexible hierarchical columnar data model that can represent
+complex, highly dynamic and evolving data models, and allows for efficient
+processing of it without the need to flatten or materialize it at design time
+or at execution time. Relational data in Drill is treated as a special or
+simplified case of complex/multi-structured data.
+
+**_De-centralized metadata_**
+
+Unlike other SQL-on-Hadoop technologies or any traditional relational
+database, Drill does not have a centralized metadata requirement. In order to
+query data through Drill, users do not need to create and manage tables and
+views in a metadata repository, or rely on a database administrator group for
+such a function.
+
+Drill metadata is derived from the storage plugins that correspond to data
+sources. Drill supports a varied set of storage plugins that provide a
+spectrum of metadata ranging from full metadata such as for Hive, partial
+metadata such as for HBase, or no central metadata such as for files.
+
+De-centralized metadata also means that Drill is NOT tied to a single Hive
+repository. Users can query multiple Hive repositories at once and then
+combine the data with information from HBase tables or with a file in a
+distributed file system.
+
+Users also have the ability to create metadata (tables/views/databases) within
+Drill using the SQL DDL syntax. De-centralized metadata is applicable during
+metadata creation. Drill allows persisting metadata in one of the underlying
+data sources.
+
+From a client access perspective, Drill metadata is organized just like a
+traditional DB (Databases->Tables/Views->Columns). The metadata is accessible
+through the ANSI standard INFORMATION_SCHEMA database
+
+For more information on how to configure and work various data sources with
+Drill, refer to [Connect Apache Drill to Data Sources](https://cwiki.apache.or
+g/confluence/display/DRILL/Connect+Apache+Drill+to+Data+Sources).
+
+**_Extensibility_**
+
+Drill provides an extensible architecture at all layers, including the storage
+plugin, query, query optimization/execution, and client API layers. You can
+customize any layer for the specific needs of an organization or you can
+extend the layer to a broader array of use cases.
+
+Drill provides a built in classpath scanning and plugin concept to add
+additional storage plugins, functions, and operators with minimal
+configuration.
+
+The following list provides a few examples of Drill’s extensible architectural
+capabilities:
+
+* A high performance Java API to implement custom UDFs/UDAFs
+* Ability to go beyond Hadoop by implementing custom storage plugins to other data sources such as Oracle/MySQL or NoSQL stores, such as Mongo or Cassandra
+* An API to implement custom operators
+* Support for direct execution of strongly specified JSON based logical and physical plans to help with the simplification of testing, and to enable integration of alternative query languages other than SQL.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/arch/arch-hilite/002-performance.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/arch/arch-hilite/002-performance.md b/_docs/drill-docs/arch/arch-hilite/002-performance.md
new file mode 100644
index 0000000..e18a5b4
--- /dev/null
+++ b/_docs/drill-docs/arch/arch-hilite/002-performance.md
@@ -0,0 +1,56 @@
+---
+title: "Performance"
+parent: "Architectural Highlights"
+---
+Drill is designed from the ground up for high performance on large datasets.
+The following core elements of Drill processing are responsible for Drill's
+performance:
+
+**_Distributed engine_**
+
+Drill provides a powerful distributed execution engine for processing queries.
+Users can submit requests to any node in the cluster. You can simply add new
+nodes to the cluster to scale for larger volumes of data, support more users
+or to improve performance.
+
+**_Columnar execution_**
+
+Drill optimizes for both columnar storage and execution by using an in-memory
+data model that is hierarchical and columnar. When working with data stored in
+columnar formats such as Parquet, Drill avoids disk access for columns that
+are not involved in an analytic query. Drill also provides an execution layer
+that performs SQL processing directly on columnar data without row
+materialization. The combination of optimizations for columnar storage and
+direct columnar execution significantly lowers memory footprints and provides
+faster execution of BI/Analytic type of workloads.
+
+**_Vectorization_**
+
+Rather than operating on single values from a single table record at one time,
+vectorization in Drill allows the CPU to operate on vectors, referred to as a
+Record Batches. Record Batches are arrays of values from many different
+records. The technical basis for efficiency of vectorized processing is modern
+chip technology with deep-pipelined CPU designs. Keeping all pipelines full to
+achieve efficiency near peak performance is something impossible to achieve in
+traditional database engines, primarily due to code complexity**.**
+
+**_Runtime compilation_**
+
+Runtime compilation is faster compared to the interpreted execution. Drill
+generates highly efficient custom code for every single query for every single
+operator. Here is a quick overview of the Drill compilation/code generation
+process at a glance.
+
+
+
+**Optimistic and pipelined query execution**
+
+Drill adopts an optimistic execution model to process queries. Drill assumes
+that failures are infrequent within the short span of a query and therefore
+does not spend time creating boundaries or checkpoints to minimize recovery
+time. Failures at node level are handled gracefully. In the instance of a
+single query failure, the query is rerun. Drill execution uses a pipeline
+model where all tasks are scheduled at once. The query execution happens in-
+memory as much as possible to move data through task pipelines, persisting to
+disk only if there is memory overflow.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/archive/001-how-to-demo.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/archive/001-how-to-demo.md b/_docs/drill-docs/archive/001-how-to-demo.md
new file mode 100644
index 0000000..1b46e88
--- /dev/null
+++ b/_docs/drill-docs/archive/001-how-to-demo.md
@@ -0,0 +1,309 @@
+---
+title: "How to Run the Drill Demo"
+parent: "Archived Pages"
+---
+# How To Run the Drill Demo
+This section describes how to get started by running the Drill demo.
+
+## Pre-requisites
+
+ * Maven 2 or higher
+
+ On Ubuntu, you can do this as root:
+
+ apt-get install maven2
+
+ On the Mac, maven is pre-installed.
+
+ Note that installing maven can result in installing java 1.6 and setting that
+to your default version. Make sure you check java version before compiling or
+running.
+
+ * Java 1.7
+
+ You will need java 1.7 to compile and run the Drill demo.
+
+ On Ubuntu you can get the right version of Java by doing this as root:
+
+ apt-get install openjdk-7-jdk
+ sudo update-alternatives --set java $(update-alternatives --list java | grep 7 | head -1)
+
+ On a Mac, go to [Oracle's web-
+site](http://www.oracle.com/technetwork/java/javase/downloads/java-se-
+jdk-7-download-432154.html) to download and install java 7. You will also need
+to set JAVA_HOME in order to use the right version of java.
+
+ Drill will not compile correctly using java 6. There is also a subtle problem
+that can occur if you have both
+java 6 and java 7 with the default version set to 6. In that case, you may be
+able to compile, but execution may not work correctly.
+
+ Send email to the dev list if this is a problem for you.
+
+ * Protobuf
+
+ Drill requires Protobuf 2.5. Install this on Ubuntu using:
+
+ apt-get install protobuf-compiler
+
+ On Centos 6.4, OEL or RHEL you will need to compile protobuf-compiler:
+
+ wget http://protobuf.googlecode.com/files/protobuf-2.5.0.tar.bz2
+ tar xfj protobuf-2.5.0.tar.bz2
+ pushd protobuf-2.5.0
+ ./configure
+ make
+ sudo make install
+
+ * git
+
+ On Ubuntu you can install git by doing this as root:
+
+ apt-get install git-all
+
+On the Mac or Windows, go to [this site](http://git-scm.com/downloads) to
+download and install git.
+
+## Check your installation versions
+
+Run
+
+ java -version
+ mvn -version
+
+Verify that your default java and maven versions are correct, and that maven
+runs the right version of java. On my Mac, you see something like this:
+
+ ted:apache-drill-1.0.0-m1$ java -version
+ java version "1.7.0_11"
+ Java(TM) SE Runtime Environment (build 1.7.0_11-b21)
+ Java HotSpot(TM) 64-Bit Server VM (build 23.6-b04, mixed mode)
+ ted:apache-drill-1.0.0-m1$ mvn -version
+ Apache Maven 3.0.3 (r1075438; 2011-02-28 09:31:09-0800)
+ Maven home: /usr/share/maven
+ Java version: 1.7.0_11, vendor: Oracle Corporation
+ Java home: /Library/Java/JavaVirtualMachines/jdk1.7.0_11.jdk/Contents/Home/jre
+ Default locale: en_US, platform encoding: UTF-8
+ OS name: "mac os x", version: "10.7.5", arch: "x86_64", family: "mac"
+ ted:apache-drill-1.0.0-m1$
+
+## Get the Source
+
+ git clone https://git-wip-us.apache.org/repos/asf/incubator-drill.git
+
+## Compile the Code
+
+ cd incubator-drill/sandbox/prototype
+ mvn clean install -DskipTests
+ rm .classpath
+
+This takes about a minute on a not-terribly-current MacBook.
+
+## Run the interactive Drill shell
+
+ ./sqlline -u jdbc:drill:schema=parquet-local -n admin -p admin
+
+The first time you run this program, you will get reams of output. What is
+happening is that the program is running maven in order to build a
+(voluminous) class path for the actual program and stores this classpath into
+the file called `.classpath`. When you run this program again, it will note
+that this file already exists and avoid re-creating it. You should delete this
+file every time the dependencies of Drill are changed. If you start getting
+"class not found" errors, that is a good hint that `.classpath` is out of date
+and needs to be deleted and recreated.
+
+The `-u` argument to [sqlline](https://github.com/julianhyde/sqlline) is a
+JDBC connection string that directs sqlline to connect to drill. The Drill
+JDBC driver currently includes enough smarts to run Drill in embedded mode so
+this command also effectively starts a local drill bit. The `schema=` part of
+the JDBC connection string causes Drill to consider the "parquet-local"
+storage engine to be default. Other storage engines can be specified. The list
+of supported storage engines can be found in the file
+`./sqlparser/src/main/resources/storage-engines.json`. Each storage engine
+specifies the format of the data and how to get the data. See the section
+below on "Storage Engines" for more detail.
+
+When you run sqlline, you should see something like this after quite a lot of
+log messages:
+
+ Connected to: Drill (version 1.0)
+ Driver: Apache Drill JDBC Driver (version 1.0)
+ Autocommit status: true
+ Transaction isolation: TRANSACTION_REPEATABLE_READ
+ sqlline version ??? by Marc Prud'hommeaux
+ 0: jdbc:drill:schema=parquet-local>
+
+**Tip:** To quit sqlline at any time, type "!quit" at the prompt.
+
+ 0: jdbc:drill:schema=parquet-local> !quit
+
+## Run a Query
+
+Once you have sqlline running you can now try out some queries:
+
+ select * from "sample-data/region.parquet";
+
+You should see a number of debug messages and then something like this:
+
+ +--------------------------------------------------------------------------------------------------------------------------------------------+
+ | _MAP |
+ +--------------------------------------------------------------------------------------------------------------------------------------------+
+ | {"R_REGIONKEY":0,"R_NAME":"AFRICA","R_COMMENT":"lar deposits. blithely final packages cajole. regular waters are final requests. regular a |
+ | {"R_REGIONKEY":1,"R_NAME":"AMERICA","R_COMMENT":"hs use ironic, even requests. s"} |
+ | {"R_REGIONKEY":2,"R_NAME":"ASIA","R_COMMENT":"ges. thinly even pinto beans ca"} |
+ | {"R_REGIONKEY":3,"R_NAME":"EUROPE","R_COMMENT":"ly final courts cajole furiously final excuse"} |
+ | {"R_REGIONKEY":4,"R_NAME":"MIDDLE EAST","R_COMMENT":"uickly special accounts cajole carefully blithely close requests. carefully final asy |
+ +--------------------------------------------------------------------------------------------------------------------------------------------+
+ 5 rows selected (1.103 seconds)
+
+Drill has no idea what the structure of this
+file is in terms of what fields exist, but Drill does know that every record
+has a pseudo-field called `_MAP`. This field contains a map of all of the
+actual fields to values. When returned via JDBC, this fields is rendered as
+JSON since JDBC doesn't really understand maps.
+
+This can be made more readable by using a query like this:
+
+ select _MAP['R_REGIONKEY'] as region_key, _MAP['R_NAME'] as name, _MAP['R_COMMENT'] as comment
+ from "sample-data/region.parquet";
+
+The output will look something like this:
+
+ +-------------+--------------+---------------------------------------------------------------------------------------------------------------+
+ | REGION_KEY | NAME | COMMENT |
+ +-------------+--------------+---------------------------------------------------------------------------------------------------------------+
+ | 0 | AFRICA | lar deposits. blithely final packages cajole. regular waters are final requests. regular accounts are accordi |
+ | 1 | AMERICA | hs use ironic, even requests. s |
+ | 2 | ASIA | ges. thinly even pinto beans ca |
+ | 3 | EUROPE | ly final courts cajole furiously final excuse |
+ | 4 | MIDDLE EAST | uickly special accounts cajole carefully blithely close requests. carefully final asymptotes haggle furiousl |
+ +-------------+--------------+---------------------------------------------------------------------------------------------------------------+
+
+In upcoming versions, Drill will insert the `_MAP[ ... ]` goo and will also
+unwrap the contents of `_MAP` in results so that things seem much more like
+ordinary SQL. The reason that things work this way now is that SQL itself
+requires considerable type information for queries to be parsed and that
+information doesn't necessarily exist for all kinds of files, especially those
+with very flexible schemas. To avoid all these problems, Drill adopts the
+convention of the _MAP fields for all kinds of input.
+
+## A Note Before You Continue
+
+Drill currently supports a wide variety of queries. It currently also has a
+fair number of deficiencies in terms of the number of operators that are
+actually supported and exactly which expressions are passed through to the
+execution engine in the correct form for execution.
+
+These problems fall into roughly three categories,
+
+ * missing operators. Many operators have been implemented for only a subset of the types available in Drill. This will cause queries to work for some types of data, but not for others. This is particularly true for operators with many possible type signatures such as comparisons. This lack is being remedied at a fast pace so check back in frequently if you suspect this might be a problem.
+
+ Missing operators will result in error messages like this:
+
+ `UnsupportedOperationException:[ Missing function implementation: compare_to
+(BIT-OPTIONAL, BIT-OPTIONAL) ]`
+
+ * missing casts. The SQL parser currently has trouble producing a valid logical plan without sufficient type information. Ironically, this type information is often not necessary to the execution engine because Drill generates the code on the fly based on the types of the data it encounters as data are processed. Currently, the work-around is to cast fields in certain situations to give the parser enough information to proceed. This problem will be remedied soon, but probably not quite as quickly as the missing operators.
+
+ The typical error message that indicates you need an additional cast looks
+like
+
+ `Cannot apply '>' to arguments of type '<ANY> > <CHAR(1)>'. Supported
+form(s): '<COMPARABLE_TYPE> > <COMPARABLE_TYPE>'`
+
+ * weak optimizer. The current optimizer that transforms the logical plan into a physical plan is not the fully-featured cost based optimizer that Optiq normally uses. This is because some of the transformations that are needed for Drill are not yet fully supported by Optiq. In order to allow end-to-end execution of queries, a deterministic peep-hole optimizer has been used instead. This optimizer cannot handle large plan transformations and so some queries cannot be transformed correctly from logical to physical plan. We expect that the necessary changes to the cost-based optimizer will allow it to be used in an upcoming release, but didn't want to delay the current release waiting for that to happen.
+
+## Try Fancier Queries
+
+This query does a join between two files:
+
+ SELECT nations.name, regions.name FROM (
+ SELECT _MAP['N_REGIONKEY'] as regionKey, _MAP['N_NAME'] as name
+ FROM "sample-data/nation.parquet") nations
+ join (
+ SELECT _MAP['R_REGIONKEY'] as regionKey, _MAP['R_NAME'] as name
+ FROM "sample-data/region.parquet") regions
+ on nations.regionKey = regions.regionKey
+ order by nations.name;
+
+Notice the use of sub-queries to avoid the spread of the `_MAP` idiom.
+
+This query illustrates how a cast is currently necessary to make the parser
+happy:
+
+ SELECT
+ _MAP['N_REGIONKEY'] as regionKey,
+ _MAP['N_NAME'] as name
+ FROM
+ "sample-data/nation.parquet"
+ WHERE
+ cast(_MAP['N_NAME'] as varchar) IN ('MOROCCO', 'MOZAMBIQUE');
+
+Here are more queries that you can try.
+
+ // count distinct
+ SELECT count(distinct _MAP['N_REGIONKEY']) FROM "sample-data/nation.parquet";
+
+ // aliases
+ SELECT
+ _MAP['N_REGIONKEY'] as regionKey,
+ _MAP['N_NAME'] as name
+ FROM "sample-data/nation.parquet";
+
+ // order by
+ SELECT
+ _MAP['N_REGIONKEY'] as regionKey,
+ _MAP['N_NAME'] as name
+ FROM
+ "sample-data/nation.parquet"
+ ORDER BY
+ _MAP['N_NAME'] DESC;
+
+ // subquery order by
+ select * from (
+ SELECT
+ _MAP['N_REGIONKEY'] as regionKey,
+ _MAP['N_NAME'] as name
+ FROM
+ "sample-data/nation.parquet"
+ ) as x
+ ORDER BY
+ name DESC;
+
+ // String where
+ SELECT
+ _MAP['N_REGIONKEY'] as regionKey,
+ _MAP['N_NAME'] as name
+ FROM
+ "sample-data/nation.parquet"
+ WHERE
+ cast(_MAP['N_NAME'] as varchar) > 'M';
+
+ // INNER Join + Order (parquet)
+ SELECT n.name, r.name FROM
+ (SELECT _MAP['N_REGIONKEY'] as regionKey, _MAP['N_NAME'] as name FROM "sample-data/nation.parquet")n
+ join (SELECT _MAP['R_REGIONKEY'] as regionKey, _MAP['R_NAME'] as name FROM "sample-data/region.parquet")r
+ using (regionKey);
+
+ // INNER Join + Order (parquet)
+ SELECT n.name, r.name FROM
+ (SELECT _MAP['N_REGIONKEY'] as regionKey, _MAP['N_NAME'] as name FROM "sample-data/nation.parquet")n
+ join (SELECT _MAP['R_REGIONKEY'] as regionKey, _MAP['R_NAME'] as name FROM "sample-data/region.parquet")r
+ on n.regionKey = r.regionKey
+ order by n.name;
+
+## Analyze the Execution of Queries
+
+Drill sends log events to a logback socket appender. This makes it easy to
+catch and filter these log events using a tool called Lilith. You can download
+[Lilith](http://www.huxhorn.de/) and install it easily. A tutorial can be
+[found here](http://ekkescorner.wordpress.com/2009/09/05/osgi-logging-part-8
+-viewing-log-events-lilith/). This is especially important if you find errors
+that you want to report back to the mailing list since Lilith will help you
+isolate the stack trace of interest.
+
+By default, Lilith uses a slightly lurid splash page based on a pre-Raphaelite
+image of the mythical Lilith. This is easily disabled if the image is not to
+your taste (or if your work-mates are not well-versed in Victorian views of
+Sumerian mythology).
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/archive/002-meet-drill.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/archive/002-meet-drill.md b/_docs/drill-docs/archive/002-meet-drill.md
new file mode 100644
index 0000000..32d0ba1
--- /dev/null
+++ b/_docs/drill-docs/archive/002-meet-drill.md
@@ -0,0 +1,41 @@
+---
+title: "How to Run the Drill Demo"
+parent: "Archived Pages"
+---
+### What is Apache Drill
+
+Apache Drill by Apache Foundation is the first open source implementation of
+the Google's Dremel paper for interactive query processing. Apache Drill
+provides low latency ad-hoc queries to many different data sources & nested
+data. Drill is designed to scale to 10,000 servers and query petabytes of data
+in seconds.
+
+
+
+In a nutshell, Few key points about Apache Drill are:
+
+ * Inspired by Google's Dremel
+ * Supports standard SQL 2003
+ * Supports plug-able data sources (HBase, Mongo, HDFS etc)
+ * Supports nested data (JSON, ProtoBufs, Parquet etc)
+ * Supports optional schema
+ * Community driven
+
+### Where Apache Drill fits in
+
+Apache Drill is designed as an answer to the Interactive queries problems that
+we face while dealing with huge data. A standard Drill Query might take 100ms
+- 3 minutes for its execution as compared to Apache Hadoop or HIVE/PIG. Below
+is a diagram to help you relate in terms of the execution times:
+
+
+
+### Drill is powerful
+
+Below are few things that make Apache Drill really powerful:
+
+ * **Speed**: Apache Drill uses an efficient columnar storage format, an optimistic execution engine and a cache-conscious memory layout. Coordination, query planning, optimization, scheduling, and execution are all distributed throughout nodes in a system to maximize parallelization. Apache Drill is blazing fast. Period.
+ * **Plug-able data sources**: Apache drill brings in the support of pluggable data sources like HBase, Mongo, HDFS etc. It means that Drill will still work comfortably while your data is exploring new data stores.
+ * **Nested data**: With the support for data sources like HBase, Cassandra, MongoDB etc, Drill allows interactive analysis on all of your data, including nested and schema-less forms. Drill also supports querying against nested data formats like JSON and Parquet.
+ * **Flexibility**: Apache Dril strongly defined tiers and APIs for straightforward integration with a wide array of technologies.
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/connect/001-plugin-reg.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/connect/001-plugin-reg.md b/_docs/drill-docs/connect/001-plugin-reg.md
new file mode 100644
index 0000000..88c3c0b
--- /dev/null
+++ b/_docs/drill-docs/connect/001-plugin-reg.md
@@ -0,0 +1,39 @@
+---
+title: "Storage Plugin Registration"
+parent: "Connect to Data Sources"
+---
+You can connect Drill to a file system, Hive, or HBase data source. To connect
+Drill to a data source, you must register the data source as a storage plugin
+instance in the Drill Web UI. You register an instance of a data source as a
+`file`, `hive`, or `hbase` storage plugin type. You can register multiple
+storage plugin instances for each storage plugin type.
+
+Each node with a Drillbit installed has a running browser that provides access
+to the Drill Web UI at `http://localhost:8047/`. The Drill Web UI includes
+`cp`, `dfs`, `hive`, and `hbase` storage plugin instances by default, though
+the `hive` and `hbase` instances are disabled. You can update the `hive` and
+`hbase` instances with configuration details and then enable them.
+
+The `cp` instance points to a JAR file in Drill’s classpath that contains
+sample data that you can query. By default, the `dfs` instance points to the
+local file system on your machine, but you can configure this instance to
+point to any distributed file system, such as a Hadoop or S3 file system.
+
+When you add or update storage plugin instances on one Drill node in a Drill
+cluster, Drill broadcasts the information to all of the other Drill nodes so
+they all have identical storage plugin configurations. You do not need to
+restart any of the Drillbits when you add or update a storage plugin instance.
+
+Each storage plugin instance that you register with Drill must have a distinct
+name. For example, if you register two storage plugin instances for a Hadoop
+file system, you might name one storage plugin instance `hdfstest` and the
+other instance `hdfsproduction`.
+
+The following example shows an HDFS data source registered in the Drill Web UI
+as a storage plugin instance of plugin type "`file"`:
+
+
+
+Click the following link to learn about workspaces:
+
+ * [Workspaces](/confluence/display/DRILL/Workspaces)
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/connect/002-mongo-plugin.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/connect/002-mongo-plugin.md b/_docs/drill-docs/connect/002-mongo-plugin.md
new file mode 100644
index 0000000..665c79b
--- /dev/null
+++ b/_docs/drill-docs/connect/002-mongo-plugin.md
@@ -0,0 +1,169 @@
+---
+title: "MongoDB Plugin for Apache Drill"
+parent: "Connect to Data Sources"
+---
+# Overview
+
+You can leverage the power of Apache Drill to query data without any upfront
+schema definitions. Drill enables you to create an architecture that works
+with nested and dynamic schemas, making it the perfect SQL query tool to use
+on NoSQL databases, such as MongoDB.
+
+As of Apache Drill 0.6, you can configure MongoDB as a Drill data source.
+Drill provides a mongodb format plugin to connect to MongoDB, and run queries
+on the data using ANSI SQL.
+
+This tutorial assumes that you have Drill installed locally (embedded mode),
+as well as MongoDB. Examples in this tutorial use zip code aggregation data
+provided by MongDB. Before You Begin provides links to download tools and data
+used throughout the tutorial.
+
+**Note:** A local instance of Drill is used in this tutorial for simplicity. You can also run Drill and MongoDB together in distributed mode.
+
+### Before You Begin
+
+Before you can query MongoDB with Drill, you must have Drill and MongoDB
+installed on your machine. You may also want to import the MongoDB zip code
+data to run the example queries on your machine.
+
+ 1. [Install Drill](https://cwiki.apache.org/confluence/display/DRILL/Installing+Drill+in+Embedded+Mode), if you do not already have it installed on your machine.
+ 2. [Install MongoDB](http://docs.mongodb.org/manual/installation/), if you do not already have it installed on your machine.
+ 3. [Import the MongoDB zip code sample data set](http://docs.mongodb.org/manual/tutorial/aggregation-zip-code-data-set/). You can use Mongo Import to get the data.
+
+## Configuring MongoDB
+
+Start Drill and configure the MongoDB storage plugin instance in the Drill Web
+UI to connect to Drill. Drill must be running in order to access the Web UI.
+
+Complete the following steps to configure MongoDB as a data source for Drill:
+
+ 1. Navigate to `<drill_installation_directory>/drill-<version>,` and enter the following command to invoke SQLLine and start Drill:
+
+ bin/sqlline -u jdbc:drill:zk=local -n admin -p admin
+
+ When Drill starts, the following prompt appears:
+`0: jdbc:drill:zk=local>
+`Do not enter any commands. You will return to the command prompt after
+completing the configuration in the Drill Web UI.
+
+ 2. Open a browser window, and navigate to the Drill Web UI at `http://localhost:8047`.
+ 3. In the navigation bar, click **Storage**.
+ 4. Under Disabled Storage Plugins, select **Update** next to the `mongo` instance if the instance exists. If the instance does not exist, create an instance for MongoDB.
+ 5. In the Configuration window, verify that `"enabled"` is set to ``"true."``
+
+ **Example**
+
+ {
+ "type": "mongo",
+ "connection": "mongodb://localhost:27017/",
+ "enabled": true
+ }
+
+ **Note:** 27017 is the default port for `mongodb` instances.
+
+ 6. Click **Enable **to enable the instance, and save the** **configuration.
+ 7. Navigate back to the Drill command line so you can query MongoDB.
+
+## Querying MongoDB
+
+You can issue the `SHOW DATABASES `command to see a list of databases from all
+Drill data sources, including MongoDB. If you downloaded the zip codes file,
+you should see `mongo.zipdb` in the results.
+
+ 0: jdbc:drill:zk=local> SHOW DATABASES;
+ +-------------+
+ | SCHEMA_NAME |
+ +-------------+
+ | dfs.default |
+ | dfs.root |
+ | dfs.tmp |
+ | sys |
+ | mongo.zipdb |
+ | cp.default |
+ | INFORMATION_SCHEMA |
+ +-------------+
+
+If you want all queries that you submit to run on `mongo.zipdb`, you can issue
+the `USE` command to change schema.
+
+### Example Queries
+
+The following example queries are included for reference. However, you can use
+the SQL power of Apache Drill directly on MongoDB. For more information about,
+refer to the [Apache Drill SQL
+Reference](https://cwiki.apache.org/confluence/display/DRILL/SQL+Reference).
+
+**Example 1: View mongo.zipdb Dataset**
+
+ 0: jdbc:drill:zk=local> SELECT * FROM zipcodes LIMIT 10;
+ +------------+
+ | * |
+ +------------+
+ | { "city" : "AGAWAM" , "loc" : [ -72.622739 , 42.070206] , "pop" : 15338 , "state" : "MA"} |
+ | { "city" : "CUSHMAN" , "loc" : [ -72.51565 , 42.377017] , "pop" : 36963 , "state" : "MA"} |
+ | { "city" : "BARRE" , "loc" : [ -72.108354 , 42.409698] , "pop" : 4546 , "state" : "MA"} |
+ | { "city" : "BELCHERTOWN" , "loc" : [ -72.410953 , 42.275103] , "pop" : 10579 , "state" : "MA"} |
+ | { "city" : "BLANDFORD" , "loc" : [ -72.936114 , 42.182949] , "pop" : 1240 , "state" : "MA"} |
+ | { "city" : "BRIMFIELD" , "loc" : [ -72.188455 , 42.116543] , "pop" : 3706 , "state" : "MA"} |
+ | { "city" : "CHESTER" , "loc" : [ -72.988761 , 42.279421] , "pop" : 1688 , "state" : "MA"} |
+ | { "city" : "CHESTERFIELD" , "loc" : [ -72.833309 , 42.38167] , "pop" : 177 , "state" : "MA"} |
+ | { "city" : "CHICOPEE" , "loc" : [ -72.607962 , 42.162046] , "pop" : 23396 , "state" : "MA"} |
+ | { "city" : "CHICOPEE" , "loc" : [ -72.576142 , 42.176443] , "pop" : 31495 , "state" : "MA"} |
+
+**Example 2: Aggregation**
+
+ 0: jdbc:drill:zk=local> select state,city,avg(pop) +------------+------------+------------+
+ | state | city | EXPR$2 |
+ +------------+------------+------------+
+ | MA | AGAWAM | 15338.0 |
+ | MA | CUSHMAN | 36963.0 |
+ | MA | BARRE | 4546.0 |
+ | MA | BELCHERTOWN | 10579.0 |
+ | MA | BLANDFORD | 1240.0 |
+ | MA | BRIMFIELD | 3706.0 |
+ | MA | CHESTER | 1688.0 |
+ | MA | CHESTERFIELD | 177.0 |
+ | MA | CHICOPEE | 27445.5 |
+ | MA | WESTOVER AFB | 1764.0 |
+ +------------+------------+------------+
+
+**Example 3: Nested Data Column Array**
+
+ 0: jdbc:drill:zk=local> SELECT loc FROM zipcodes LIMIT 10;
+ +------------------------+
+ | loc |
+ +------------------------+
+ | [-72.622739,42.070206] |
+ | [-72.51565,42.377017] |
+ | [-72.108354,42.409698] |
+ | [-72.410953,42.275103] |
+ | [-72.936114,42.182949] |
+ | [-72.188455,42.116543] |
+ | [-72.988761,42.279421] |
+ | [-72.833309,42.38167] |
+ | [-72.607962,42.162046] |
+ | [-72.576142,42.176443] |
+ +------------------------+
+
+ 0: jdbc:drill:zk=local> SELECT loc[0] FROM zipcodes LIMIT 10;
+ +------------+
+ | EXPR$0 |
+ +------------+
+ | -72.622739 |
+ | -72.51565 |
+ | -72.108354 |
+ | -72.410953 |
+ | -72.936114 |
+ | -72.188455 |
+ | -72.988761 |
+ | -72.833309 |
+ | -72.607962 |
+ | -72.576142 |
+ +------------+
+
+## Using ODBC/JDBC Drivers
+
+You can leverage the power of Apache Drill to query MongoDB through standard
+BI tools, such as Tableau and SQuirreL.
+
+For information about Drill ODBC and JDBC drivers, refer to [Drill Interfaces](https://cwiki.apache.org/confluence/display/DRILL/Drill+Interfaces).
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/connect/003-mapr-db-plugin.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/connect/003-mapr-db-plugin.md b/_docs/drill-docs/connect/003-mapr-db-plugin.md
new file mode 100644
index 0000000..26d6a9c
--- /dev/null
+++ b/_docs/drill-docs/connect/003-mapr-db-plugin.md
@@ -0,0 +1,32 @@
+---
+title: "MapR-DB Plugin for Apache Drill"
+parent: "Connect to Data Sources"
+---
+Drill includes a `maprdb` format plugin for MapR-DB that is defined within the
+default `dfs` storage plugin instance when you install Drill from the `mapr-
+drill` package on a MapR node. The `maprdb` format plugin improves the
+estimated number of rows that Drill uses to plan a query. It also enables you
+to query tables like you would query files in a file system because MapR-DB
+and MapR-FS share the same namespace.
+
+You can query tables stored across multiple directories. You do not need to
+create a table mapping to a directory before you query a table in the
+directory. You can select from any table in any directory the same way you
+would select from files in MapR-FS, using the same syntax.
+
+Instead of including the name of a file, you include the table name in the
+query.
+
+**Example**
+
+ SELECT * FROM mfs.`/users/max/mytable`;
+
+Drill stores the `maprdb` format plugin in the `dfs` storage plugin instance,
+which you can view in the Drill Web UI. You can access the Web UI at
+[http://localhost:8047/storage](http://localhost:8047/storage). Click **Update** next to the `dfs` instance
+in the Web UI to view the configuration for the `dfs` instance.
+
+The following image shows a portion of the configuration with the `maprdb`
+format plugin for the` dfs` instance:
+
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/connect/workspaces/001-workspaces.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/connect/workspaces/001-workspaces.md b/_docs/drill-docs/connect/workspaces/001-workspaces.md
new file mode 100644
index 0000000..4ec1f1b
--- /dev/null
+++ b/_docs/drill-docs/connect/workspaces/001-workspaces.md
@@ -0,0 +1,82 @@
+---
+title: "Workspaces"
+parent: "Storage Plugin Registration"
+---
+When you register an instance of a file system data source, you can configure
+one or more workspaces for the instance. A workspace is a directory within the
+file system that you define. Drill searches the workspace to locate data when
+you run a query.
+
+Each workspace that you register defines a schema that you can connect to and
+query. Configuring workspaces is useful when you want to run multiple queries
+on files or tables in a specific directory. You cannot create workspaces for
+`hive` and `hbase` instances, though Hive databases show up as workspaces in
+Drill.
+
+The following example shows an instance of a file type storage plugin with a
+workspace named `json` configured to point Drill to the
+`/users/max/drill/json/` directory in the local file system `(dfs)`:
+
+ {
+ "type" : "file",
+ "enabled" : true,
+ "connection" : "file:///",
+ "workspaces" : {
+ "json" : {
+ "location" : "/users/max/drill/json/",
+ "writable" : false,
+ "storageformat" : json
+ }
+ },
+
+**Note:** The `connection` parameter in the configuration above is "`file:///`", connecting Drill to the local file system (`dfs`). To connect to a Hadoop or MapR file system the `connection` parameter would be "`hdfs:///" `or` "maprfs:///", `respectively.
+
+To query a file in the example `json` workspace, you can issue the `USE`
+command to tell Drill to use the `json` workspace configured in the `dfs`
+instance for each query that you issue:
+
+**Example**
+
+ USE dfs.json;
+ SELECT * FROM dfs.json.`donuts.json` WHERE type='frosted'
+
+If the `json `workspace did not exist, the query would have to include the
+full path to the `donuts.json` file:
+
+ SELECT * FROM dfs.`/users/max/drill/json/donuts.json` WHERE type='frosted';
+
+Using a workspace alleviates the need to repeatedly enter the directory path
+in subsequent queries on the directory.
+
+### Default Workspaces
+
+Each `file` and `hive` instance includes a `default` workspace. The `default`
+workspace points to the file system or to the Hive metastore. When you query
+files and tables in the` file` or `hive default` workspaces, you can omit the
+workspace name from the query.
+
+For example, you can issue a query on a Hive table in the `default workspace`
+using either of the following formats and get the the same results:
+
+**Example**
+
+ SELECT * FROM hive.customers LIMIT 10;
+ SELECT * FROM hive.`default`.customers LIMIT 10;
+
+**Note:** Default is a reserved word. You must enclose reserved words in back ticks.
+
+Because HBase instances do not have workspaces, you can use the following
+format to query a table in HBase:
+
+ SELECT * FROM hbase.customers LIMIT 10;
+
+After you register a data source as a storage plugin instance with Drill, and
+optionally configure workspaces, you can query the data source.
+
+Click any of the following links to learn how to register a data source as a
+storage plugin instance:
+
+ * [Registering a File System](/confluence/display/DRILL/Registering+a+File+System)
+ * [Registering HBase](/confluence/display/DRILL/Registering+HBase)
+ * [Registering Hive](/confluence/display/DRILL/Registering+Hive)
+ * [Drill Default Input Format](/confluence/display/DRILL/Drill+Default+Input+Format)
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/connect/workspaces/002-reg-fs.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/connect/workspaces/002-reg-fs.md b/_docs/drill-docs/connect/workspaces/002-reg-fs.md
new file mode 100644
index 0000000..4abb134
--- /dev/null
+++ b/_docs/drill-docs/connect/workspaces/002-reg-fs.md
@@ -0,0 +1,69 @@
+---
+title: "Registering a File System"
+parent: "Workspaces"
+---
+You can register a storage plugin instance that connects Drill to a local file
+system or a distributed file system registered in `core-site.xml`, such as S3
+or HDFS. When you register a storage plugin instance for a file system,
+provide a unique name for the instance, and identify the type as “`file`”. By
+default, Drill includes an instance named `dfs `that points to the local file
+system on your machine. You can update this configuration to point to a
+distributed file system or you can create a new instance to point to a
+distributed file system.
+
+To register a local or a distributed file system with Apache Drill, complete
+the following steps:
+
+ 1. Navigate to `[http://localhost:8047](http://localhost:8047/)`, and select the **Storage** tab.
+ 2. In the New Storage Plugin window, enter a unique name and then click **Create**.
+ 3. In the Configuration window, provide the following configuration information for the type of file system that you are configuring as a data source.
+ a. Local file system example:
+
+ {
+ "type": "file",
+ "enabled": true,
+ "connection": "file:///",
+ "workspaces": {
+ "root": {
+ "location": "/user/max/donuts",
+ "writable": false,
+ "storageformat": null
+ }
+ },
+ "formats" : {
+ "json" : {
+ "type" : "json"
+ }
+ }
+ }
+
+ b. Distributed file system example:
+
+ {
+ "type" : "file",
+ "enabled" : true,
+ "connection" : "hdfs://10.10.30.156:8020/",
+ "workspaces" : {
+ "root : {
+ "location" : "/user/root/drill",
+ "writable" : true,
+ "storageformat" : "null"
+ }
+ },
+ "formats" : {
+ "json" : {
+ "type" : "json"
+ }
+ }
+ }
+
+ To connect to a Hadoop file system, you must include the IP address of the
+name node and the port number.
+
+ 4. Click **Enable**.
+
+Once you have configured a storage plugin instance for the file system, you
+can issue Drill queries against it. For information about querying a file
+system, refer to [Querying a File System](https://cwiki.apache.org/confluence/
+display/DRILL/Connecting+to+Data+Sources#ConnectingtoDataSources-
+QueryingaFileSystem).
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/connect/workspaces/003-reg-hbase.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/connect/workspaces/003-reg-hbase.md b/_docs/drill-docs/connect/workspaces/003-reg-hbase.md
new file mode 100644
index 0000000..b97de4c
--- /dev/null
+++ b/_docs/drill-docs/connect/workspaces/003-reg-hbase.md
@@ -0,0 +1,34 @@
+---
+title: "Registering HBase"
+parent: "Workspaces"
+---
+Register a storage plugin instance and specify a zookeeper quorum to connect
+Drill to an HBase data source. When you register a storage plugin instance for
+an HBase data source, provide a unique name for the instance, and identify the
+type as “hbase” in the Drill Web UI.
+
+Currently, Drill only works with HBase version 0.94.
+
+To register HBase with Drill, complete the following steps:
+
+ 1. Navigate to [http://localhost:8047](http://localhost:8047/), and select the **Storage** tab
+ 2. In the disabled storage plugins section, click **Update** next to the `hbase` instance.
+ 3. In the Configuration window, specify the Zookeeper quorum and port.
+
+ **Example**
+
+ {
+ "type": "hbase",
+ "config": {
+ "hbase.zookeeper.quorum": "<zk1host,zk2host,zk3host> or <localhost>",
+ "hbase.zookeeper.property.clientPort": "2181"
+ },
+ "enabled": false
+ }
+
+ 4. Click **Enable**.
+
+Once you have configured a storage plugin instance for the HBase, you can
+issue Drill queries against it. For information about querying an HBase data
+source, refer to [Querying
+HBase](https://cwiki.apache.org/confluence/display/DRILL/Querying+HBase).
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/connect/workspaces/004-reg-hive.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/connect/workspaces/004-reg-hive.md b/_docs/drill-docs/connect/workspaces/004-reg-hive.md
new file mode 100644
index 0000000..c226fd6
--- /dev/null
+++ b/_docs/drill-docs/connect/workspaces/004-reg-hive.md
@@ -0,0 +1,99 @@
+---
+title: "Registering Hive"
+parent: "Workspaces"
+---
+You can register a storage plugin instance that connects Drill to a Hive data
+source that has a remote or embedded metastore service. When you register a
+storage plugin instance for a Hive data source, provide a unique name for the
+instance, and identify the type as “`hive`”. You must also provide the
+metastore connection information.
+
+Currently, Drill only works with Hive version 0.12. To access Hive tables
+using custom SerDes or InputFormat/OutputFormat, all nodes running Drillbits
+must have the SerDes or InputFormat/OutputFormat `JAR` files in the
+`<drill_installation_directory>/jars/3rdparty `folder.
+
+Hive Remote Metastore
+
+In this configuration, the Hive metastore runs as a separate service outside
+of Hive. Drill communicates with the Hive metastore through Thrift. The
+metastore service communicates with the Hive database over JDBC. Point Drill
+to the Hive metastore service address, and provide the connection parameters
+in the Drill Web UI to configure a connection to Drill.
+
+**Note:** Verify that the Hive metastore service is running before you register the Hive metastore.
+
+To register a remote Hive metastore with Drill, complete the following steps:
+
+ 1. Issue the following command to start the Hive metastore service on the system specified in the `hive.metastore.uris`:
+
+
+ hive --service metastore
+
+ 2. Navigate to [http://localhost:8047](http://localhost:8047/), and select the **Storage** tab.
+ 3. In the disabled storage plugins section, click **Update** next to the `hive` instance.
+
+ 4. In the configuration window, add the `Thrift URI` and port to `hive.metastore.uris`.
+
+ **Example**
+
+ {
+ "type": "hive",
+ "enabled": true,
+ "configProps": {
+ "hive.metastore.uris": "thrift://<localhost>:<port>",
+ "hive.metastore.sasl.enabled": "false"
+ }
+ }
+
+ 5. Click **Enable**.
+
+ 6. Verify that `HADOOP_CLASSPATH` is set in `drill-env.sh`. If you need to set the classpath, add the following line to `drill-env.sh`.
+
+Once you have configured a storage plugin instance for a Hive data source, you
+can issue Drill queries against it. For information about querying a Hive data
+source, refer to [Querying
+Hive](https://cwiki.apache.org/confluence/display/DRILL/Querying+Hive).
+
+### Hive Embedded Metastore
+
+In this configuration, the Hive metastore is embedded within the Drill
+process. Provide the metastore database configuration settings in the Drill
+Web UI. Before you register Hive, verify that the driver you use to connect to
+the Hive metastore is in the Drill classpath located in `/<drill installation
+dirctory>/lib/. `If the driver is not there, copy the driver to `/<drill
+installation directory>/lib` on the Drill node. For more information about
+storage types and configurations, refer to [AdminManual
+MetastoreAdmin](/confluence/display/Hive/AdminManual+MetastoreAdmin).
+
+To register an embedded Hive metastore with Drill, complete the following
+steps:
+
+ 1. Navigate to `[http://localhost:8047](http://localhost:8047/), `and select the **Storage** tab
+
+ 2. In the disabled storage plugins section, click **Update** next to `hive` instance.
+
+ 3. In the configuration window, add the database configuration settings.
+
+ **Example**
+
+ {
+ "type": "hive",
+ "enabled": true,
+ "configProps": {
+ "javax.jdo.option.ConnectionURL": "jdbc:<database>://<host:port>/<metastore database>;create=true",
+ "hive.metastore.warehouse.dir": "/tmp/drill_hive_wh",
+ "fs.default.name": "file:///",
+ }
+ }
+
+ 4. Click** Enable.**
+
+ 5. Verify that `HADOOP_CLASSPATH` is set in `drill-env.sh`. If you need to set the classpath, add the following line to `drill-env.sh`.
+
+ export HADOOP_CLASSPATH=/<directory path>/hadoop/hadoop-0.20.2
+
+Once you have configured a storage plugin instance for the Hive, you can issue
+Drill queries against it. For information about querying a Hive data source,
+refer to [Querying
+Hive](https://cwiki.apache.org/confluence/display/DRILL/Querying+Hive).
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/connect/workspaces/005-default-frmt.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/connect/workspaces/005-default-frmt.md b/_docs/drill-docs/connect/workspaces/005-default-frmt.md
new file mode 100644
index 0000000..335f415
--- /dev/null
+++ b/_docs/drill-docs/connect/workspaces/005-default-frmt.md
@@ -0,0 +1,61 @@
+---
+title: "Drill Default Input Format"
+parent: "Workspaces"
+---
+You can define a default input format to tell Drill what file type exists in a
+workspace within a file system. Drill determines the file type based on file
+extensions and magic numbers when searching a workspace.
+
+Magic numbers are file signatures that Drill uses to identify Parquet files.
+If Drill cannot identify the file type based on file extensions or magic
+numbers, the query fails. Defining a default input format can prevent queries
+from failing in situations where Drill cannot determine the file type.
+
+If you incorrectly define the file type in a workspace and Drill cannot
+determine the file type, the query fails. For example, if the directory for
+which you have defined a workspace contains JSON files and you defined the
+default input format as CSV, the query fails against the workspace.
+
+You can define one default input format per workspace. If you do not define a
+default input format, and Drill cannot detect the file format, the query
+fails. You can define a default input format for any of the file types that
+Drill supports. Currently, Drill supports the following types:
+
+ * CSV
+ * TSV
+ * PSV
+ * Parquet
+ * JSON
+
+## Defining a Default Input Format
+
+You define the default input format for a file system workspace through the
+Drill Web UI. You must have a defined workspace before you can define a
+default input format. For information about how to define a workspace, refer
+to [Workspaces](https://cwiki.apache.org/confluence/display/DRILL/Workspaces).
+
+To define a default input format for a workspace, complete the following
+steps:
+
+ 1. Navigate to the Drill Web UI at `<drill_node_ip_address>:8047`. The Drillbit process must be running on the node before you connect to the Drill Web UI.
+ 2. Select **Storage** in the toolbar.
+ 3. Click **Update** next to the file system for which you want to define a default input format for a workspace.
+ 4. In the Configuration area, locate the workspace for which you would like to define the default input format, and change the `defaultInputFormat` attribute to any of the supported file types.
+
+ **Example**
+
+ {
+ "type": "file",
+ "enabled": true,
+ "connection": "hdfs:///",
+ "workspaces": {
+ "root": {
+ "location": "/drill/testdata",
+ "writable": false,
+ "defaultInputFormat": csv
+ },
+ "local" : {
+ "location" : "/max/proddata",
+ "writable" : true,
+ "defaultInputFormat" : "json"
+ }
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/contribute/001-guidelines.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/contribute/001-guidelines.md b/_docs/drill-docs/contribute/001-guidelines.md
new file mode 100644
index 0000000..0b24541
--- /dev/null
+++ b/_docs/drill-docs/contribute/001-guidelines.md
@@ -0,0 +1,230 @@
+---
+title: "Apache Drill Contribution Guidelines"
+parent: "Contribute to Drill"
+---
+# How to Contribute to Apache Drill
+
+Disclaimer: These contribution guidelines are largely based on Apache Hive
+contribution guidelines.
+
+This page describes the mechanics of _how_ to contribute software to Apache
+Drill. For ideas about _what_ you might contribute, please see open tickets in
+[Jira](https://issues.apache.org/jira/browse/DRILL).
+
+These guidelines include the following topics:
+
+* Getting the source code
+ * Making Changes
+ * Coding Convention
+ * Formatter configuration
+ * Understanding Maven
+ * Creating a patch
+ * Applying a patch
+ * Where is a good place to start contributing?
+ * Contributing your work
+ * JIRA Guidelines
+ * See Also
+
+## Getting the source code
+
+First, you need the Drill source code.
+
+Get the source code on your local drive using [Git](https://git-wip-
+us.apache.org/repos/asf/incubator-drill.git). Most development is done on
+"master":
+
+ git clone https://git-wip-us.apache.org/repos/asf/drill.git
+
+## Making Changes
+
+Before you start, send a message to the [Drill developer mailing list](http
+://mail-archives.apache.org/mod_mbox/incubator-drill-dev/), or file a bug
+report in [JIRA](https://issues.apache.org/jira/browse/DRILL). Describe your
+proposed changes and check that they fit in with what others are doing and
+have planned for the project. Be patient, it may take folks a while to
+understand your requirements.
+
+Modify the source code and add some features using your favorite IDE.
+
+### Coding Convention
+
+Please take care about the following points
+
+ * All public classes and methods should have informative [Javadoc comments](http://www.oracle.com/technetwork/java/javase/documentation/index-137868.html).
+ * Do not use @author tags.
+ * Code should be formatted according to [Sun's conventions](http://www.oracle.com/technetwork/java/codeconv-138413.html), with one exception:
+ * Indent two (2) spaces per level, not four (4).
+ * Line length limit is 120 chars, instead of 80 chars.
+ * Contributions should not introduce new Checkstyle violations.
+ * Contributions should pass existing unit tests.
+ * New unit tests should be provided to demonstrate bugs and fixes. [JUnit](http://www.junit.org) 4.1 is our test framework:
+ * You must implement a class that contain test methods annotated with JUnit's 4.x @Test annotation and whose class name ends with `Test`.
+ * Define methods within your class whose names begin with `test`, and call JUnit's many assert methods to verify conditions; these methods will be executed when you run `mvn clean test`.
+
+### Formatter configuration
+
+Setting up IDE formatters is recommended and can be done by importing the
+following settings into your browser:
+
+IntelliJ IDEA formatter: [settings
+jar](/confluence/download/attachments/30757399/idea-
+settings.jar?version=1&modificationDate=1363022308000&api=v2)
+
+Eclipse: [formatter xml from HBase](https://issues.apache.org/jira/secure/atta
+chment/12474245/eclipse_formatter_apache.xml)
+
+### Understanding Maven
+
+Drill is built by Maven, a Java build tool.
+
+ * Good Maven tutorial: <http://maven.apache.org/guides/getting-started/maven-in-five-minutes.html>
+
+To build Drill, run
+
+ mvn clean install
+
+
+### Creating a patch
+
+Check to see what files you have modified:
+
+ git status
+
+Add any new files with:
+
+ git add .../MyNewClass.java
+ git add .../TestMyNewClass.java
+ git add .../XXXXXX.q
+ git add .../XXXXXX.q.out
+
+In order to create a patch, type (from the base directory of drill):
+
+ git format-patch origin/master --stdout > DRILL-1234.1.patch.txt
+
+This will report all modifications done on Drill sources on your local disk
+and save them into the _DRILL-1234.1.patch.txt_ file. Read the patch file.
+Make sure it includes ONLY the modifications required to fix a single issue.
+
+Please do not:
+
+ * reformat code unrelated to the bug being fixed: formatting changes should be separate patches/commits.
+ * comment out code that is now obsolete: just remove it.
+ * insert comments around each change, marking the change: folks can use subversion to figure out what's changed and by whom.
+ * make things public which are not required by end users.
+
+Please do:
+
+ * try to adhere to the coding style of files you edit;
+ * comment code whose function or rationale is not obvious;
+ * update documentation (e.g., _package.html_ files, this wiki, etc.)
+
+Updating a patch
+
+For patch updates, our convention is to number them like
+DRILL-1856.1.patch.txt, DRILL-1856.2.patch.txt, etc. And then click the
+"Submit Patch" button again when a new one is uploaded; this makes sure it
+gets back into the review queue. Appending '.txt' to the patch file name makes
+it easy to quickly view the contents of the patch in a web browser.
+
+### Applying a patch
+
+To apply a patch either you generated or found from JIRA, you can issue
+
+ git am < cool_patch.patch
+
+if you just want to check whether the patch applies you can run patch with
+--dry-run option.
+
+
+
+Review Process
+
+ * Use Hadoop's [code review checklist](http://wiki.apache.org/hadoop/CodeReviewChecklist) as a rough guide when doing reviews.
+ * In JIRA, use attach file to notify that you've submitted a patch for that issue.
+ * Create a Review Request in [Review Board](https://reviews.apache.org/r/). The review request's name should start with the JIRA issue number (e.g. DRILL-XX) and should be assigned to the "drill-git" group.
+ * If a committer requests changes, set the issue status to 'Resume Progress', then once you're ready, submit an updated patch with necessary fixes and then request another round of review with 'Submit Patch' again.
+ * Once your patch is accepted, be sure to upload a final version which grants rights to the ASF.
+
+## Where is a good place to start contributing?
+
+After getting the source code, building and running a few simple queries, one
+of the simplest places to start is to implement a DrillFunc.
+DrillFuncs is way that Drill express all scalar functions (UDF or system).
+First you can put together a JIRA for one of the DrillFunc's we don't yet have
+but should (referencing the capabilities of something like Postgres
+or SQL Server). Then try to implement one.
+
+One example DrillFunc:
+
+[https://github.com/apache/incubator-
+drill/blob/103072a619741d5e228fdb181501ec2f82e111a3/sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/ComparisonFunction
+s.java](https://github.com/apache/incubator-
+drill/blob/103072a619741d5e228fdb181501ec2f82e111a3/sandbox/prototype/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/ComparisonFunction
+s.java)
+
+Also one can visit the JIRA issues and implement one of those too. A list of
+functions which need to be implemented can be found
+[here](https://docs.google.com/spreadsheet/ccc?key=0AgAGbQ6asvQ-
+dDRrUUxVSVlMVXRtanRoWk9JcHgteUE&usp=sharing#gid=0) (WIP).
+
+More contribution ideas are located on the [Contribution Ideas](https://cwiki.
+apache.org/confluence/display/DRILL/Apache+Drill+Contribution+Ideas) page.
+
+## Contributing your work
+
+Finally, patches should be _attached_ to an issue report in
+[JIRA](http://issues.apache.org/jira/browse/DRILL) via the **Attach File**
+link on the issue's JIRA. Please add a comment that asks for a code review.
+Please note that the attachment should be granted license to ASF for inclusion
+in ASF works (as per the [Apache
+License](http://www.apache.org/licenses/LICENSE-2.0)).
+
+Folks should run `mvn clean install` before submitting a patch. Tests should
+all pass. If your patch involves performance optimizations, they should be
+validated by benchmarks that demonstrate an improvement.
+
+If your patch creates an incompatibility with the latest major release, then
+you must set the **Incompatible change** flag on the issue's JIRA 'and' fill
+in the **Release Note** field with an explanation of the impact of the
+incompatibility and the necessary steps users must take.
+
+If your patch implements a major feature or improvement, then you must fill in
+the **Release Note** field on the issue's JIRA with an explanation of the
+feature that will be comprehensible by the end user.
+
+A committer should evaluate the patch within a few days and either: commit it;
+or reject it with an explanation.
+
+Please be patient. Committers are busy people too. If no one responds to your
+patch after a few days, please make friendly reminders. Please incorporate
+other's suggestions into your patch if you think they're reasonable. Finally,
+remember that even a patch that is not committed is useful to the community.
+
+Should your patch receive a "-1" select the **Resume Progress** on the issue's
+JIRA, upload a new patch with necessary fixes, and then select the **Submit
+Patch** link again.
+
+Committers: for non-trivial changes, it is best to get another committer to
+review your patches before commit. Use **Submit Patch** link like other
+contributors, and then wait for a "+1" from another committer before
+committing. Please also try to frequently review things in the patch queue.
+
+## JIRA Guidelines
+
+Please comment on issues in JIRA, making their concerns known. Please also
+vote for issues that are a high priority for you.
+
+Please refrain from editing descriptions and comments if possible, as edits
+spam the mailing list and clutter JIRA's "All" display, which is otherwise
+very useful. Instead, preview descriptions and comments using the preview
+button (on the right) before posting them. Keep descriptions brief and save
+more elaborate proposals for comments, since descriptions are included in
+JIRA's automatically sent messages. If you change your mind, note this in a
+new comment, rather than editing an older comment. The issue should preserve
+this history of the discussion.
+
+## See Also
+
+ * [Apache contributor documentation](http://www.apache.org/dev/contributors.html)
+ * [Apache voting documentation](http://www.apache.org/foundation/voting.html)
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/contribute/002-ideas.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/contribute/002-ideas.md b/_docs/drill-docs/contribute/002-ideas.md
new file mode 100644
index 0000000..ef42adf
--- /dev/null
+++ b/_docs/drill-docs/contribute/002-ideas.md
@@ -0,0 +1,158 @@
+---
+title: "Apache Drill Contribution Guidelines"
+parent: "Contribute to Drill"
+---
+ * Fixing JIRAs
+ * SQL functions
+ * Support for new file format readers/writers
+ * Support for new data sources
+ * New query language parsers
+ * Application interfaces
+ * BI Tool testing
+ * General CLI improvements
+ * Eco system integrations
+ * MapReduce
+ * Hive views
+ * YARN
+ * Spark
+ * Hue
+ * Phoenix
+
+# Fixing JIRAs
+
+This is a good place to begin if you are new to Drill. Feel free to pick
+issues from the Drill JIRA list. When you pick an issue, assign it to
+yourself, inform the team, and start fixing it.
+
+For any questions, seek help from the team by sending email to [drill-
+dev@incubator.apache.org](mailto:drill-dev@incubator.apache.org).
+
+[https://issues.apache.org/jira/browse/DRILL/?selectedTab=com.atlassian.jira
+.jira-projects-plugin:summary-panel](https://issues.apache.org/jira/browse/DRILL/?selectedTab=com.atlassian.jira
+.jira-projects-plugin:summary-panel)
+
+# SQL functions
+
+One of the next simple places to start is to implement a DrillFunc.
DrillFuncs
+is way that Drill express all scalar functions (UDF or system).
First you can
+put together a JIRA for one of the DrillFunc's we don't yet have but should
+(referencing the capabilities of something like Postgres
or SQL Server or your
+own use case). Then try to implement one.
+
+One example DrillFunc:
+[https://github.com/apache/incubator-
+drill/blob/103072a619741d5e228fdb181501ec2f82e111a3/sandbox/prototype/exec
+/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/ComparisonFunction
+s.java](https://github.com/apache/incubator-
+drill/blob/103072a619741d5e228fdb181501ec2f82e111a3/sandbox/prototype/exec
+/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/ComparisonFunction
+s.java)** **
+
+**Additional ideas on functions that can be added to Drill SQL support**
+
+ * Madlib integration
+ * Machine learning functions
+ * Approximate aggregate functions (such as what is available in BlinkDB)
+
+# Support for new file format readers/writers
+
+Currently Drill supports text, JSON and Parquet file formats natively when
+interacting with file system. More readers/writers can be introduced by
+implementing custom storage plugins. Example formats include below.
+
+ * AVRO
+ * Sequence
+ * RC
+ * ORC
+ * Protobuf
+ * XML
+ * Thrift
+ * ....
+
+# Support for new data sources
+
+Implement custom storage plugins for the following non-Hadoop data sources:
+
+ * NoSQL databases (such as Mongo, Cassandra, Couch etc)
+ * Search engines (such as Solr, Lucidworks, Elastic Search etc)
+ * SQL databases (MySQL< PostGres etc)
+ * Generic JDBC/ODBC data sources
+ * HTTP URL
+ * \----
+
+# New query language parsers
+
+Drill exposes strongly typed JSON APIs for logical and physical plans (plan
+syntax at [https://docs.google.com/a/maprtech.com/document/d/1QTL8warUYS2KjldQ
+rGUse7zp8eA72VKtLOHwfXy6c7I/edit#heading=h.n9gdb1ek71hf](https://docs.google.com/a/maprtech.com/document/d/1QTL8warUYS2KjldQ
+rGUse7zp8eA72VKtLOHwfXy6c7I/edit#heading=h.n9gdb1ek71hf) ). Drill provides a
+SQL language parser today, but any language parser that can generate
+logical/physical plans can use Drill's power on the backend as the distributed
+low latency query execution engine along with its support for self-describing
+data and complex/multi-structured data.
+
+ * Pig parser : Use Pig as the language to query data from Drill. Great for existing Pig users.
+ * Hive parser : Use HiveQL as the language to query data from Drill. Great for existing Hive users.
+
+# Application interfaces
+
+Drill currently provides JDBC/ODBC drivers for the applications to interact
+along with a basic version of REST API and a C++ API. The following list
+provides a few possible application interface opportunities:
+
+ * Enhancements to REST APIs (<https://issues.apache.org/jira/browse/DRILL-77>)
+ * Expose Drill tables/views as REST APIs
+ * Language drivers for Drill (python etc)
+ * Thrift support
+ * ....
+
+## BI Tool testing
+
+Drill provides JDBC/ODBC drivers to connect to BI tools. We need to make sure
+Drill works with all major BI tools. Doing a quick sanity testing with your
+favorite BI tool is a good place to learn Drill and also uncover issues in
+being able to do so.
+
+# General CLI improvements
+
+Currently Drill uses SQLLine as the CLI. The goal of this effort is to improve
+the CLI experience by adding functionality such as execute statements from a
+file, output results to a file, display version information, and so on.
+
+# Eco system integrations
+
+## MapReduce
+
+Allow using result set from Drill queries as input to the Hadoop/MapReduce
+jobs.
+
+## Hive views
+
+Query data from existing Hive views using Drill queries. Drill needs to parse
+the HiveQL and translate them appropriately (into Drill's SQL or
+logical/physical plans) to execute the requests.
+
+## YARN
+
+[https://issues.apache.org/jira/browse/_DRILL_-1170](https://issues.apache.org
+/jira/browse/DRILL-1170)
+
+## Spark
+
+Provide ability to invoke Drill queries as part of Apache Spark programs. This
+gives ability for Spark developers/users to leverage Drill richness of the
+query layer , for data source access and as low latency execution engine.
+
+## Hue
+
+Hue is a GUI for users to interact with various Hadoop eco system components
+(such as Hive, Oozie, Pig, HBase, Impala ...). The goal of this project is to
+expose Drill as an application inside Hue so users can explore Drill metadata
+and do SQL queries.
+
+## Phoenix
+
+Phoenix provides a low latency query layer on HBase for operational
+applications. The goal of this effort is to explore opportunities for
+integrating Phoenix with Drill.
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/datasets/001-aol.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/datasets/001-aol.md b/_docs/drill-docs/datasets/001-aol.md
new file mode 100644
index 0000000..472f52f
--- /dev/null
+++ b/_docs/drill-docs/datasets/001-aol.md
@@ -0,0 +1,47 @@
+---
+title: "AOL Search"
+parent: "Sample Datasets"
+---
+## Quick Stats
+
+The [AOL Search dataset](http://en.wikipedia.org/wiki/AOL_search_data_leak) is
+a collection of real query log data that is based on real users.
+
+## The Data Source
+
+The dataset consists of 20M Web queries from 650k users over a period of three
+months, 440MB in total and available [for
+download](http://zola.di.unipi.it/smalltext/datasets.html). The format used in
+the dataset is:
+
+ AnonID, Query, QueryTime, ItemRank, ClickURL
+
+... with:
+
+ * AnonID, an anonymous user ID number.
+ * Query, the query issued by the user, case shifted with most punctuation removed.
+ * QueryTime, the time at which the query was submitted for search.
+ * ItemRank, if the user clicked on a search result, the rank of the item on which they clicked is listed.
+ * [ClickURL](http://www.dietkart.com/), if the user clicked on a search result, the domain portion of the URL in the clicked result is listed.
+
+Each line in the data represents one of two types of events
+
+ * A query that was NOT followed by the user clicking on a result item.
+ * A click through on an item in the result list returned from a query.
+
+In the first case (query only) there is data in only the first three columns,
+in the second case (click through), there is data in all five columns. For
+click through events, the query that preceded the click through is included.
+Note that if a user clicked on more than one result in the list returned from
+a single query, there will be TWO lines in the data to represent the two
+events.
+
+## The Queries
+
+Interesting queries, for example
+
+ * Users querying for topic X
+ * Users that click on the first (second, third) ranked item
+ * TOP 10 domains searched
+ * TOP 10 domains clicked at
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/datasets/002-enron.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/datasets/002-enron.md b/_docs/drill-docs/datasets/002-enron.md
new file mode 100644
index 0000000..2ddbef6
--- /dev/null
+++ b/_docs/drill-docs/datasets/002-enron.md
@@ -0,0 +1,21 @@
+---
+title: "Enron Emails"
+parent: "Sample Datasets"
+---
+## Quick Stats
+
+The [Enron Email dataset](http://www.cs.cmu.edu/~enron/) contains data from
+about 150 users, mostly senior management of Enron.
+
+## The Data Source
+
+Totalling some 500,000 messages, the [raw
+data](http://www.cs.cmu.edu/~enron/enron_mail_20110402.tgz) (2009 version of
+the dataset; ~423MB) is available for download as well as a [MySQL
+dump](ftp://ftp.isi.edu/sims/philpot/data/enron-mysqldump.sql.gz) (~177MB).
+
+## The Queries
+
+Interesting queries, for example
+
+ * Via [Query Dataset for Email Search](https://dbappserv.cis.upenn.edu/spell/)
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/datasets/003-wikipedia.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/datasets/003-wikipedia.md b/_docs/drill-docs/datasets/003-wikipedia.md
new file mode 100644
index 0000000..99e6e24
--- /dev/null
+++ b/_docs/drill-docs/datasets/003-wikipedia.md
@@ -0,0 +1,105 @@
+---
+title: "Wikipedia Edit History"
+parent: "Sample Datasets"
+---
+# Quick Stats
+
+The Wikipedia Edit History is a public dump of the website made available by
+the wikipedia foundation. You can find details
+[here](http://en.wikipedia.org/wiki/Wikipedia:Database_download). The dumps
+are made available as SQL or XML dumps. You can find the entire schema drawn
+together in this great [diagram](http://upload.wikimedia.org/wikipedia/commons
+/thumb/4/42/MediaWiki_1.20_%2844edaa2%29_database_schema.svg/2193px-
+MediaWiki_1.20_%2844edaa2%29_database_schema.svg.png).
+
+# Approach
+
+The _main_ distribution files are:
+
+ * Current Pages: As of January 2013 this SQL dump was 9.0GB in its compressed format.
+ * Complere Archive: This is what we actually want, but at a size of multiple terrabytes, clearly exceeds the storage available at home.
+
+To have some real historic data, it is recommended to download a _Special
+Export_ use this
+[link](http://en.wikipedia.org/w/index.php?title=Special:Export). Using this
+tool you generate a category specific XML dump and configure various export
+options. There are some limits like a maximum of 1000 revisions per export,
+but otherwise this should work out just fine.
+
+
+
+The entities used in the query use cases.
+
+# Use Cases
+
+## Select Change Volume Based on Time
+
+**Query**
+
+ select rev.::parent.title, rev.::parent.id, sum(rev.text.bytes)
+ from mediawiki.page.revision as rev
+ where rev.timestamp.between(?, ?)
+ group by rev.::parent;
+
+_Explanation_: This is my attempt in mixing records and structures. The `from`
+statement refers to `mediawiki` as a record type / row, but also mixes in
+structural information, i.e. `page.revision`, internal to the record. The
+query now uses `page.revision` as base to all other statements, in this case
+the `select`, `where` and the `goup by`. The `where` statement again uses a
+JSON like expression to state, that the timestamp must be between two values,
+paramaeters are written as question marks, similar to JDBC. The `group by`
+statement instructs the query to aggregate results based on the parent of a
+`revision`, in this case a `page`. The `::parent` syntax is borrowed from
+XPath. As we are aggregating on `page` it is safe to select the `title` and
+`id` from the element in the `select`. We also use an aggregation function to
+add the number of bytes changed in the given time frame, this should be self
+explanatory.
+
+_Discussion_:
+
+ * I am not very satisfied using the `::` syntax, as it is _ugly_. We probably wont need that many axis specifiers, e.g. we dont need any attribute specifiers, but for now, I could not think of anything better,
+ * Using an `as` expression in the `from` statement is optional, you would simply have to replace all references to `rev` with `revision`.
+ * I am not sure if this is desired, but you cannot see on first glance, where the _hierarchical_ stuff starts. This may be confusing to a RDBMS purist, at least it was for me at the beginning. But now I think this strikes the right mix between verbosity and elegance.
+ * I assume we would need some good indexing, but this should be achievable. We would need to translate the relative index `rev.timestamp` to an record absolute index `$.mediawiki.page.revision.timestamp` . Unclear to me now is whether the index would point to the record, or would it point to some kind of record substructure?
+
+## Select Change Volume Aggregated on Time
+
+**Query**
+
+ select rev.::parent.title, rev.::parent.id, sum(rev.text.bytes), rev.timestamp.monthYear()
+ from mediawiki.page.revision as rev
+ where rev.timestamp.between(?, ?)
+ group by rev.::parent, rev.timestamp.monthYear()
+ order by rev.::parent.id, rev.timestamp.monthYear();
+
+_Explanation_: This is refinement of the previous query. In this case we are
+again returning a flat list, but are using an additional scalar result and
+`group` statement. In the previous example we were returning one result per
+found page, now we are returning one result per page and month of changes.
+`Order by` is nothing special, in this case.
+
+_Discussion_:
+
+ * I always considered mySQL confusing using implicit group by statements, as I prefer fail fast mechanisms. Hence I would opt for explicit `group by` operators.
+ * I would not provide implicit nodes into the records, i.e. if you want some attribute of a timestamp, call a function and not expect an automatically added element. So we want `rev.timestamp.monthYear()` and not `rev.timestamp.monthYear`. This may be quite confusing, especially if we have heterogenous record structures. We might even go ahead and support namespaces for custom, experimental features like `rev.timestamp.custom.maya:doomsDay()`.
+
+## Select Change Volume Based on Contributor
+
+**Query**
+
+ select ctrbr.username, ctbr.ip, ctbr.userid, sum(ctbr::parent.bytes) as bytesContributed
+ from mediawiki.page..contributor as ctbr
+ group by ctbr.canonize()
+ order by bytesContributed;
+
+_Explanation_: This query looks quite similar to the previous queries, but I
+added this one nonetheless, as it hints on an aggregation which may spawn
+multiple records. The previous examples were based on pages, which are unique
+to a record, where as the contributor may appear many times in many different
+records.
+
+_Discussion_:
+
+ * I have added the `..` operator in this example. Besides of being syntactic sugar, it also allows us to search for `revision` and `upload` which are both children of `page` and may both have a `contributor`. The more RBMS like alternative would be a `union`, but this was not natural enough.
+ * I am sure the `ctbr.canonize()` will cause lots of discussions :-). The thing is, that a contributor may repeat itself in many different records, and we dont really have an id. If you look at the wikimedia XSD, all three attributes are optional, and the data says the same, so we cannot just simply say `ctbr.userid`. Hence the canonize function should create a scalar value containing all available information of the node in a canonical form.
+ * Last but not least, I always hated, that mySQL would not be able to reuse column definitions from the `select` statement in the `order` statements. So I added on my wishlist, that the `bytesContributed` definition is reusable.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/design/001-plan.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/design/001-plan.md b/_docs/drill-docs/design/001-plan.md
new file mode 100644
index 0000000..66147cb
--- /dev/null
+++ b/_docs/drill-docs/design/001-plan.md
@@ -0,0 +1,25 @@
+---
+title: "Drill Plan Syntax"
+parent: "Design Docs"
+---
+### Whats the plan?
+
+This section is about the end-to-end plan flow for Drill. The incoming query
+to Drill can be a SQL 2003 query/DrQL or MongoQL. The query is converted to a
+_Logical Plan_ that is a Drill's internal representation of the query
+(language-agnostic). Drill then uses its optimization rules over the Logical
+Plan to optimize it for best performance and crafts out a _Physical Plan_. The
+Physical Plan is the actual plan the Drill then executes for the final data
+processing. Below is a diagram to illustrate the flow:
+
+
+
+**The Logical Plan** describes the abstract data flow of a language independent query i.e. it would be a representation of the input query which would not be dependent on the actual input query language. It generally tries to work with primitive operations without focus on optimization. This makes it more verbose than traditional query languages. This is to allow a substantial level of flexibility in defining higher-level query language features. It would be forwarded to the optimizer to get a physical plan.
+
+**The Physical Plan** is often called the execution plan, since it is the input to the execution engine. Its a description of the physical operations the execution engine will undertake to get the desired result. It is the output of the query planner and is a transformation of the logical plan after applying the optimization rules.
+
+Typically, the physical and execution plans will be represented using the same
+JSON format as the logical plan.
+
+**Detailed document**: Here is a document that explains the Drill logical & physical plans in full detail. [Drill detailed plan syntax document](https://docs.google.com/document/d/1QTL8warUYS2KjldQrGUse7zp8eA72VKtLOHwfXy6c7I/edit).
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/design/002-rpc.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/design/002-rpc.md b/_docs/drill-docs/design/002-rpc.md
new file mode 100644
index 0000000..05cb1d6
--- /dev/null
+++ b/_docs/drill-docs/design/002-rpc.md
@@ -0,0 +1,19 @@
+---
+title: "RPC Overview"
+parent: "Design Docs"
+---
+Drill leverages the Netty 4 project as an RPC underlayment. From there, we
+built a simple protobuf based communication layer optimized to minimize the
+requirement for on heap data transformations. Both client and server utilize
+the CompleteRpcMessage protobuf envelope to communicate requests, responses
+and errors. The communication model is that each endpoint sends a stream of
+CompleteRpcMessages to its peer. The CompleteRpcMessage is prefixed by a
+protobuf encoded length.
+
+CompleteRpcMessage is broken into three key components: RpcHeader, Protobuf
+Body (bytes), RawBody (bytes).
+
+RpcHeader has the following fields:
+
+Drillbits communicate through the BitCom intermediary. BitCom manages...
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/design/003-query-stages.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/design/003-query-stages.md b/_docs/drill-docs/design/003-query-stages.md
new file mode 100644
index 0000000..5c54249
--- /dev/null
+++ b/_docs/drill-docs/design/003-query-stages.md
@@ -0,0 +1,42 @@
+---
+title: "Query Stages"
+parent: "Design Docs"
+---
+## Overview
+
+Apache Drill is a system for interactive analysis of large-scale datasets. It
+was designed to allow users to query across multiple large big data systems
+using traditional query technologies such as SQL. It is built as a flexible
+framework to support a wide variety of data operations, query languages and
+storage engines.
+
+## Query Parsing
+
+A Drillbit is capable of parsing a provided query into a logical plan. In
+theory, Drill is capable of parsing a large range of query languages. At
+launch, this will likely be restricted to an enhanced SQL2003 language.
+
+## Physical Planning
+
+Once a query is parsed into a logical plan, a Drillbit will then translate the
+plan into a physical plan. The physical plan will then be optimized for
+performance. Since plan optimization can be computationally intensive, a
+distributed in-memory cache will provide LRU retrieval of previously generated
+optimized plans to speed query execution.
+
+## Execution Planning
+
+Once a physical plan is generated, the physical plan is then rendered into a
+set of detailed executional plan fragments (EPFs). This rendering is based on
+available resources, cluster load, query priority and detailed information
+about data distribution. In the case of large clusters, a subset of nodes will
+be responsible for rendering the EPFs. Shared state will be managed through
+the use of a distributed in-memory cache.
+
+## Execution Operation
+
+Query execution starts with each Drillbit being provided with one or more EPFs
+associated with query execution. A portion of these EPFs may be identified as
+initial EPFs and thus they are executed immediately. Other EPFs are executed
+as data flows into them.
+
[4/6] drill git commit: Added Drill docs
Posted by ts...@apache.org.
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/design/004-research.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/design/004-research.md b/_docs/drill-docs/design/004-research.md
new file mode 100644
index 0000000..77be828
--- /dev/null
+++ b/_docs/drill-docs/design/004-research.md
@@ -0,0 +1,48 @@
+---
+title: "Useful Research"
+parent: "Design Docs"
+---
+## Drill itself
+
+ * Apache Proposal: <http://wiki.apache.org/incubator/DrillProposal>
+ * Mailing List Archive: <http://mail-archives.apache.org/mod_mbox/incubator-drill-dev/>
+ * DrQL ANTLR grammar: <https://gist.github.com/3483314>
+ * Apache Drill, Architecture outlines: <http://www.slideshare.net/jasonfrantz/drill-architecture-20120913>
+
+## Background info
+
+ * Dremel Paper: <http://research.google.com/pubs/pub36632.html>
+ * Dremel Presentation: <http://www.slideshare.net/robertlz/dremel-interactive-analysis-of-webscale-datasets>
+ * Query Language: <http://developers.google.com/bigquery/docs/query-reference>
+ * Protobuf: <http://developers.google.com/protocol-buffers/docs/proto>
+ * Dryad: <http://research.microsoft.com/en-us/projects/dryad/>
+ * SQLServer Query Plan: <http://msdn.microsoft.com/en-us/library/ms191158.aspx>
+ * CStore: <http://db.csail.mit.edu/projects/cstore/>
+ * Vertica (commercial evolution of C-Store): <http://vldb.org/pvldb/vol5/p1790_andrewlamb_vldb2012.pdf>
+ * <http://pdf.aminer.org/000/094/728/database_cracking.pdf>
+ * <http://homepages.cwi.nl/~idreos/NoDBsigmod2012.pdf>
+ * <http://db.csail.mit.edu/projects/cstore/abadiicde2007.pdf>
+ * Hive Architecture: <https://cwiki.apache.org/confluence/display/Hive/Design#Design-HiveArchitecture>
+ * Fast Response in an unreliable world: <http://research.google.com/people/jeff/latency.html>
+ * Column-Oriented Database Systems: <http://www.vldb.org/pvldb/2/vldb09-tutorial6.pdf> (SLIDES: <http://phdopen.mimuw.edu.pl/lato10/boncz_mimuw.pdf>)
+
+## OpenDremel
+
+ * OpenDremel site: <http://code.google.com/p/dremel/>
+ * Design Proposal for Drill: <http://www.slideshare.net/CamuelGilyadov/apache-drill-14071739>
+
+## Dazo (second generation OpenDremel)
+
+ * Dazo repos: <https://github.com/Dazo-org>
+ * ZeroVM (multi-tenant executor): <http://zerovm.org/>
+ * ZeroVM elaboration: <http://news.ycombinator.com/item?id=3746222>
+
+## Rob Grzywinski Dremel adventures
+
+ * <https://github.com/rgrzywinski/field-stripe/>
+
+## Code generation / Physical plan generation
+
+ * <http://www.vldb.org/pvldb/vol4/p539-neumann.pdf> (SLIDES: <http://www.vldb.org/2011/files/slides/research9/rSession9-3.pdf>)
+ * <http://www.vldb.org/pvldb/2/vldb09-327.pdf> (SLIDES: <http://www.slideserve.com/cher/simd-scan-ultra-fast-in-memory-table-scan-using-on-chip-vector-processing-units>)
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/design/005-value.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/design/005-value.md b/_docs/drill-docs/design/005-value.md
new file mode 100644
index 0000000..0d19a96
--- /dev/null
+++ b/_docs/drill-docs/design/005-value.md
@@ -0,0 +1,191 @@
+---
+title: "Value Vectors"
+parent: "Design Docs"
+---
+This document defines the data structures required for passing sequences of
+columnar data between [Operators](https://docs.google.com/a/maprtech.com/docum
+ent/d/1zaxkcrK9mYyfpGwX1kAV80z0PCi8abefL45zOzb97dI/edit#bookmark=id.iip15ful18
+mm).
+
+# Goals
+
+#### Support Operators Written in Multiple Language
+
+ValueVectors should support operators written in C/C++/Assembly. To support
+this, the underlying ByteBuffer will not require modification when passed
+through the JNI interface. The ValueVector will be considered immutable once
+constructed. Endianness has not yet been considered.
+
+#### Access
+
+Reading a random element from a ValueVector must be a constant time operation.
+To accomodate, elements are identified by their offset from the start of the
+buffer. Repeated, nullable and variable width ValueVectors utilize in an
+additional fixed width value vector to index each element. Write access is not
+supported once the ValueVector has been constructed by the RecordBatch.
+
+#### Efficient Subsets of Value Vectors
+
+When an operator returns a subset of values from a ValueVector, it should
+reuse the original ValueVector. To accomplish this, a level of indirection is
+introduced to skip over certain values in the vector. This level of
+indirection is a sequence of offsets which reference an offset in the original
+ValueVector and the count of subsequent values which are to be included in the
+subset.
+
+#### Pooled Allocation
+
+ValueVectors utilize one or more buffers under the covers. These buffers will
+be drawn from a pool. Value vectors are themselves created and destroyed as a
+schema changes during the course of record iteration.
+
+#### Homogenous Value Types
+
+Each value in a Value Vector is of the same type. The [Record Batch](https://d
+ocs.google.com/a/maprtech.com/document/d/1zaxkcrK9mYyfpGwX1kAV80z0PCi8abefL45z
+Ozb97dI/edit#bookmark=kix.s2xuoqnr8obe) implementation is responsible for
+creating a new Value Vector any time there is a change in schema.
+
+# Definitions
+
+Data Types
+
+The canonical source for value type definitions is the [Drill
+Datatypes](http://bit.ly/15JO9bC) document. The individual types are listed
+under the ‘Basic Data Types’ tab, while the value vector types can be found
+under the ‘Value Vectors’ tab.
+
+Operators
+
+An operator is responsible for transforming a stream of fields. It operates on
+Record Batches or constant values.
+
+Record Batch
+
+A set of field values for some range of records. The batch may be composed of
+Value Vectors, in which case each batch consists of exactly one schema.
+
+Value Vector
+
+The value vector is comprised of one or more contiguous buffers; one which
+stores a sequence of values, and zero or more which store any metadata
+associated with the ValueVector.
+
+# Data Structure
+
+A ValueVector stores values in a ByteBuf, which is a contiguous region of
+memory. Additional levels of indirection are used to support variable value
+widths, nullable values, repeated values and selection vectors. These levels
+of indirection are primarily lookup tables which consist of one or more fixed
+width ValueVectors which may be combined (e.g. for nullable, variable width
+values). A fixed width ValueVector of non-nullable, non-repeatable values does
+not require an indirect lookup; elements can be accessed directly by
+multiplying position by stride.
+
+Fixed Width Values
+
+Fixed width ValueVectors simply contain a packed sequence of values. Random
+access is supported by accessing element n at ByteBuf[0] + Index * Stride,
+where Index is 0-based. The following illustrates the underlying buffer of
+INT4 values [1 .. 6]:
+
+
+<!--https://lh5.googleusercontent.com/iobQUgeF4dyrWFeqVfhIBZKbkjrLk5sBJqYhWdzm
+IyMmmcX1pzZaeQiKZ5OzYeafxcY5IZHXDKuG_JkPwJrjxeLJITpXBbn7r5ep1V07a3JBQC0cJg4qKf
+VhzPZ0PDeh-->
+
+Nullable Values
+
+Nullable values are represented by a vector of bit values. Each bit in the
+vector corresponds to an element in the ValueVector. If the bit is not set,
+the value is NULL. Otherwise the value is retrieved from the underlying
+buffer. The following illustrates a NullableValueVector of INT4 values 2, 3
+and 6:
+
+
+
+<!---->
+
+
+
+#### Repeated Values
+
+A repeated ValueVector is used for elements which can contain multiple values
+(e.g. a JSON array). A table of offset and count pairs is used to represent
+each repeated element in the ValueVector. A count of zero means the element
+has no values (note the offset field is unused in this case). The following
+illustrates three fields; one with two values, one with no values, and one
+with a single value:
+
+
+<!---->
+
+ValueVector Representation of the equivalent JSON:
+
+x:[1, 2]
+
+x:[ ]
+
+x:[3]
+
+Variable Width Values
+
+Variable width values are stored contiguously in a ByteBuf. Each element is
+represented by an entry in a fixed width ValueVector of offsets. The length of
+an entry is deduced by subtracting the offset of the following field. Because
+of this, the offset table will always contain one more entry than total
+elements, with the last entry pointing to the end of the buffer.
+
+
+
+<!---->
+
+Repeated Map Vectors
+
+A repeated map vector contains one or more maps (akin to an array of objects
+in JSON). The values of each field in the map are stored contiguously within a
+ByteBuf. To access a specific record, a lookup table of count and offset pairs
+is used. This lookup table points to the first repeated field in each column,
+while the count indicates the maximum number of elements for the column. The
+following example illustrates a RepeatedMap with two records; one with two
+objects, and one with a single object:
+
+
+<!---->
+
+ValueVector representation of the equivalent JSON:
+
+x: [ {name:’Sam’, age:1}, {name:’Max’, age:2} ]
+
+x: [ {name:’Joe’, age:3} ]
+
+Selection Vectors
+
+A Selection Vector represents a subset of a ValueVector. It is implemented
+with a list of offsets which identify each element in the ValueVector to be
+included in the SelectionVector. In the case of a fixed width ValueVector, the
+offsets reference the underlying ByteBuf. In the case of a nullable, repeated
+or variable width ValueVector, the offset references the corresponding lookup
+table. The following illustrates a SelectionVector of INT4 (fixed width)
+values 2, 3 and 5 from the original vector of [1 .. 6]:
+
+
+<!---->
+
+The following illustrates the same ValueVector with nullable fields:
+
+
+<!---->
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/dev-custom-fcn/001-dev-simple.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/dev-custom-fcn/001-dev-simple.md b/_docs/drill-docs/dev-custom-fcn/001-dev-simple.md
new file mode 100644
index 0000000..47be7d9
--- /dev/null
+++ b/_docs/drill-docs/dev-custom-fcn/001-dev-simple.md
@@ -0,0 +1,51 @@
+---
+title: "Develop a Simple Function"
+parent: "Develop Custom Functions"
+---
+Create a class within a Java package that implements Drill’s simple interface
+into the program, and include the required information for the function type.
+Your function must include data types that Drill supports, such as int or
+BigInt. For a list of supported data types, refer to the Apache Drill SQL
+Reference.
+
+Complete the following steps to develop a simple function using Drill’s simple
+function interface:
+
+ 1. Create a Maven project and add the following dependency:
+
+ <dependency>
+ <groupId>org.apache.drill.exec</groupId>
+ <artifactId>drill-java-exec</artifactId>
+ <version>1.0.0-m2-incubating-SNAPSHOT</version>
+ </dependency>
+
+ 2. Create a class that implements the `DrillSimpleFunc` interface and identify the scope as `FunctionScope.SIMPLE`.
+
+ **Example**
+
+ @FunctionTemplate(name = "myaddints", scope = FunctionScope.SIMPLE, nulls = NullHandling.NULL_IF_NULL)
+ public static class IntIntAdd implements DrillSimpleFunc {
+
+ 3. Provide the variables used in the code in the `Param` and `Output` bit holders.
+
+ **Example**
+
+ @Param IntHolder in1;
+ @Param IntHolder in2;
+ @Output IntHolder out;
+
+ 4. Add the code that performs operations for the function in the `eval()` method.
+
+ **Example**
+
+ public void setup(RecordBatch b) {
+ }
+ public void eval() {
+ out.value = (int) (in1.value + in2.value);
+ }
+
+ 5. Use the maven-source-plugin to compile the sources and classes JAR files. Verify that an empty `drill-module.conf` is included in the resources folder of the JARs.
+Drill searches this module during classpath scanning. If the file is not
+included in the resources folder, you can add it to the JAR file or add it to
+`etc/drill/conf`.
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/dev-custom-fcn/002-dev-aggregate.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/dev-custom-fcn/002-dev-aggregate.md b/_docs/drill-docs/dev-custom-fcn/002-dev-aggregate.md
new file mode 100644
index 0000000..fe6f406
--- /dev/null
+++ b/_docs/drill-docs/dev-custom-fcn/002-dev-aggregate.md
@@ -0,0 +1,59 @@
+---
+title: "Developing an Aggregate Function"
+parent: "Develop Custom Functions"
+---
+Create a class within a Java package that implements Drill’s aggregate
+interface into the program. Include the required information for the function.
+Your function must include data types that Drill supports, such as int or
+BigInt. For a list of supported data types, refer to the Drill SQL Reference.
+
+Complete the following steps to create an aggregate function:
+
+ 1. Create a Maven project and add the following dependency:
+
+ <dependency>
+ <groupId>org.apache.drill.exec</groupId>
+ <artifactId>drill-java-exec</artifactId>
+ <version>1.0.0-m2-incubating-SNAPSHOT</version>
+ </dependency>
+
+ 2. Create a class that implements the `DrillAggFunc` interface and identify the scope as `FunctionTemplate.FunctionScope.POINT_AGGREGATE`.
+
+ **Example**
+
+ @FunctionTemplate(name = "count", scope = FunctionTemplate.FunctionScope.POINT_AGGREGATE)
+ public static class BitCount implements DrillAggFunc{
+
+ 3. Provide the variables used in the code in the `Param, Workspace, `and `Output` bit holders.
+
+ **Example**
+
+ @Param BitHolder in;
+ @Workspace BitHolder value;
+ @Output BitHolder out;
+
+ 4. Include the `setup(), add(), output(),` and `reset()` methods.
+
+ **Example**
+ public void setup(RecordBatch b) {
+ value = new BitHolder();
+ value.value = 0;
+ }
+
+ @Override
+ public void add() {
+ value.value++;
+ }
+ @Override
+ public void output() {
+ out.value = value.value;
+ }
+ @Override
+ public void reset() {
+
+ value.value = 0;
+
+ 5. Use the maven-source-plugin to compile the sources and classes JAR files. Verify that an empty `drill-module.conf` is included in the resources folder of the JARs.
+Drill searches this module during classpath scanning. If the file is not
+included in the resources folder, you can add it to the JAR file or add it to
+`etc/drill/conf`.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/dev-custom-fcn/003-add-custom.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/dev-custom-fcn/003-add-custom.md b/_docs/drill-docs/dev-custom-fcn/003-add-custom.md
new file mode 100644
index 0000000..7efcdce
--- /dev/null
+++ b/_docs/drill-docs/dev-custom-fcn/003-add-custom.md
@@ -0,0 +1,28 @@
+---
+title: "Adding Custom Functions to Drill"
+parent: "Develop Custom Functions"
+---
+After you develop your custom function and generate the sources and classes
+JAR files, add both JAR files to the Drill classpath, and include the name of
+the package that contains the classes to the main Drill configuration file.
+Restart the Drillbit on each node to refresh the configuration.
+
+To add a custom function to Drill, complete the following steps:
+
+ 1. Add the sources JAR file and the classes JAR file for the custom function to the Drill classpath on all nodes running a Drillbit. To add the JAR files, copy them to `<drill installation directory>/jars/3rdparty`.
+ 2. On all nodes running a Drillbit, add the name of the package that contains the classes to the main Drill configuration file in the following location:
+
+ <drill installation directory>/conf/drill-override.conf
+
+ To add the package, add the package name to
+ `drill.logical.function.package+=`. Separate package names with a comma.
+
+ **Example**
+
+ drill.logical.function.package+= [“org.apache.drill.exec.expr.fn.impl","org.apache.drill.udfs”]
+
+ 3. On each Drill node in the cluster, navigate to the Drill installation directory, and issue the following command to restart the Drillbit:
+
+ <drill installation directory>/bin/drillbit.sh restart
+
+ Now you can issue queries with your custom functions to Drill.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/dev-custom-fcn/004-use-custom.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/dev-custom-fcn/004-use-custom.md b/_docs/drill-docs/dev-custom-fcn/004-use-custom.md
new file mode 100644
index 0000000..6a0245a
--- /dev/null
+++ b/_docs/drill-docs/dev-custom-fcn/004-use-custom.md
@@ -0,0 +1,55 @@
+---
+title: "Using Custom Functions in Queries"
+parent: "Develop Custom Functions"
+---
+When you issue a query with a custom function to Drill, Drill searches the
+classpath for the function that matches the request in the query. Once Drill
+locates the function for the request, Drill processes the query and applies
+the function during processing.
+
+Your Drill installation includes sample files in the Drill classpath. One
+sample file, `employee.json`, contains some fictitious employee data that you
+can query with a custom function.
+
+## Simple Function Example
+
+This example uses the `myaddints` simple function in a query on the
+`employee.json` file.
+
+If you issue the following query to Drill, you can see all of the employee
+data within the `employee.json` file:
+
+ 0: jdbc:drill:zk=local> SELECT * FROM cp.`employee.json`;
+
+The query returns the following results:
+
+ | employee_id | full_name | first_name | last_name | position_id | position_title | store_id | department_id | birth_da |
+ +-------------+------------+------------+------------+-------------+----------------+------------+---------------+----------+-----------
+ | 1101 | Steve Eurich | Steve | Eurich | 16 | Store Temporary Checker | 12 | 16 |
+ | 1102 | Mary Pierson | Mary | Pierson | 16 | Store Temporary Checker | 12 | 16 |
+ | 1103 | Leo Jones | Leo | Jones | 16 | Store Temporary Checker | 12 | 16 |
+ …
+
+Since the `postion_id` and `store_id` columns contain integers, you can issue
+a query with the `myaddints` custom function on these columns to add the
+integers in the columns.
+
+The following query tells Drill to apply the `myaddints` function to the
+`position_id` and `store_id` columns in the `employee.json` file:
+
+ 0: jdbc:drill:zk=local> SELECT myaddints(CAST(position_id AS int),CAST(store_id AS int)) FROM cp.`employee.json`;
+
+Since JSON files do not store information about data types, you must apply the
+`CAST` function in the query to tell Drill that the columns contain integer
+values.
+
+The query returns the following results:
+
+ +------------+
+ | EXPR$0 |
+ +------------+
+ | 28 |
+ | 28 |
+ | 36 |
+ +------------+
+ …
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/dev-custom-fcn/005-cust-interface.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/dev-custom-fcn/005-cust-interface.md b/_docs/drill-docs/dev-custom-fcn/005-cust-interface.md
new file mode 100644
index 0000000..b84cad0
--- /dev/null
+++ b/_docs/drill-docs/dev-custom-fcn/005-cust-interface.md
@@ -0,0 +1,14 @@
+---
+title: "Custom Function Interfaces"
+parent: "Develop Custom Functions"
+---
+Implement the Drill interface appropriate for the type of function that you
+want to develop. Each interface provides a set of required holders where you
+input data types that your function uses and required methods that Drill calls
+to perform your function’s operations.
+
+Click on either of the links for more information about custom function
+interfaces for Drill:
+
+ * [Simple Function Interface](/confluence/display/DRILL/Simple+Function+Interface)
+ * [Aggregate Function Interface](/confluence/display/DRILL/Aggregate+Function+Interface)
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/develop/001-compile.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/develop/001-compile.md b/_docs/drill-docs/develop/001-compile.md
new file mode 100644
index 0000000..85db854
--- /dev/null
+++ b/_docs/drill-docs/develop/001-compile.md
@@ -0,0 +1,37 @@
+---
+title: "Compiling Drill From source"
+parent: "Develop Drill"
+---
+## Prerequisites
+
+ * Maven 3.0.4 or later
+ * Oracle JDK 7 or later
+
+Run the following commands to verify that you have the correct versions of
+Maven and JDK installed:
+
+ java -version
+ mvn -version
+
+## 1\. Clone the Repository
+
+ git clone https://git-wip-us.apache.org/repos/asf/incubator-drill.git
+
+## 2\. Compile the Code
+
+ cd incubator-drill
+ mvn clean install -DskipTests
+
+## 3\. Explode the Tarball in the Installation Directory
+
+ mkdir ~/compiled-drill
+ tar xvzf distribution/target/*.tar.gz --strip=1 -C ~/compiled-drill
+
+Now that you have Drill installed, you can connect to Drill and query sample
+data or you can connect Drill to your data sources.
+
+ * To connect Drill to your data sources, refer to [Connecting to Data Sources](https://cwiki.apache.org/confluence/display/DRILL/Connecting+to+Data+Sources) for instructions.
+ * To connect to Drill and query sample data, refer to the following topics:
+ * [Start Drill ](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=44994063)(For Drill installed in embedded mode)
+ * [Query Data ](https://cwiki.apache.org/confluence/display/DRILL/Query+Data)
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/develop/002-setup.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/develop/002-setup.md b/_docs/drill-docs/develop/002-setup.md
new file mode 100644
index 0000000..19fb554
--- /dev/null
+++ b/_docs/drill-docs/develop/002-setup.md
@@ -0,0 +1,5 @@
+---
+title: "Setting Up Your Development Environment"
+parent: "Develop Drill"
+---
+TBD
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/develop/003-patch-tool.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/develop/003-patch-tool.md b/_docs/drill-docs/develop/003-patch-tool.md
new file mode 100644
index 0000000..5b94577
--- /dev/null
+++ b/_docs/drill-docs/develop/003-patch-tool.md
@@ -0,0 +1,160 @@
+---
+title: "Compiling Drill From source"
+parent: "Develop Drill"
+---
+ * Drill JIRA and Reviewboard script
+ * 1\. Setup
+ * 2\. Usage
+ * 3\. Upload patch
+ * 4\. Update patch
+ * JIRA command line tool
+ * 1\. Download the JIRA command line package
+ * 2\. Configure JIRA username and password
+ * Reviewboard
+ * 1\. Install the post-review tool
+ * 2\. Configure Stuff
+ * FAQ
+ * When I run the script, it throws the following error and exits
+ * When I run the script, it throws the following error and exits
+
+### Drill JIRA and Reviewboard script
+
+#### 1\. Setup
+
+ 1. Follow instructions [here](https://cwiki.apache.org/confluence/display/DRILL/Drill+Patch+Review+Tool#Drillpatchreviewtool-JIRAcommandlinetool) to setup the jira-python package
+ 2. Follow instructions [here](https://cwiki.apache.org/confluence/display/DRILL/Drill+Patch+Review+Tool#Drillpatchreviewtool-Reviewboard) to setup the reviewboard python tools
+ 3. Install the argparse module
+
+ On Linux -> sudo yum install python-argparse
+ On Mac -> sudo easy_install argparse
+
+#### 2\. Usage
+
+ nnarkhed-mn: nnarkhed$ python drill-patch-review.py --help
+ usage: drill-patch-review.py [-h] -b BRANCH -j JIRA [-s SUMMARY]
+ [-d DESCRIPTION] [-r REVIEWBOARD] [-t TESTING]
+ [-v VERSION] [-db] -rbu REVIEWBOARDUSER -rbp REVIEWBOARDPASSWORD
+
+ Drill patch review tool
+
+ optional arguments:
+ -h, --help show this help message and exit
+ -b BRANCH, --branch BRANCH
+ Tracking branch to create diff against
+ -j JIRA, --jira JIRA JIRA corresponding to the reviewboard
+ -s SUMMARY, --summary SUMMARY
+ Summary for the reviewboard
+ -d DESCRIPTION, --description DESCRIPTION
+ Description for reviewboard
+ -r REVIEWBOARD, --rb REVIEWBOARD
+ Review board that needs to be updated
+ -t TESTING, --testing-done TESTING
+ Text for the Testing Done section of the reviewboard
+ -v VERSION, --version VERSION
+ Version of the patch
+ -db, --debug Enable debug mode
+ -rbu, --reviewboard-user Reviewboard user name
+ -rbp, --reviewboard-password Reviewboard password
+
+#### 3\. Upload patch
+
+ 1. Specify the branch against which the patch should be created (-b)
+ 2. Specify the corresponding JIRA (-j)
+ 3. Specify an **optional** summary (-s) and description (-d) for the reviewboard
+
+Example:
+
+ python drill-patch-review.py -b origin/master -j DRILL-241 -rbu tnachen -rbp password
+
+#### 4\. Update patch
+
+ 1. Specify the branch against which the patch should be created (-b)
+ 2. Specify the corresponding JIRA (--jira)
+ 3. Specify the rb to be updated (-r)
+ 4. Specify an **optional** summary (-s) and description (-d) for the reviewboard, if you want to update it
+ 5. Specify an **optional** version of the patch. This will be appended to the jira to create a file named JIRA-<version>.patch. The purpose is to be able to upload multiple patches to the JIRA. This has no bearing on the reviewboard update.
+
+Example:
+
+ python drill-patch-review.py -b origin/master -j DRILL-241 -r 14081 rbp tnachen -rbp password
+
+### JIRA command line tool
+
+#### 1\. Download the JIRA command line package
+
+Install the jira-python package.
+
+ sudo easy_install jira-python
+
+#### 2\. Configure JIRA username and password
+
+Include a jira.ini file in your $HOME directory that contains your Apache JIRA
+username and password.
+
+ nnarkhed-mn:~ nnarkhed$ cat ~/jira.ini
+ user=nehanarkhede
+ password=***********
+
+### Reviewboard
+
+This is a quick tutorial on using [Review Board](https://reviews.apache.org)
+with Drill.
+
+#### 1\. Install the post-review tool
+
+If you are on RHEL, Fedora or CentOS, follow these steps:
+
+ sudo yum install python-setuptools
+ sudo easy_install -U RBTools
+
+If you are on Mac, follow these steps:
+
+ sudo easy_install -U setuptools
+ sudo easy_install -U RBTools
+
+For other platforms, follow the [instructions](http://www.reviewboard.org/docs/manual/dev/users/tools/post-review/) to
+setup the post-review tool.
+
+#### 2\. Configure Stuff
+
+Then you need to configure a few things to make it work.
+
+First set the review board url to use. You can do this from in git:
+
+ git config reviewboard.url https://reviews.apache.org
+
+If you checked out using the git wip http url that confusingly won't work with
+review board. So you need to configure an override to use the non-http url.
+You can do this by adding a config file like this:
+
+ jkreps$ cat ~/.reviewboardrc
+ REPOSITORY = 'git://git.apache.org/incubator-drill.git'
+ TARGET_GROUPS = 'drill-git'
+GUESS_FIELDS = True
+
+
+
+### FAQ
+
+#### When I run the script, it throws the following error and exits
+
+ nnarkhed$python drill-patch-review.py -b trunk -j DRILL-241
+ There don't seem to be any diffs
+
+There are two reasons for this:
+
+ * The code is not checked into your local branch
+ * The -b branch is not pointing to the remote branch. In the example above, "trunk" is specified as the branch, which is the local branch. The correct value for the -b (--branch) option is the remote branch. "git branch -r" gives the list of the remote branch names.
+
+#### When I run the script, it throws the following error and exits
+
+Error uploading diff
+
+Your review request still exists, but the diff is not attached.
+
+One of the most common root causes of this error are that the git remote
+branches are not up-to-date. Since the script already does that, it is
+probably due to some other problem. You can run the script with the --debug
+option that will make post-review run in the debug mode and list the root
+cause of the issue.
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/install/001-drill-in-10.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/install/001-drill-in-10.md b/_docs/drill-docs/install/001-drill-in-10.md
new file mode 100644
index 0000000..bd60141
--- /dev/null
+++ b/_docs/drill-docs/install/001-drill-in-10.md
@@ -0,0 +1,395 @@
+---
+title: "Apache Drill in 10 Minutes"
+parent: "Install Drill"
+---
+* Objective
+* A Few Bits About Apache Drill
+* Process Overview
+* Install Drill
+ * Installing Drill on Linux
+ * Installing Drill on Mac OS X
+ * Installing Drill on Windows
+* Start Drill
+* Query Sample Data
+* Summary
+* Next Steps
+* More Information
+
+
+
+# Objective
+
+Use Apache Drill to query sample data in 10 minutes. For simplicity, you’ll
+run Drill in _embedded_ mode rather than _distributed_ mode to try out Drill
+without having to perform any setup tasks.
+
+# A Few Bits About Apache Drill
+
+Drill is a clustered, powerful MPP (Massively Parallel Processing) query
+engine for Hadoop that can process petabytes of data, fast. Drill is useful
+for short, interactive ad-hoc queries on large-scale data sets. Drill is
+capable of querying nested data in formats like JSON and Parquet and
+performing dynamic schema discovery. Drill does not require a centralized
+metadata repository.
+
+### **_Dynamic schema discovery _**
+
+Drill does not require schema or type specification for data in order to start
+the query execution process. Drill starts data processing in record-batches
+and discovers the schema during processing. Self-describing data formats such
+as Parquet, JSON, AVRO, and NoSQL databases have schema specified as part of
+the data itself, which Drill leverages dynamically at query time. Because
+schema can change over the course of a Drill query, all Drill operators are
+designed to reconfigure themselves when schemas change.
+
+### **_Flexible data model_**
+
+Drill allows access to nested data attributes, just like SQL columns, and
+provides intuitive extensions to easily operate on them. From an architectural
+point of view, Drill provides a flexible hierarchical columnar data model that
+can represent complex, highly dynamic and evolving data models. Drill allows
+for efficient processing of these models without the need to flatten or
+materialize them at design time or at execution time. Relational data in Drill
+is treated as a special or simplified case of complex/multi-structured data.
+
+### **_De-centralized metadata_**
+
+Drill does not have a centralized metadata requirement. You do not need to
+create and manage tables and views in a metadata repository, or rely on a
+database administrator group for such a function. Drill metadata is derived
+from the storage plugins that correspond to data sources. Storage plugins
+provide a spectrum of metadata ranging from full metadata (Hive), partial
+metadata (HBase), or no central metadata (files). De-centralized metadata
+means that Drill is NOT tied to a single Hive repository. You can query
+multiple Hive repositories at once and then combine the data with information
+from HBase tables or with a file in a distributed file system. You can also
+use SQL DDL syntax to create metadata within Drill, which gets organized just
+like a traditional database. Drill metadata is accessible through the ANSI
+standard INFORMATION_SCHEMA database.
+
+### **_Extensibility_**
+
+Drill provides an extensible architecture at all layers, including the storage
+plugin, query, query optimization/execution, and client API layers. You can
+customize any layer for the specific needs of an organization or you can
+extend the layer to a broader array of use cases. Drill provides a built in
+classpath scanning and plugin concept to add additional storage plugins,
+functions, and operators with minimal configuration.
+
+# Process Overview
+
+Download the Apache Drill archive and extract the contents to a directory on
+your machine. The Apache Drill archive contains sample JSON and Parquet files
+that you can query immediately.
+
+Query the sample JSON and parquet files using SQLLine. SQLLine is a pure-Java
+console-based utility for connecting to relational databases and executing SQL
+commands. SQLLine is used as the shell for Drill. Drill follows the ANSI SQL:
+2011 standard with a few extensions for nested data formats.
+
+### Prerequisite
+
+You must have the following software installed on your machine to run Drill:
+
+<div class="table-wrap"><table class="confluenceTable"><tbody><tr><td class="confluenceTd"><p><strong>Software</strong></p></td><td class="confluenceTd"><p><strong>Description</strong></p></td></tr><tr><td class="confluenceTd"><p><a href="http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html" class="external-link" rel="nofollow">Oracle JDK version 7</a></p></td><td class="confluenceTd"><p>A set of programming tools for developing Java applications.</p></td></tr></tbody></table></div>
+
+
+### Prerequisite Validation
+
+Run the following command to verify that the system meets the software
+prerequisite:
+<table class="confluenceTable"><tbody><tr><td class="confluenceTd"><p><strong>Command </strong></p></td><td class="confluenceTd"><p><strong>Example Output</strong></p></td></tr><tr><td class="confluenceTd"><p><code>java –version</code></p></td><td class="confluenceTd"><p><code>java version "1.7.0_65"</code><br /><code>Java(TM) SE Runtime Environment (build 1.7.0_65-b19)</code><br /><code>Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)</code></p></td></tr></tbody></table>
+
+# Install Drill
+
+You can install Drill on a machine running Linux, Mac OS X, or Windows.
+
+## Installing Drill on Linux
+
+Complete the following steps to install Drill:
+
+ 1. Issue the following command to download the latest, stable version of Apache Drill to a directory on your machine:
+
+ wget http://www.apache.org/dyn/closer.cgi/drill/drill-0.7.0/apache-drill-0.7.0.tar.gz
+
+ 2. Issue the following command to create a new directory to which you can extract the contents of the Drill `tar.gz` file:
+
+ sudo mkdir -p /opt/drill
+
+ 3. Navigate to the directory where you downloaded the Drill `tar.gz` file.
+
+
+ 4. Issue the following command to extract the contents of the Drill `tar.gz` file:
+
+ sudo tar -xvzf apache-drill-<version>.tar.gz -C /opt/drill
+
+ 5. Issue the following command to navigate to the Drill installation directory:
+
+ cd /opt/drill/apache-drill-<version>
+
+At this point, you can [start Drill](https://cwiki.apache.org/confluence/displ
+ay/DRILL/Apache+Drill+in+10+Minutes#ApacheDrillin10Minutes-StartDrill).
+
+## Installing Drill on Mac OS X
+
+Complete the following steps to install Drill:
+
+ 1. Open a Terminal window, and create a `drill` directory inside your home directory (or in some other location if you prefer).
+
+ **Example**
+
+ $ pwd
+ /Users/max
+ $ mkdir drill
+ $ cd drill
+ $ pwd
+ /Users/max/drill
+
+ 2. Click the following link to download the latest, stable version of Apache Drill:
+
+ [http://www.apache.org/dyn/closer.cgi/drill/drill-0.7.0/apache-drill-0.7.0.tar.gz](http://www.apache.org/dyn/closer.cgi/drill/drill-0.7.0/apache-drill-0.7.0.tar.gz)
+
+ 3. Open the downloaded `TAR` file with the Mac Archive utility or a similar tool for unzipping files.
+
+ 4. Move the resulting `apache-drill-<version>` folder into the `drill` directory that you created.
+
+ 5. Issue the following command to navigate to the `apache-drill-<version>` directory:
+
+ cd /Users/max/drill/apache-drill-<version>
+
+At this point, you can [start Drill](https://cwiki.apache.org/confluence/displ
+ay/DRILL/Apache+Drill+in+10+Minutes#ApacheDrillin10Minutes-StartDrill).
+
+## Installing Drill on Windows
+
+You can install Drill on Windows 7 or 8. To install Drill on Windows, you must
+have JDK 7, and you must set the `JAVA_HOME` path in the Windows Environment
+Variables. You must also have a utility, such as
+[7-zip](http://www.7-zip.org/), installed on your machine. These instructions
+assume that the [7-zip](http://www.7-zip.org/) decompression utility is
+installed to extract a Drill archive file that you download.
+
+#### Setting JAVA_HOME
+
+Complete the following steps to set `JAVA_HOME`:
+
+ 1. Navigate to `Control Panel\All Control Panel Items\System`, and select **Advanced System Settings**. The System Properties window appears.
+ 2. On the Advanced tab, click **Environment Variables**. The Environment Variables window appears.
+ 3. Add/Edit `JAVA_HOME` to point to the location where the JDK software is located.
+
+ **Example**
+
+ C:\Program Files\Java\jdk1.7.0_65
+
+ 4. Click **OK** to exit the windows.
+
+#### Installing Drill
+
+Complete the following steps to install Drill:
+
+ 1. Create a `drill` directory on your `C:\` drive, (or in some other location if you prefer).
+
+ **Example**
+
+ C:\drill
+
+ Do not include spaces in your directory path. If you include spaces in the
+directory path, Drill fails to run.
+
+ 2. Click the following link to download the latest, stable version of Apache Drill:
+
+ [http://www.apache.org/dyn/closer.cgi/drill/drill-0.7.0/apache-drill-0.7.0.tar.gz](http://www.apache.org/dyn/closer.cgi/drill/drill-0.7.0/apache-drill-0.7.0.tar.gz)
+
+ 3. Move the `apache-drill-<version>.tar.gz` file to the `drill` directory that you created on your `C:\` drive.
+
+ 4. Unzip the `TAR.GZ` file and the resulting `TAR` file.
+
+ 1. Right-click `apache-drill-<version>.tar.gz,` and select` 7-Zip>Extract Here`. The utility extracts the `apache-drill-<version>.tar` file.
+ 2. Right-click `apache-drill-<version>.tar, `and select`` 7-Zip>Extract Here`. `The utility extracts the` apache-drill-<version> `folder.
+ 5. Open the `apache-drill-<version> `folder.
+
+ 6. Open the `bin` folder, and double-click on the `sqlline.bat` file. The Windows command prompt opens.
+ 7. At the `sqlline>` prompt, type `!connect jdbc:drill:zk=local` and then press `Enter`.
+ 8. Enter the username and password.
+ a. When prompted, enter the user name `admin` and then press Enter.
+ b. When prompted, enter the password `admin` and then press Enter. The cursor blinks for a few seconds and then `0: jdbc:drill:zk=local> `displays in the prompt.
+
+At this point, you can submit queries to Drill. Refer to the [Query Sample Dat
+a](https://cwiki.apache.org/confluence/display/DRILL/Apache+Drill+in+10+Minute
+s#ApacheDrillin10Minutes-QuerySampleData) section of this document.
+
+# Start Drill
+
+Launch SQLLine, the Drill shell, to start and run Drill in embedded mode.
+Launching SQLLine automatically starts a new Drillbit within the shell. In a
+production environment, Drillbits are the daemon processes that run on each
+node in a Drill cluster.
+
+Complete the following steps to launch SQLLine and start Drill:
+
+ 1. Verify that you are in the Drill installation directory.
+Example: `~/apache-drill-<version>`
+
+ 2. Issue the following command to launch SQLLine:
+
+ bin/sqlline -u jdbc:drill:zk=local
+
+ `-u` is a JDBC connection string that directs SQLLine to connect to Drill. It
+also starts a local Drillbit. If you are connecting to an Apache Drill
+cluster, the value of `zk=` would be a list of Zookeeper quorum nodes. For
+more information about how to run Drill in clustered mode, go to [Deploying
+Apache Drill in a Clustered Environment](/confluence/display/DRILL/Deploying+A
+pache+Drill+in+a+Clustered+Environment).
+
+When SQLLine starts, the system displays the following prompt:
+`0: jdbc:drill:zk=local>`
+
+Issue the following command when you want to exit SQLLine:
+
+ !quit
+
+
+# Query Sample Data
+
+Your Drill installation includes a `sample-date` directory with JSON and
+Parquet files that you can query. The local file system on your machine is
+configured as the `dfs` storage plugin instance by default when you install
+Drill in embedded mode. For more information about storage plugin
+configuration, refer to [Storage Plugin Registration](https://cwiki.apache.org
+/confluence/display/DRILL/Connecting+to+Data+Sources#ConnectingtoDataSources-
+StoragePluginRegistration).
+
+Use SQL syntax to query the sample `JSON` and `Parquet` files in the `sample-
+data` directory on your local file system.
+
+### Querying a JSON File
+
+A sample JSON file, `employee.json`, contains fictitious employee data.
+
+To view the data in the `employee.json` file, submit the following SQL query
+to Drill:
+
+ 0: jdbc:drill:zk=local> SELECT * FROM cp.`employee.json`;
+
+The query returns the following results:
+
+**Example of partial output**
+
+ +-------------+------------+------------+------------+-------------+-----------+
+ | employee_id | full_name | first_name | last_name | position_id | position_ |
+ +-------------+------------+------------+------------+-------------+-----------+
+ | 1101 | Steve Eurich | Steve | Eurich | 16 | Store T |
+ | 1102 | Mary Pierson | Mary | Pierson | 16 | Store T |
+ | 1103 | Leo Jones | Leo | Jones | 16 | Store Tem |
+ | 1104 | Nancy Beatty | Nancy | Beatty | 16 | Store T |
+ | 1105 | Clara McNight | Clara | McNight | 16 | Store |
+ | 1106 | Marcella Isaacs | Marcella | Isaacs | 17 | Stor |
+ | 1107 | Charlotte Yonce | Charlotte | Yonce | 17 | Stor |
+ | 1108 | Benjamin Foster | Benjamin | Foster | 17 | Stor |
+ | 1109 | John Reed | John | Reed | 17 | Store Per |
+ | 1110 | Lynn Kwiatkowski | Lynn | Kwiatkowski | 17 | St |
+ | 1111 | Donald Vann | Donald | Vann | 17 | Store Pe |
+ | 1112 | William Smith | William | Smith | 17 | Store |
+ | 1113 | Amy Hensley | Amy | Hensley | 17 | Store Pe |
+ | 1114 | Judy Owens | Judy | Owens | 17 | Store Per |
+ | 1115 | Frederick Castillo | Frederick | Castillo | 17 | S |
+ | 1116 | Phil Munoz | Phil | Munoz | 17 | Store Per |
+ | 1117 | Lori Lightfoot | Lori | Lightfoot | 17 | Store |
+ +-------------+------------+------------+------------+-------------+-----------+
+ 1,155 rows selected (0.762 seconds)
+ 0: jdbc:drill:zk=local>
+
+### Querying a Parquet File
+
+Query the `region.parquet` and `nation.parquet` files in the `sample-data`
+directory on your local file system.
+
+#### Region File
+
+If you followed the Apache Drill in 10 Minutes instructions to install Drill
+in embedded mode, the path to the parquet file varies between operating
+systems.
+
+**Note:** When you enter the query, include the version of Drill that you are currently running.
+
+To view the data in the `region.parquet` file, issue the query appropriate for
+your operating system:
+
+* Linux
+`SELECT * FROM dfs.`/opt/drill/apache-drill-<version>/sample-
+data/region.parquet`; `
+* Mac OS X
+`SELECT * FROM dfs.`/Users/max/drill/apache-drill-<version>/sample-
+data/region.parquet`;`
+* Windows
+`SELECT * FROM dfs.`C:\drill\apache-drill-<version>\sample-
+data\region.parquet`;`
+
+The query returns the following results:
+
+ +------------+------------+
+ | EXPR$0 | EXPR$1 |
+ +------------+------------+
+ | AFRICA | lar deposits. blithely final packages cajole. regular waters ar |
+ | AMERICA | hs use ironic, even requests. s |
+ | ASIA | ges. thinly even pinto beans ca |
+ | EUROPE | ly final courts cajole furiously final excuse |
+ | MIDDLE EAST | uickly special accounts cajole carefully blithely close reques |
+ +------------+------------+
+ 5 rows selected (0.165 seconds)
+ 0: jdbc:drill:zk=local>
+
+#### Nation File
+
+If you followed the Apache Drill in 10 Minutes instructions to install Drill
+in embedded mode, the path to the parquet file varies between operating
+systems.
+
+**Note:** When you enter the query, include the version of Drill that you are currently running.
+
+To view the data in the `nation.parquet` file, issue the query appropriate for
+your operating system:
+
+* Linux
+
+ ``SELECT * FROM dfs.`/opt/drill/apache-drill-<version>/sample-
+data/nation.parquet`;``
+* Mac OS X
+
+ ``SELECT * FROM dfs.`/Users/max/drill/apache-drill-<version>/sample-
+data/nation.parquet`;``
+
+* Windows
+
+ ``SELECT * FROM dfs.`C:\drill\apache-drill-<version>\sample-
+data\nation.parquet`;``
+
+The query returns the following results:
+
+# Summary
+
+Now you know a bit about Apache Drill. To summarize, you have completed the
+following tasks:
+
+ * Learned that Apache Drill supports nested data, schema-less execution, and decentralized metadata.
+ * Downloaded and installed Apache Drill.
+ * Invoked SQLLine with Drill in embedded mode.
+ * Queried the sample JSON file, `employee.json`, to view its data.
+ * Queried the sample `region.parquet` file to view its data.
+ * Queried the sample `nation.parquet` file to view its data.
+
+# Next Steps
+
+Now that you have an idea about what Drill can do, you might want to:
+
+ * [Deploy Drill in a clustered environment.](https://cwiki.apache.org/confluence/display/DRILL/Deploying+Apache+Drill+in+a+Clustered+Environment)
+ * [Configure storage plugins to connect Drill to your data sources](https://cwiki.apache.org/confluence/display/DRILL/Connecting+to+Data+Sources).
+ * Query [Hive](https://cwiki.apache.org/confluence/display/DRILL/Connecting+to+Data+Sources#ConnectingtoDataSources-QueryingHiveTables) and [HBase](https://cwiki.apache.org/confluence/display/DRILL/Connecting+to+Data+Sources#ConnectingtoDataSources-QueryingHiveTables) data.
+ * [Query Complex Data](https://cwiki.apache.org/confluence/display/DRILL/Querying+Complex+Data)
+
+ * [Query Plain Text Files](https://cwiki.apache.org/confluence/display/DRILL/Querying+Plain+Text+Files)
+
+# More Information
+
+For more information about Apache Drill, go to [Apache Drill
+Wiki](https://cwiki.apache.org/confluence/display/DRILL/Apache+Drill+Wiki).
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/install/002-deploy.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/install/002-deploy.md b/_docs/drill-docs/install/002-deploy.md
new file mode 100644
index 0000000..49ba68b
--- /dev/null
+++ b/_docs/drill-docs/install/002-deploy.md
@@ -0,0 +1,102 @@
+---
+title: "Deploying Apache Drill in a Clustered Environment"
+parent: "Install Drill"
+---
+# Overview
+
+To run Drill in a clustered environment, complete the following steps:
+
+ 1. Install Drill on each designated node in the cluster.
+ 2. Configure a cluster ID and add Zookeeper information.
+ 3. Connect Drill to your data sources.
+ 4. Start Drill.
+
+### Prerequisites
+
+Before you install Apache Drill on nodes in your cluster, you must have the
+following software and services installed:
+
+ * [Oracle JDK version 7](http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html)
+ * Configured and running ZooKeeper quorum
+ * Configured and running Hadoop cluster (Recommended)
+ * DNS (Recommended)
+
+Installing Drill
+
+Complete the following steps to install Drill on designated nodes:
+
+ 1. Download the Drill tarball.
+
+ curl http://www.apache.org/dyn/closer.cgi/drill/drill-0.7.0/apache-drill-0.7.0.tar.gz
+
+ 2. Issue the following command to create a Drill installation directory and then explode the tarball to the directory:
+
+ mkdir /opt/drill
+ tar xzf apache-drill-<version>.tar.gz --strip=1 -C /opt/drill
+
+ 3. If you are using external JAR files, edit `drill-env.sh, `located in `/opt/drill/conf/`, and define `HADOOP_HOME:`
+
+ export HADOOP_HOME="~/hadoop/hadoop-0.20.2/"
+
+ 4. In `drill-override.conf,`create a unique Drill `cluster ID`, and provide Zookeeper host names and port numbers to configure a connection to your Zookeeper quorum.
+
+ a. Edit `drill-override.conf `located in `~/drill/drill-<version>/conf/`.
+
+ b. Provide a unique `cluster-id` and the Zookeeper host names and port numbers in `zk.connect`. If you install Drill on multiple nodes, assign the same `cluster ID` to each Drill node so that all Drill nodes share the same ID. The default Zookeeper port is 2181.
+
+**Example**
+
+ drill.exec: {
+ cluster-id: "<mydrillcluster>",
+ zk.connect: "<zkhostname1>:<port>,<zkhostname2>:<port>,<zkhostname3>:<port>",
+ debug.error_on_leak: false,
+ buffer.size: 6,
+ functions: ["org.apache.drill.expr.fn.impl", "org.apache.drill.udfs"]
+ }
+
+Connecting Drill to Data Sources
+
+You can connect Drill to various types of data sources. Refer to [Connect
+Apache Drill to Data Sources](https://cwiki.apache.org/confluence/display/DRIL
+L/Connecting+to+Data+Sources) to get configuration instructions for the
+particular type of data source that you want to connect to Drill.
+
+## Starting Drill
+
+Complete the following steps to start Drill:
+
+ 1. Navigate to the Drill installation directory, and issue the following command to start a Drillbit:
+
+ bin/drillbit.sh restart
+
+ 2. Issue the following command to invoke SQLLine and start Drill:
+
+ bin/sqlline -u jdbc:drill:
+
+ When connected, the Drill prompt appears.
+ Example:
+
+ `0: jdbc:drill:zk=<zk1host>:<port>>`
+
+ If you cannot connect to Drill, invoke SQLLine with the ZooKeeper quorum:
+
+ `bin/sqlline -u jdbc:drill:zk=<zk1host>:<port>,<zk2host>:<port>,<zk3host>:<port>`
+
+ 3. Issue the following query to Drill to verify that all Drillbits have joined the cluster:
+
+ 0: jdbc:drill:zk=<zk1host>:<port>> select * from sys.drillbits;
+
+Drill provides a list of Drillbits that have joined.
+
+ +------------+------------+--------------+--------------------+
+ | host | user_port | control_port | data_port |
+ +------------+------------+--------------+--------------------+
+ | <host address> | <port number>| <port number>| <port number>|
+ +------------+------------+--------------+--------------------+
+
+**Example**
+
+Now you can query data with Drill. The Drill installation includes sample data
+that you can query. Refer to [Query Sample Data](https://cwiki.apache.org/conf
+luence/display/DRILL/Apache+Drill+in+10+Minutes#ApacheDrillin10Minutes-
+QuerySampleData).
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/install/003-install-embedded.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/install/003-install-embedded.md b/_docs/drill-docs/install/003-install-embedded.md
new file mode 100644
index 0000000..eb4fa2a
--- /dev/null
+++ b/_docs/drill-docs/install/003-install-embedded.md
@@ -0,0 +1,30 @@
+---
+title: "Installing Drill in Embedded Mode"
+parent: "Install Drill"
+---
+Installing Drill in embedded mode installs Drill locally on your machine.
+Embedded mode is a quick, easy way to install and try Drill without having to
+perform any configuration tasks. When you install Drill in embedded mode, the
+Drillbit service is installed locally and starts automatically when you invoke
+SQLLine, the Drill shell. You can install Drill in embedded mode on a machine
+running Linux, Mac OS X, or Windows.
+
+**Prerequisite:**
+
+You must have the following software installed on your machine to run Drill:
+
+<div class="table-wrap"><table class="confluenceTable"><tbody><tr><td class="confluenceTd"><p><strong>Software</strong></p></td><td class="confluenceTd"><p><strong>Description</strong></p></td></tr><tr><td class="confluenceTd"><p><a class="external-link" href="http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html" rel="nofollow">Oracle JDK version 7</a></p></td><td class="confluenceTd"><p>A set of programming tools for developing Java applications.</p></td></tr></tbody></table></div>
+
+
+A set of programming tools for developing Java applications.
+
+You can run the following command to verify that the system meets the software
+prerequisite:
+
+<div class="table-wrap"><table class="confluenceTable"><tbody><tr><td class="confluenceTd"><p><strong>Command</strong></p></td><td class="confluenceTd"><p><strong>Example Output</strong></p></td></tr><tr><td class="confluenceTd"><p><code>java –version</code></p></td><td class="confluenceTd"><p><code>java version "1.7.0_65"</code><br /><code>Java(TM) SE Runtime Environment (build 1.7.0_65-b19)</code><br /><code>Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)</code></p></td></tr></tbody></table></div>
+
+Click on the installation link appropriate for your operating system:
+
+ * [Installing Drill on Linux](/confluence/display/DRILL/Installing+Drill+on+Linux)
+ * [Installing Drill on Mac OS X](/confluence/display/DRILL/Installing+Drill+on+Mac+OS+X)
+ * [Installing Drill on Windows](/confluence/display/DRILL/Installing+Drill+on+Windows)
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/install/004-install-distributed.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/install/004-install-distributed.md b/_docs/drill-docs/install/004-install-distributed.md
new file mode 100644
index 0000000..3d993e7
--- /dev/null
+++ b/_docs/drill-docs/install/004-install-distributed.md
@@ -0,0 +1,61 @@
+---
+title: "Installing Drill in Distributed Mode"
+parent: "Install Drill"
+---
+You can install Apache Drill in distributed mode on one or multiple nodes to
+run it in a clustered environment.
+
+To install Apache Drill in distributed mode, complete the following steps:
+
+ 1. Install Drill on each designated node in the cluster.
+ 2. Configure a cluster ID and add Zookeeper information.
+ 3. Connect Drill to your data sources.
+ 4. Start Drill.
+
+**Prerequisites**
+
+Before you install Apache Drill on nodes in your cluster, you must have the
+following software and services installed:
+
+ * [Oracle JDK version 7](http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html)
+ * Configured and running ZooKeeper quorum
+ * Configured and running Hadoop cluster (Recommended)
+ * DNS (Recommended)
+
+## Installing Drill
+
+Complete the following steps to install Drill on designated nodes:
+
+ 1. Download the Drill tarball.
+
+ curl http://www.apache.org/dyn/closer.cgi/drill/drill-0.7.0/apache-drill-0.7.0.tar.gz
+
+ 2. Issue the following command to create a Drill installation directory and then explode the tarball to the directory:
+
+ mkdir /opt/drill
+ tar xzf apache-drill-<version>.tar.gz --strip=1 -C /opt/drill
+
+ 3. If you are using external JAR files, edit `drill-env.sh, `located in `/opt/drill/conf/`, and define `HADOOP_HOME:`
+
+ export HADOOP_HOME="~/hadoop/hadoop-0.20.2/"
+
+ 4. In `drill-override.conf,`create a unique Drill `cluster ID`, and provide Zookeeper host names and port numbers to configure a connection to your Zookeeper quorum.
+
+ a. Edit `drill-override.conf `located in `~/drill/drill-<version>/conf/`.
+
+ b. Provide a unique `cluster-id` and the Zookeeper host names and port numbers in `zk.connect`. If you install Drill on multiple nodes, assign the same `cluster ID` to each Drill node so that all Drill nodes share the same ID. The default Zookeeper port is 2181.
+
+ **Example**
+
+ drill.exec:{
+ cluster-id: "<mydrillcluster>",
+ zk.connect: "<zkhostname1>:<port>,<zkhostname2>:<port>,<zkhostname3>:<port>",
+ debug.error_on_leak: false,
+ buffer.size: 6,
+ functions: ["org.apache.drill.expr.fn.impl", "org.apache.drill.udfs"]
+ }
+
+You can connect Drill to various types of data sources. Refer to [Connect
+Apache Drill to Data Sources](https://cwiki.apache.org/confluence/display/DRIL
+L/Connecting+to+Data+Sources) to get configuration instructions for the
+particular type of data source that you want to connect to Drill.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/install/install-embedded/001-install-linux.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/install/install-embedded/001-install-linux.md b/_docs/drill-docs/install/install-embedded/001-install-linux.md
new file mode 100644
index 0000000..4dbc3c0
--- /dev/null
+++ b/_docs/drill-docs/install/install-embedded/001-install-linux.md
@@ -0,0 +1,30 @@
+---
+title: "Installing Drill on Linux"
+parent: "Installing Drill in Embedded Mode"
+---
+Complete the following steps to install Apache Drill on a machine running
+Linux:
+
+ 1. Issue the following command to download the latest, stable version of Apache Drill to a directory on your machine:
+
+ wget http://www.apache.org/dyn/closer.cgi/drill/drill-0.7.0/apache-drill-0.7.0.tar.gz
+
+
+ 2. Issue the following command to create a new directory to which you can extract the contents of the Drill `tar.gz` file:
+
+ sudo mkdir -p /opt/drill
+
+ 3. Navigate to the directory where you downloaded the Drill `tar.gz` file.
+
+
+ 4. Issue the following command to extract the contents of the Drill `tar.gz` file to the directory you created:
+
+ sudo tar -xvzf apache-drill-<version>.tar.gz -C /opt/drill
+
+ 5. Issue the following command to navigate to the Drill installation directory:
+
+ cd /opt/drill/apache-drill-<version>
+
+At this point, you can [invoke
+SQLLine](/confluence/pages/viewpage.action?pageId=44994063#Starting
+/StoppingDrill-invokeSQLLine) to run Drill.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/install/install-embedded/002-install-mac.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/install/install-embedded/002-install-mac.md b/_docs/drill-docs/install/install-embedded/002-install-mac.md
new file mode 100644
index 0000000..f20a01d
--- /dev/null
+++ b/_docs/drill-docs/install/install-embedded/002-install-mac.md
@@ -0,0 +1,33 @@
+---
+title: "Installing Drill on Mac OS X"
+parent: "Installing Drill in Embedded Mode"
+---
+Complete the following steps to install Apache Drill on a machine running Mac
+OS X:
+
+ 1. Open a Terminal window, and create a `drill` directory inside your home directory (or in some other location if you prefer).
+
+ **Example**
+
+ $ pwd
+ /Users/max
+ $ mkdir drill
+ $ cd drill
+ $ pwd
+ /Users/max/drill
+
+ 2. Click the following link to download the latest, stable version of Apache Drill:
+
+ [http://www.apache.org/dyn/closer.cgi/drill/drill-0.7.0/apache-drill-0.7.0.tar.gz](http://www.apache.org/dyn/closer.cgi/drill/drill-0.7.0/apache-drill-0.7.0.tar.gz)
+
+ 3. Open the downloaded `TAR` file with the Mac Archive utility or a similar tool for unzipping files.
+
+ 4. Move the resulting `apache-drill-<version>` folder into the `drill` directory that you created.
+
+ 5. Issue the following command to navigate to the `apache-drill-<version>` directory:
+
+ cd /Users/max/drill/apache-drill-<version>
+
+At this point, you can [invoke SQLLine](https://cwiki.apache.org/confluence/pa
+ges/viewpage.action?pageId=44994063#Starting/StoppingDrill-invokeSQLLine) to
+run Drill.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/install/install-embedded/003-install-win.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/install/install-embedded/003-install-win.md b/_docs/drill-docs/install/install-embedded/003-install-win.md
new file mode 100644
index 0000000..285b584
--- /dev/null
+++ b/_docs/drill-docs/install/install-embedded/003-install-win.md
@@ -0,0 +1,57 @@
+---
+title: "Installing Drill on Windows"
+parent: "Installing Drill in Embedded Mode"
+---
+You can install Drill on Windows 7 or 8. To install Drill on Windows, you must
+have JDK 7, and you must set the `JAVA_HOME` path in the Windows Environment
+Variables. You must also have a utility, such as
+[7-zip](http://www.7-zip.org/), installed on your machine. These instructions
+assume that the [7-zip](http://www.7-zip.org/) decompression utility is
+installed to extract the Drill archive file that you download.
+
+#### Setting JAVA_HOME
+
+Complete the following steps to set `JAVA_HOME`:
+
+ 1. Navigate to `Control Panel\All Control Panel Items\System`, and select **Advanced System Settings**. The System Properties window appears.
+ 2. On the Advanced tab, click **Environment Variables**. The Environment Variables window appears.
+ 3. Add/Edit `JAVA_HOME` to point to the location where the JDK software is located.
+
+ **Example**
+
+ C:\Program Files\Java\jdk1.7.0_65
+
+ 4. Click **OK** to exit the windows.
+
+#### Installing Drill
+
+Complete the following steps to install Drill:
+
+ 1. Create a `drill` directory on your `C:\` drive, (or in some other location if you prefer).
+
+ **Example**
+
+ Do not include spaces in your directory path. If you include spaces in the
+directory path, Drill fails to run.
+
+ 2. Click the following link to download the latest, stable version of Apache Drill:
+
+ [http://www.apache.org/dyn/closer.cgi/drill/drill-0.7.0/apache-drill-0.7.0.tar.gz](http://www.apache.org/dyn/closer.cgi/drill/drill-0.7.0/apache-drill-0.7.0.tar.gz)
+
+ 3. Move the `apache-drill-<version>.tar.gz` file to the `drill` directory that you created on your `C:\` drive.
+
+ 4. Unzip the `TAR.GZ` file and the resulting `TAR` file.
+
+ a. Right-click `apache-drill-<version>.tar.gz,` and select` 7-Zip>Extract Here`. The utility extracts the `apache-drill-<version>.tar` file.
+ b. Right-click `apache-drill-<version>.tar, `and select`` 7-Zip>Extract Here`. `The utility extracts the` apache-drill-<version> `folder.
+ 5. Open the `apache-drill-<version> `folder.
+
+ 6. Open the `bin` folder, and double-click on the `sqlline.bat` file. The Windows command prompt opens.
+ 7. At the `sqlline>` prompt, type `!connect jdbc:drill:zk=local` and then press `Enter`.
+ 8. Enter the username and password.
+ a. When prompted, enter the user name `admin` and then press Enter.
+ b. When prompted, enter the password `admin` and then press Enter. The cursor blinks for a few seconds and then `0: jdbc:drill:zk=local> `displays in the prompt.
+
+At this point, you can submit queries to Drill. Refer to the [Query Sample Dat
+a](https://cwiki.apache.org/confluence/display/DRILL/Apache+Drill+in+10+Minute
+s#ApacheDrillin10Minutes-QuerySampleData) section of this document.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/manage/001-conf.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/manage/001-conf.md b/_docs/drill-docs/manage/001-conf.md
new file mode 100644
index 0000000..5b68b40
--- /dev/null
+++ b/_docs/drill-docs/manage/001-conf.md
@@ -0,0 +1,20 @@
+---
+title: "Configuration Options"
+parent: "Manage Drill"
+---
+Drill provides several configuration options that you can enable, disable, or
+modify. Modifying certain configuration options can impact Drill’s
+performance. Many of Drill's configuration options reside in the `drill-
+env.sh` and `drill-override.conf` files. Drill stores these files in the
+`/conf` directory. Drill sources` /etc/drill/conf` if it exists. Otherwise,
+Drill sources the local `<drill_installation_directory>/conf` directory.
+
+Refer to the following documentation for information about configuration
+options that you can modify:
+
+ * [Memory Allocation](/confluence/display/DRILL/Memory+Allocation)
+ * [Start-Up Options](/confluence/display/DRILL/Start-Up+Options)
+ * [Planning and Execution Options](/confluence/display/DRILL/Planning+and+Execution+Options)
+ * [Persistent Configuration Storage](/confluence/display/DRILL/Persistent+Configuration+Storage)
+
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/manage/002-start-stop.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/manage/002-start-stop.md b/_docs/drill-docs/manage/002-start-stop.md
new file mode 100644
index 0000000..c533bd3
--- /dev/null
+++ b/_docs/drill-docs/manage/002-start-stop.md
@@ -0,0 +1,45 @@
+---
+title: "Starting/Stopping Drill"
+parent: "Manage Drill"
+---
+How you start Drill depends on the installation method you followed. If you
+installed Drill in embedded mode, invoking SQLLine automatically starts a
+Drillbit locally. If you installed Drill in distributed mode on one or
+multiple nodes in a cluster, you must start the Drillbit service and then
+invoke SQLLine. Once SQLLine starts, you can issue queries to Drill.
+
+### Starting a Drillbit
+
+If you installed Drill in embedded mode, you do not need to start the
+Drillbit.
+
+To start a Drillbit, navigate to the Drill installation directory, and issue
+the following command:
+
+`bin/drillbit.sh restart`
+
+### Invoking SQLLine/Connecting to a Schema
+
+SQLLine is used as the Drill shell. SQLLine connects to relational databases
+and executes SQL commands. You invoke SQLLine for Drill in embedded or
+distributed mode. If you want to connect directly to a particular schema, you
+can indicate the schema name when you invoke SQLLine.
+
+To start SQLLine, issue the appropriate command for your Drill installation
+type:
+
+<div class="table-wrap"><table class="confluenceTable"><tbody><tr><td class="confluenceTd"><p><strong>Drill Install Type</strong></p></td><td class="confluenceTd"><p><strong>Example</strong></p></td><td class="confluenceTd"><p><strong>Command</strong></p></td></tr><tr><td class="confluenceTd"><p>Embedded</p></td><td class="confluenceTd"><p>Drill installed locally (embedded mode);</p><p>Hive with embedded metastore</p></td><td class="confluenceTd"><p>To connect without specifying a schema, navigate to the Drill installation directory and issue the following command:</p><p><code>$ bin/sqlline -u jdbc:drill:zk=local -n admin -p admin </code><span> </span></p><p>Once you are in the prompt, you can issue<code> USE <schema> </code>or you can use absolute notation: <code>schema.table.column.</code></p><p>To connect to a schema directly, issue the command with the schema name:</p><p><code>$ bin/sqlline -u jdbc:drill:schema=<database>;zk=local -n admin -p admin</code></p></td></t
r><tr><td class="confluenceTd"><p>Distributed</p></td><td class="confluenceTd"><p>Drill installed in distributed mode;</p><p>Hive with remote metastore;</p><p>HBase</p></td><td class="confluenceTd"><p>To connect without specifying a schema, navigate to the Drill installation directory and issue the following command:</p><p><code>$ bin/sqlline -u jdbc:drill:zk=<zk1host>:<port>,<zk2host>:<port>,<zk3host>:<port> -n admin -p admin</code></p><p>Once you are in the prompt, you can issue<code> USE <schema> </code>or you can use absolute notation: <code>schema.table.column.</code></p><p>To connect to a schema directly, issue the command with the schema name:</p><p><code>$ bin/sqlline -u jdbc:drill:schema=<database>;zk=<zk1host>:<port>,<zk2host>:<port>,<zk3host>:<port> -n admin -p admin</code></p></td></tr></tbody></table></div>
+
+When SQLLine starts, the system displays the following prompt:
+
+`0: [jdbc:drill](http://jdbcdrill):schema=<database>;zk=<zkhost>:<port>>`
+
+At this point, you can use Drill to query your data source or you can discover
+metadata.
+
+### Exiting SQLLine
+
+To exit SQLLine, issue the following command:
+
+`!quit`
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/manage/003-ports.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/manage/003-ports.md b/_docs/drill-docs/manage/003-ports.md
new file mode 100644
index 0000000..70539de
--- /dev/null
+++ b/_docs/drill-docs/manage/003-ports.md
@@ -0,0 +1,9 @@
+---
+title: "Ports Used by Drill"
+parent: "Manage Drill"
+---
+The following table provides a list of the ports that Drill uses, the port
+type, and a description of how Drill uses the port:
+
+<div class="table-wrap"><table class="confluenceTable"><tbody><tr><th class="confluenceTh">Port</th><th colspan="1" class="confluenceTh">Type</th><th class="confluenceTh">Description</th></tr><tr><td valign="top" class="confluenceTd">8047</td><td valign="top" colspan="1" class="confluenceTd">TCP</td><td valign="top" class="confluenceTd">Needed for <span style="color: rgb(34,34,34);">the Drill Web UI.</span><span style="color: rgb(34,34,34);"> </span></td></tr><tr><td valign="top" class="confluenceTd">31010</td><td valign="top" colspan="1" class="confluenceTd">TCP</td><td valign="top" class="confluenceTd">User port address. Used between nodes in a Drill cluster. <br />Needed for an external client, such as Tableau, to connect into the<br />cluster nodes. Also needed for the Drill Web UI.</td></tr><tr><td valign="top" class="confluenceTd">31011</td><td valign="top" colspan="1" class="confluenceTd">TCP</td><td valign="top" class="confluenceTd">Control port address. Used between nodes i
n a Drill cluster. <br />Needed for multi-node installation of Apache Drill.</td></tr><tr><td valign="top" colspan="1" class="confluenceTd">31012</td><td valign="top" colspan="1" class="confluenceTd">TCP</td><td valign="top" colspan="1" class="confluenceTd">Data port address. Used between nodes in a Drill cluster. <br />Needed for multi-node installation of Apache Drill.</td></tr><tr><td valign="top" colspan="1" class="confluenceTd">46655</td><td valign="top" colspan="1" class="confluenceTd">UDP</td><td valign="top" colspan="1" class="confluenceTd">Used for JGroups and Infinispan. Needed for multi-node installation of Apache Drill.</td></tr></tbody></table></div>
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/manage/004-partition-prune.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/manage/004-partition-prune.md b/_docs/drill-docs/manage/004-partition-prune.md
new file mode 100644
index 0000000..fa81034
--- /dev/null
+++ b/_docs/drill-docs/manage/004-partition-prune.md
@@ -0,0 +1,75 @@
+---
+title: "Partition Pruning"
+parent: "Manage Drill"
+---
+Partition pruning is a performance optimization that limits the number of
+files and partitions that Drill reads when querying file systems and Hive
+tables. Drill only reads a subset of the files that reside in a file system or
+a subset of the partitions in a Hive table when a query matches certain filter
+criteria.
+
+For Drill to apply partition pruning to Hive tables, you must have created the
+tables in Hive using the `PARTITION BY` clause:
+
+`CREATE TABLE <table_name> (<column_name>) PARTITION BY (<column_name>);`
+
+When you create Hive tables using the `PARTITION BY` clause, each partition of
+data is automatically split out into different directories as data is written
+to disk. For more information about Hive partitioning, refer to the [Apache
+Hive wiki](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL/#LanguageManualDDL-PartitionedTables).
+
+Typically, table data in a file system is organized by directories and
+subdirectories. Queries on table data may contain `WHERE` clause filters on
+specific directories.
+
+Drill’s query planner evaluates the filters as part of a Filter operator. If
+no partition filters are present, the underlying Scan operator reads all files
+in all directories and then sends the data to operators downstream, such as
+Filter.
+
+When partition filters are present, the query planner determines if it can
+push the filters down to the Scan such that the Scan only reads the
+directories that match the partition filters, thus reducing disk I/O.
+
+## Partition Pruning Example
+
+The /`Users/max/data/logs` directory in a file system contains subdirectories
+that span a few years.
+
+The following image shows the hierarchical structure of the `…/logs` directory
+and (sub) directories:
+
+
+
+The following query requests log file data for 2013 from the `…/logs`
+directory in the file system:
+
+ SELECT * FROM dfs.`/Users/max/data/logs` WHERE cust_id < 10 and dir0 = 2013 limit 2;
+
+If you run the `EXPLAIN PLAN` command for the query, you can see that the`
+…/logs` directory is filtered by the scan operator.
+
+ EXPLAIN PLAN FOR SELECT * FROM dfs.`/Users/max/data/logs` WHERE cust_id < 10 and dir0 = 2013 limit 2;
+
+The following image shows a portion of the physical plan when partition
+pruning is applied:
+
+
+
+## Filter Examples
+
+The following queries include examples of the types of filters eligible for
+partition pruning optimization:
+
+**Example 1: Partition filters ANDed together**
+
+ SELECT * FROM dfs.`/Users/max/data/logs` WHERE dir0 = '2014' AND dir1 = '1'
+
+**Example 2: Partition filter ANDed with regular column filter**
+
+ SELECT * FROM dfs.`/Users/max/data/logs` WHERE cust_id < 10 AND dir0 = 2013 limit 2;
+
+**Example 3: Combination of AND, OR involving partition filters**
+
+ SELECT * FROM dfs.`/Users/max/data/logs` WHERE (dir0 = '2013' AND dir1 = '1') OR (dir0 = '2014' AND dir1 = '2')
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/manage/005-monitor-cancel.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/manage/005-monitor-cancel.md b/_docs/drill-docs/manage/005-monitor-cancel.md
new file mode 100644
index 0000000..6888eea
--- /dev/null
+++ b/_docs/drill-docs/manage/005-monitor-cancel.md
@@ -0,0 +1,30 @@
+---
+title: "Monitoring and Canceling Queries in the Drill Web UI"
+parent: "Manage Drill"
+---
+You can monitor and cancel queries from the Drill Web UI. To access the Drill
+Web UI, the Drillbit process must be running on the Drill node that you use to
+access the Drill Web UI.
+
+To monitor or cancel a query from the Drill Web UI, complete the following
+steps:
+
+ 1. Navigate to the Drill Web UI at `<drill_node_ip_address>:8047.`
+When you access the Drill Web UI, you see some general information about Drill
+running in your cluster, such as the nodes running the Drillbit process, the
+various ports Drill is using, and the amount of direct memory assigned to
+Drill.
+
+
+ 2. Select **Profiles** in the toolbar. A list of running and completed queries appears. Drill assigns a query ID to each query and lists the Foreman node. The Foreman is the Drillbit node that receives the query from the client or application. The Foreman drives the entire query.
+
+
+ 3. Click the **Query ID** for the query that you want to monitor or cancel. The Query and Planning window appears.
+
+
+ 4. Select **Edit Query**.
+ 5. Click **Cancel query **to cancel the** **query. The following message appears:
+
+
+ 6. Optionally, you can re-run the query to see a query summary in this window.
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/manage/conf/001-mem-alloc.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/manage/conf/001-mem-alloc.md b/_docs/drill-docs/manage/conf/001-mem-alloc.md
new file mode 100644
index 0000000..4508935
--- /dev/null
+++ b/_docs/drill-docs/manage/conf/001-mem-alloc.md
@@ -0,0 +1,31 @@
+---
+title: "Memory Allocation"
+parent: "Configuration Options"
+---
+You can configure the amount of direct memory allocated to a Drillbit for
+query processing. The default limit is 8G, but Drill prefers 16G or more
+depending on the workload. The total amount of direct memory that a Drillbit
+allocates to query operations cannot exceed the limit set.
+
+Drill mainly uses Java direct memory and performs well when executing
+operations in memory instead of storing the operations on disk. Drill does not
+write to disk unless absolutely necessary, unlike MapReduce where everything
+is written to disk during each phase of a job.
+
+The JVM’s heap memory does not limit the amount of direct memory available in
+a Drillbit. The on-heap memory for Drill is only about 4-8G, which should
+suffice because Drill avoids having data sit in heap memory.
+
+#### Modifying Drillbit Memory
+
+You can modify memory for each Drillbit node in your cluster. To modify the
+memory for a Drillbit, edit the `XX:MaxDirectMemorySize` parameter in the
+Drillbit startup script located in `<drill_installation_directory>/conf/drill-
+env.sh`.
+
+**Note:** If this parameter is not set, the limit depends on the amount of available system memory.
+
+After you edit `<drill_installation_directory>/conf/drill-env.sh`, [restart
+the Drillbit
+](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=44994063)on
+the node.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/manage/conf/002-startup-opt.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/manage/conf/002-startup-opt.md b/_docs/drill-docs/manage/conf/002-startup-opt.md
new file mode 100644
index 0000000..923139f
--- /dev/null
+++ b/_docs/drill-docs/manage/conf/002-startup-opt.md
@@ -0,0 +1,50 @@
+---
+title: "Start-Up Options"
+parent: "Configuration Options"
+---
+Drill’s start-up options reside in a HOCON configuration file format, which is
+a hybrid between a properties file and a JSON file. Drill start-up options
+consist of a group of files with a nested relationship. At the core of the
+file hierarchy is `drill-default.conf`. This file is overridden by one or more
+`drill-module.conf` files, which are overridden by the `drill-override.conf`
+file that you define.
+
+You can see the following group of files throughout the source repository in
+Drill:
+
+ common/src/main/resources/drill-default.conf
+ common/src/main/resources/drill-module.conf
+ contrib/storage-hbase/src/main/resources/drill-module.conf
+ contrib/storage-hive/core/src/main/resources/drill-module.conf
+ contrib/storage-hive/hive-exec-shade/src/main/resources/drill-module.conf
+ exec/java-exec/src/main/resources/drill-module.conf
+ distribution/src/resources/drill-override.conf
+
+These files are listed inside the associated JAR files in the Drill
+distribution tarball.
+
+Each Drill module has a set of options that Drill incorporates. Drill’s
+modular design enables you to create new storage plugins, set new operators,
+or create UDFs. You can also include additional configuration options that you
+can override as necessary.
+
+When you add a JAR file to Drill, you must include a `drill-module.conf` file
+in the root directory of the JAR file that you add. The `drill-module.conf`
+file tells Drill to scan that JAR file or associated object and include it.
+
+#### Viewing Startup Options
+
+You can run the following query to see a list of Drill’s startup options:
+
+ SELECT * FROM sys.options WHERE type='BOOT'
+
+#### Configuring Start-Up Options
+
+You can configure start-up options for each Drillbit in the `drill-
+override.conf` file located in Drill’s` /conf` directory.
+
+You may want to configure the following start-up options that control certain
+behaviors in Drill:
+
+<div class="table-wrap"><table class="confluenceTable"><tbody><tr><th class="confluenceTh">Option</th><th class="confluenceTh">Default Value</th><th class="confluenceTh">Description</th></tr><tr><td valign="top" class="confluenceTd"><p>drill.exec.sys.store.provider</p></td><td valign="top" class="confluenceTd"><p>ZooKeeper</p></td><td valign="top" class="confluenceTd"><p>Defines the persistent storage (PStore) provider. The PStore holds configuration and profile data. For more information about PStores, see <a href="https://cwiki.apache.org/confluence/display/DRILL/Persistent+Configuration+Storage" rel="nofollow">Persistent Configuration Storage</a>.</p></td></tr><tr><td valign="top" class="confluenceTd"><p>drill.exec.buffer.size</p></td><td valign="top" class="confluenceTd"><p> </p></td><td valign="top" class="confluenceTd"><p>Defines the amount of memory available, in terms of record batches, to hold data on the downstream side of an operation. Drill pushes data downstream as quic
kly as possible to make data immediately available. This requires Drill to use memory to hold the data pending operations. When data on a downstream operation is required, that data is immediately available so Drill does not have to go over the network to process it. Providing more memory to this option increases the speed at which Drill completes a query.</p></td></tr><tr><td valign="top" class="confluenceTd"><p>drill.exec.sort.external.directories</p><p>drill.exec.sort.external.fs</p></td><td valign="top" class="confluenceTd"><p> </p></td><td valign="top" class="confluenceTd"><p>These options control spooling. The drill.exec.sort.external.directories option tells Drill which directory to use when spooling. The drill.exec.sort.external.fs option tells Drill which file system to use when spooling beyond memory files. <span style="line-height: 1.4285715;background-color: transparent;"> </span></p><p>Drill uses a spool and sort operation for beyond memory operations. The sorting opera
tion is designed to spool to a Hadoop file system. The default Hadoop file system is a local file system in the /tmp directory. Spooling performance (both writing and reading back from it) is constrained by the file system. <span style="line-height: 1.4285715;background-color: transparent;"> </span></p><p>For MapR clusters, use MapReduce volumes or set up local volumes to use for spooling purposes. Volumes improve performance and stripe data across as many disks as possible.</p></td></tr><tr><td valign="top" colspan="1" class="confluenceTd"><p>drill.exec.debug.error_on_leak</p></td><td valign="top" colspan="1" class="confluenceTd"><p>True</p></td><td valign="top" colspan="1" class="confluenceTd"><p>Determines how Drill behaves when memory leaks occur during a query. By default, this option is enabled so that queries fail when memory leaks occur. If you disable the option, Drill issues a warning when a memory leak occurs and completes the query.</p></td></tr><tr><td valign="top" cols
pan="1" class="confluenceTd"><p>drill.exec.zk.connect</p></td><td valign="top" colspan="1" class="confluenceTd"><p>localhost:2181</p></td><td valign="top" colspan="1" class="confluenceTd"><p>Provides Drill with the ZooKeeper quorum to use to connect to data sources. Change this setting to point to the ZooKeeper quorum that you want Drill to use. You must configure this option on each Drillbit node.</p></td></tr><tr><td valign="top" colspan="1" class="confluenceTd"><p>drill.exec.cluster-id</p></td><td valign="top" colspan="1" class="confluenceTd"><p>my_drillbit_cluster</p></td><td valign="top" colspan="1" class="confluenceTd"><p>Identifies the cluster that corresponds with the ZooKeeper quorum indicated. It also provides Drill with the name of the cluster used during UDP multicast. You must change the default cluster-id if there are multiple clusters on the same subnet. If you do not change the ID, the clusters will try to connect to each other to create one cluster.</p></td></tr></t
body></table></div>
+
[6/6] drill git commit: Added Drill docs
Posted by ts...@apache.org.
Added Drill docs
Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/84b7b36d
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/84b7b36d
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/84b7b36d
Branch: refs/heads/gh-pages
Commit: 84b7b36d96875c19cef3bd3395a26e9f129c5d53
Parents: c37bc59
Author: Tomer Shiran <ts...@gmail.com>
Authored: Wed Jan 14 21:05:10 2015 -0800
Committer: Tomer Shiran <ts...@gmail.com>
Committed: Wed Jan 14 21:05:10 2015 -0800
----------------------------------------------------------------------
_config.yml | 3 +
_docs/001-drill-docs.md | 4 +
_docs/001-user-guide.md | 4 -
_docs/002-admin-guide.md | 5 -
_docs/drill-docs/001-arch.md | 58 +++
_docs/drill-docs/002-tutorial.md | 58 +++
_docs/drill-docs/003-yelp.md | 402 ++++++++++++++++++
_docs/drill-docs/004-install.md | 20 +
_docs/drill-docs/005-connect.md | 49 +++
_docs/drill-docs/006-query.md | 57 +++
_docs/drill-docs/006-sql-ref.md | 25 ++
_docs/drill-docs/007-dev-custom-func.md | 47 +++
_docs/drill-docs/008-manage.md | 23 +
_docs/drill-docs/009-develop.md | 16 +
_docs/drill-docs/010-rn.md | 192 +++++++++
_docs/drill-docs/011-contribute.md | 11 +
_docs/drill-docs/012-sample-ds.md | 11 +
_docs/drill-docs/013-design.md | 14 +
_docs/drill-docs/014-progress.md | 9 +
_docs/drill-docs/015-archived-pages.md | 9 +
_docs/drill-docs/016-bylaws.md | 171 ++++++++
_docs/drill-docs/arch/001-core-mod.md | 30 ++
_docs/drill-docs/arch/002-arch-hilite.md | 15 +
.../arch/arch-hilite/001-flexibility.md | 79 ++++
.../arch/arch-hilite/002-performance.md | 56 +++
_docs/drill-docs/archive/001-how-to-demo.md | 309 ++++++++++++++
_docs/drill-docs/archive/002-meet-drill.md | 41 ++
_docs/drill-docs/connect/001-plugin-reg.md | 39 ++
_docs/drill-docs/connect/002-mongo-plugin.md | 169 ++++++++
_docs/drill-docs/connect/003-mapr-db-plugin.md | 32 ++
.../connect/workspaces/001-workspaces.md | 82 ++++
.../drill-docs/connect/workspaces/002-reg-fs.md | 69 +++
.../connect/workspaces/003-reg-hbase.md | 34 ++
.../connect/workspaces/004-reg-hive.md | 99 +++++
.../connect/workspaces/005-default-frmt.md | 61 +++
_docs/drill-docs/contribute/001-guidelines.md | 230 ++++++++++
_docs/drill-docs/contribute/002-ideas.md | 158 +++++++
_docs/drill-docs/datasets/001-aol.md | 47 +++
_docs/drill-docs/datasets/002-enron.md | 21 +
_docs/drill-docs/datasets/003-wikipedia.md | 105 +++++
_docs/drill-docs/design/001-plan.md | 25 ++
_docs/drill-docs/design/002-rpc.md | 19 +
_docs/drill-docs/design/003-query-stages.md | 42 ++
_docs/drill-docs/design/004-research.md | 48 +++
_docs/drill-docs/design/005-value.md | 191 +++++++++
.../drill-docs/dev-custom-fcn/001-dev-simple.md | 51 +++
.../dev-custom-fcn/002-dev-aggregate.md | 59 +++
.../drill-docs/dev-custom-fcn/003-add-custom.md | 28 ++
.../drill-docs/dev-custom-fcn/004-use-custom.md | 55 +++
.../dev-custom-fcn/005-cust-interface.md | 14 +
_docs/drill-docs/develop/001-compile.md | 37 ++
_docs/drill-docs/develop/002-setup.md | 5 +
_docs/drill-docs/develop/003-patch-tool.md | 160 +++++++
_docs/drill-docs/install/001-drill-in-10.md | 395 +++++++++++++++++
_docs/drill-docs/install/002-deploy.md | 102 +++++
.../drill-docs/install/003-install-embedded.md | 30 ++
.../install/004-install-distributed.md | 61 +++
.../install-embedded/001-install-linux.md | 30 ++
.../install/install-embedded/002-install-mac.md | 33 ++
.../install/install-embedded/003-install-win.md | 57 +++
_docs/drill-docs/manage/001-conf.md | 20 +
_docs/drill-docs/manage/002-start-stop.md | 45 ++
_docs/drill-docs/manage/003-ports.md | 9 +
_docs/drill-docs/manage/004-partition-prune.md | 75 ++++
_docs/drill-docs/manage/005-monitor-cancel.md | 30 ++
_docs/drill-docs/manage/conf/001-mem-alloc.md | 31 ++
_docs/drill-docs/manage/conf/002-startup-opt.md | 50 +++
_docs/drill-docs/manage/conf/003-plan-exec.md | 37 ++
.../drill-docs/manage/conf/004-persist-conf.md | 93 ++++
_docs/drill-docs/progress/001-2014-q1.md | 204 +++++++++
_docs/drill-docs/query/001-query-fs.md | 44 ++
_docs/drill-docs/query/002-query-hbase.md | 177 ++++++++
_docs/drill-docs/query/003-query-hive.md | 67 +++
_docs/drill-docs/query/004-query-complex.md | 63 +++
_docs/drill-docs/query/005-query-info-skema.md | 109 +++++
_docs/drill-docs/query/006-query-sys-tbl.md | 176 ++++++++
_docs/drill-docs/query/007-interfaces.md | 16 +
_docs/drill-docs/query/interfaces/001-jdbc.md | 138 ++++++
_docs/drill-docs/query/interfaces/002-odbc.md | 23 +
.../query/query-complex/001-sample-donuts.md | 40 ++
.../query/query-complex/002-query1-select.md | 19 +
.../query/query-complex/003-query2-use-sql.md | 74 ++++
.../query/query-complex/004-query3-sel-nest.md | 50 +++
.../query-complex/005-query4-sel-multiple.md | 24 ++
.../drill-docs/query/query-fs/001-query-json.md | 41 ++
.../query/query-fs/002-query-parquet.md | 99 +++++
.../drill-docs/query/query-fs/003-query-text.md | 120 ++++++
.../drill-docs/query/query-fs/004-query-dir.md | 90 ++++
_docs/drill-docs/rn/001-0.5.0rn.md | 29 ++
_docs/drill-docs/rn/002-0.4.0rn.md | 42 ++
_docs/drill-docs/rn/003-alpha-rn.md | 44 ++
_docs/drill-docs/rn/004-0.6.0-rn.md | 32 ++
_docs/drill-docs/rn/005-0.7.0-rn.md | 56 +++
_docs/drill-docs/sql-ref/001-data-types.md | 96 +++++
_docs/drill-docs/sql-ref/002-operators.md | 71 ++++
_docs/drill-docs/sql-ref/003-functions.md | 185 ++++++++
_docs/drill-docs/sql-ref/004-nest-functions.md | 10 +
_docs/drill-docs/sql-ref/005-cmd-summary.md | 16 +
_docs/drill-docs/sql-ref/006-reserved-wds.md | 16 +
.../sql-ref/cmd-summary/001-create-table-as.md | 134 ++++++
.../sql-ref/cmd-summary/002-explain.md | 166 ++++++++
.../sql-ref/cmd-summary/003-select.md | 85 ++++
.../sql-ref/cmd-summary/004-show-files.md | 65 +++
_docs/drill-docs/sql-ref/data-types/001-date.md | 148 +++++++
_docs/drill-docs/sql-ref/nested/001-flatten.md | 89 ++++
_docs/drill-docs/sql-ref/nested/002-kvgen.md | 150 +++++++
.../sql-ref/nested/003-repeated-cnt.md | 34 ++
.../drill-docs/tutorial/001-install-sandbox.md | 56 +++
_docs/drill-docs/tutorial/002-get2kno-sb.md | 235 +++++++++++
_docs/drill-docs/tutorial/003-lesson1.md | 423 +++++++++++++++++++
_docs/drill-docs/tutorial/004-lesson2.md | 392 +++++++++++++++++
_docs/drill-docs/tutorial/005-lesson3.md | 379 +++++++++++++++++
_docs/drill-docs/tutorial/006-summary.md | 14 +
.../install-sandbox/001-install-mapr-vm.md | 55 +++
.../install-sandbox/002-install-mapr-vb.md | 72 ++++
_docs/img/11.png | Bin 0 -> 5224 bytes
_docs/img/18.png | Bin 0 -> 22253 bytes
_docs/img/19.png | Bin 0 -> 22248 bytes
_docs/img/21.png | Bin 0 -> 23385 bytes
_docs/img/30.png | Bin 0 -> 50936 bytes
_docs/img/4.png | Bin 0 -> 41555 bytes
_docs/img/40.png | Bin 0 -> 25898 bytes
_docs/img/42.png | Bin 0 -> 153938 bytes
_docs/img/46.png | Bin 0 -> 3597 bytes
_docs/img/51.png | Bin 0 -> 64520 bytes
_docs/img/52.png | Bin 0 -> 21243 bytes
_docs/img/53.png | Bin 0 -> 52269 bytes
_docs/img/54.png | Bin 0 -> 10704 bytes
_docs/img/7.png | Bin 0 -> 30755 bytes
_docs/img/DrillWebUI.png | Bin 0 -> 53187 bytes
_docs/img/DrillbitModules.png | Bin 0 -> 54907 bytes
_docs/img/Overview.png | Bin 0 -> 165981 bytes
_docs/img/StoragePluginConfig.png | Bin 0 -> 20403 bytes
_docs/img/drill-runtime.png | Bin 0 -> 78592 bytes
_docs/img/drill2.png | Bin 0 -> 22806 bytes
_docs/img/example_query.png | Bin 0 -> 63294 bytes
_docs/img/loginSandBox.png | Bin 0 -> 53970 bytes
_docs/img/queryFlow.png | Bin 0 -> 37031 bytes
_docs/img/slide-15-638.png | Bin 0 -> 58477 bytes
_docs/img/storageplugin.png | Bin 0 -> 7589 bytes
_docs/img/value1.png | Bin 0 -> 10017 bytes
_docs/img/value2.png | Bin 0 -> 13935 bytes
_docs/img/value3.png | Bin 0 -> 14694 bytes
_docs/img/value4.png | Bin 0 -> 20297 bytes
_docs/img/value5.png | Bin 0 -> 44110 bytes
_docs/img/value6.png | Bin 0 -> 13854 bytes
_docs/img/value7.png | Bin 0 -> 21719 bytes
_docs/img/vbApplSettings.png | Bin 0 -> 45140 bytes
_docs/img/vbEthernet.png | Bin 0 -> 31861 bytes
_docs/img/vbGenSettings.png | Bin 0 -> 56436 bytes
_docs/img/vbImport.png | Bin 0 -> 29075 bytes
_docs/img/vbMaprSetting.png | Bin 0 -> 56436 bytes
_docs/img/vbNetwork.png | Bin 0 -> 32117 bytes
_docs/img/vbloginSandBox.png | Bin 0 -> 52169 bytes
_docs/img/vmLibrary.png | Bin 0 -> 68085 bytes
_docs/img/vmShare.png | Bin 0 -> 49069 bytes
_docs/img/vmWelcome.png | Bin 0 -> 39255 bytes
_docs/user-guide/001-views.md | 5 -
_docs/user-guide/002-sql-syntax.md | 5 -
_docs/user-guide/sql-syntax/001-ddd-ddd.md | 7 -
_docs/user-guide/views/001-aaa-aaa.md | 7 -
_docs/user-guide/views/002-bbb-bbb.md | 7 -
_docs/user-guide/views/003-ccc-ccc.md | 7 -
163 files changed, 9355 insertions(+), 47 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_config.yml
----------------------------------------------------------------------
diff --git a/_config.yml b/_config.yml
index aeba641..3b254bd 100644
--- a/_config.yml
+++ b/_config.yml
@@ -13,6 +13,9 @@ baseurl: "/drill" # Base URL when hosted on GitHub Pages (drill repository under
noindex: 1
markdown: redcarpet
+redcarpet:
+ extensions: ["no_intra_emphasis", "fenced_code_blocks", "autolink", "tables", "with_toc_data"]
+
collections:
docs:
output: true
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/001-drill-docs.md
----------------------------------------------------------------------
diff --git a/_docs/001-drill-docs.md b/_docs/001-drill-docs.md
new file mode 100644
index 0000000..382e2e1
--- /dev/null
+++ b/_docs/001-drill-docs.md
@@ -0,0 +1,4 @@
+---
+title: "Apache Drill Documentation"
+---
+The Drill documentation covers how to install, configure, and use Apache Drill.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/001-user-guide.md
----------------------------------------------------------------------
diff --git a/_docs/001-user-guide.md b/_docs/001-user-guide.md
deleted file mode 100644
index be161df..0000000
--- a/_docs/001-user-guide.md
+++ /dev/null
@@ -1,4 +0,0 @@
----
-title: "User Guide"
----
-This is a user guide!
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/002-admin-guide.md
----------------------------------------------------------------------
diff --git a/_docs/002-admin-guide.md b/_docs/002-admin-guide.md
deleted file mode 100644
index ad1ce75..0000000
--- a/_docs/002-admin-guide.md
+++ /dev/null
@@ -1,5 +0,0 @@
----
-title: "Admin Guide"
-nocontent: true
----
-This is an Admin Guide...
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/001-arch.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/001-arch.md b/_docs/drill-docs/001-arch.md
new file mode 100644
index 0000000..e4b26fc
--- /dev/null
+++ b/_docs/drill-docs/001-arch.md
@@ -0,0 +1,58 @@
+---
+title: "Architectural Overview"
+parent: "Apache Drill Documentation"
+---
+Apache Drill is a low latency distributed query engine for large-scale
+datasets, including structured and semi-structured/nested data. Inspired by
+Google’s Dremel, Drill is designed to scale to several thousands of nodes and
+query petabytes of data at interactive speeds that BI/Analytics environments
+require.
+
+### High-Level Architecture
+
+Drill includes a distributed execution environment, purpose built for large-
+scale data processing. At the core of Apache Drill is the ‘Drillbit’ service,
+which is responsible for accepting requests from the client, processing the
+queries, and returning results to the client.
+
+A Drillbit service can be installed and run on all of the required nodes in a
+Hadoop cluster to form a distributed cluster environment. When a Drillbit runs
+on each data node in the cluster, Drill can maximize data locality during
+query execution without moving data over the network or between nodes. Drill
+uses ZooKeeper to maintain cluster membership and health-check information.
+
+Though Drill works in a Hadoop cluster environment, Drill is not tied to
+Hadoop and can run in any distributed cluster environment. The only pre-
+requisite for Drill is Zookeeper.
+
+### Query Flow in Drill
+
+The following image represents the flow of a Drill query:
+
+
+
+The flow of a Drill query typically involves the following steps:
+
+ 1. The Drill client issues a query. Any Drillbit in the cluster can accept queries from clients. There is no master-slave concept.
+ 2. The Drillbit then parses the query, optimizes it, and generates an optimized distributed query plan for fast and efficient execution.
+ 3. The Drillbit that accepts the query becomes the driving Drillbit node for the request. It gets a list of available Drillbit nodes in the cluster from ZooKeeper. The driving Drillbit determines the appropriate nodes to execute various query plan fragments to maximize data locality.
+ 4. The Drillbit schedules the execution of query fragments on individual nodes according to the execution plan.
+ 5. The individual nodes finish their execution and return data to the driving Drillbit.
+ 6. The driving Drillbit returns results back to the client.
+
+### Drill Clients
+
+You can access Drill through the following interfaces:
+
+ * Drill shell (SQLLine)
+ * Drill Web UI
+ * ODBC
+ * JDBC
+ * C++ API
+
+Click on either of the following links to continue reading about Drill's
+architecture:
+
+ * [Core Modules within a Drillbit](/confluence/display/DRILL/Core+Modules+within+a+Drillbit)
+ * [Architectural Highlights](/confluence/display/DRILL/Architectural+Highlights)
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/002-tutorial.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/002-tutorial.md b/_docs/drill-docs/002-tutorial.md
new file mode 100644
index 0000000..597f994
--- /dev/null
+++ b/_docs/drill-docs/002-tutorial.md
@@ -0,0 +1,58 @@
+---
+title: "Apache Drill Tutorial"
+parent: "Apache Drill Documentation"
+---
+This tutorial uses the MapR Sandbox, which is a Hadoop environment pre-
+configured with Apache Drill.
+
+To complete the tutorial on the MapR Sandbox with Apache Drill, work through
+the following pages in order:
+
+ * [Installing the Apache Drill Sandbox](/confluence/display/DRILL/Installing+the+Apache+Drill+Sandbox)
+ * [Getting to Know the Drill Setup](/confluence/display/DRILL/Getting+to+Know+the+Drill+Setup)
+ * [Lesson 1: Learn About the Data Set](/confluence/display/DRILL/Lesson+1%3A+Learn+About+the+Data+Set)
+ * [Lesson 2: Run Queries with ANSI SQL](/confluence/display/DRILL/Lesson+2%3A+Run+Queries+with+ANSI+SQL)
+ * [Lesson 3: Run Queries on Complex Data Types](/confluence/display/DRILL/Lesson+3%3A+Run+Queries+on+Complex+Data+Types)
+ * [Summary](/confluence/display/DRILL/Summary)
+
+# About Apache Drill
+
+Drill is an Apache open-source SQL query engine for Big Data exploration.
+Drill is designed from the ground up to support high-performance analysis on
+the semi-structured and rapidly evolving data coming from modern Big Data
+applications, while still providing the familiarity and ecosystem of ANSI SQL,
+the industry-standard query language. Drill provides plug-and-play integration
+with existing Apache Hive and Apache HBase deployments.Apache Drill 0.5 offers
+the following key features:
+
+ * Low-latency SQL queries
+
+ * Dynamic queries on self-describing data in files (such as JSON, Parquet, text) and MapR-DB/HBase tables, without requiring metadata definitions in the Hive metastore.
+
+ * ANSI SQL
+
+ * Nested data support
+
+ * Integration with Apache Hive (queries on Hive tables and views, support for all Hive file formats and Hive UDFs)
+
+ * BI/SQL tool integration using standard JDBC/ODBC drivers
+
+# MapR Sandbox with Apache Drill
+
+MapR includes Apache Drill as part of the Hadoop distribution. The MapR
+Sandbox with Apache Drill is a fully functional single-node cluster that can
+be used to get an overview on Apache Drill in a Hadoop environment. Business
+and technical analysts, product managers, and developers can use the sandbox
+environment to get a feel for the power and capabilities of Apache Drill by
+performing various types of queries. Once you get a flavor for the technology,
+refer to the [Apache Drill web site](http://incubator.apache.org/drill/) and
+[Apache Drill documentation
+](https://cwiki.apache.org/confluence/display/DRILL/Apache+Drill+Wiki)for more
+details.
+
+Note that Hadoop is not a prerequisite for Drill and users can start ramping
+up with Drill by running SQL queries directly on the local file system. Refer
+to [Apache Drill in 10 minutes](https://cwiki.apache.org/confluence/display/DR
+ILL/Apache+Drill+in+10+Minutes) for an introduction to using Drill in local
+(embedded) mode.
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/003-yelp.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/003-yelp.md b/_docs/drill-docs/003-yelp.md
new file mode 100644
index 0000000..b9339ed
--- /dev/null
+++ b/_docs/drill-docs/003-yelp.md
@@ -0,0 +1,402 @@
+---
+title: "Analyzing Yelp JSON Data with Apache Drill"
+parent: "Apache Drill Documentation"
+---
+[Apache Drill](https://www.mapr.com/products/apache-drill) is one of the
+fastest growing open source projects, with the community making rapid progress
+with monthly releases. The key difference is Drill’s agility and flexibility.
+Along with meeting the table stakes for SQL-on-Hadoop, which is to achieve low
+latency performance at scale, Drill allows users to analyze the data without
+any ETL or up-front schema definitions. The data could be in any file format
+such as text, JSON, or Parquet. Data could have simple types such as string,
+integer, dates, or more complex multi-structured data, such as nested maps and
+arrays. Data can exist in any file system, local or distributed, such as HDFS,
+[MapR FS](https://www.mapr.com/blog/comparing-mapr-fs-and-hdfs-nfs-and-
+snapshots), or S3. Drill, has a “no schema” approach, which enables you to get
+value from your data in just a few minutes.
+
+Let’s quickly walk through the steps required to install Drill and run it
+against the Yelp data set. The publicly available data set used for this
+example is downloadable from [Yelp](http://www.yelp.com/dataset_challenge)
+(business reviews) and is in JSON format.
+
+## Installing and Starting Drill
+
+### Step 1: Download Apache Drill onto your local machine
+
+[http://incubator.apache.org/drill/download/](http://incubator.apache.org/drill/download/)
+
+You can also [deploy Drill in clustered mode](https://cwiki.apache.org/conflue
+nce/display/DRILL/Deploying+Apache+Drill+in+a+Clustered+Environment) if you
+want to scale your environment.
+
+### Step 2 : Open the Drill tar file
+
+`tar -xvf apache-drill-0.6.0-incubating.tar`
+
+### Step 3: Launch sqlline, a JDBC application that ships with Drill
+
+`bin/sqlline -u jdbc:drill:zk=local`
+
+That’s it! You are now ready explore the data.
+
+Let’s try out some SQL examples to understand how Drill makes the raw data
+analysis extremely easy.
+
+**Note**: You need to substitute your local path to the Yelp data set in the FROM clause of each query you run.
+
+## Querying Data with Drill
+
+### **1\. View the contents of the Yelp business data**
+
+`0: jdbc:drill:zk=local> !set maxwidth 10000`
+
+``0: jdbc:drill:zk=local> select * from
+dfs.`/users/nrentachintala/Downloads/yelp/yelp_academic_dataset_business.json`
+limit 1;``
+
+ +-------------+--------------+------------+------------+------------+------------+--------------+------------+------------+------------+------------+------------+------------+------------+---------------+
+ | business_id | full_address | hours | open | categories | city | review_count | name | longitude | state | stars | latitude | attributes | type | neighborhoods |
+ +-------------+--------------+------------+------------+------------+------------+--------------+------------+------------+------------+------------+------------+------------+------------+---------------+
+ | vcNAWiLM4dR7D2nwwJ7nCA | 4840 E Indian School Rd
+ Ste 101
+ Phoenix, AZ 85018 | {"Tuesday":{"close":"17:00","open":"08:00"},"Friday":{"close":"17:00","open":"08:00"},"Monday":{"close":"17:00","open":"08:00"},"Wednesday":{"close":"17:00","open":"08:00"},"Thursday":{"close":"17:00","open":"08:00"},"Sunday":{},"Saturday":{}} | true | ["Doctors","Health & Medical"] | Phoenix | 7 | Eric Goldberg, MD | -111.983758 | AZ | 3.5 | 33.499313 | {"By Appointment Only":true,"Good For":{},"Ambience":{},"Parking":{},"Music":{},"Hair Types Specialized In":{},"Payment Types":{},"Dietary Restrictions":{}} | business | [] |
+ +-------------+--------------+------------+------------+------------+------------+--------------+------------+------------+------------+------------+------------+------------+------------+---------------+
+
+**Note: **You can directly query self-describing files such as JSON, Parquet, and text. There is no need to create metadata definitions in the Hive metastore.
+
+### **2\. Explore the business data set further**
+
+#### Total reviews in the data set
+
+``0: jdbc:drill:zk=local> select sum(review_count) as totalreviews from
+dfs.`/users/nrentachintala/Downloads/yelp/yelp_academic_dataset_business.json`
+;``
+
+ +--------------+
+ | totalreviews |
+ +--------------+
+ | 1236445 |
+ +--------------+
+
+#### Top states and cities in total number of reviews
+
+``0: jdbc:drill:zk=local> select state, city, count(*) totalreviews from
+dfs.`/users/nrentachintala/Downloads/yelp/yelp_academic_dataset_business.json`
+group by state, city order by count(*) desc limit 10;``
+
+ +------------+------------+--------------+
+ | state | city | totalreviews |
+ +------------+------------+--------------+
+ | NV | Las Vegas | 12021 |
+ | AZ | Phoenix | 7499 |
+ | AZ | Scottsdale | 3605 |
+ | EDH | Edinburgh | 2804 |
+ | AZ | Mesa | 2041 |
+ | AZ | Tempe | 2025 |
+ | NV | Henderson | 1914 |
+ | AZ | Chandler | 1637 |
+ | WI | Madison | 1630 |
+ | AZ | Glendale | 1196 |
+ +------------+------------+--------------+
+
+#### **Average number of reviews per business star rating**
+
+``0: jdbc:drill:zk=local> select stars,trunc(avg(review_count)) reviewsavg from
+dfs.`/users/nrentachintala/Downloads/yelp/yelp_academic_dataset_business.json`
+group by stars order by stars desc;``
+
+ +------------+------------+
+ | stars | reviewsavg |
+ +------------+------------+
+ | 5.0 | 8.0 |
+ | 4.5 | 28.0 |
+ | 4.0 | 48.0 |
+ | 3.5 | 35.0 |
+ | 3.0 | 26.0 |
+ | 2.5 | 16.0 |
+ | 2.0 | 11.0 |
+ | 1.5 | 9.0 |
+ | 1.0 | 4.0 |
+ +------------+------------+
+
+#### **Top businesses with high review counts (> 1000)**
+
+``0: jdbc:drill:zk=local> select name, state, city, `review_count` from
+dfs.`/users/nrentachintala/Downloads/yelp/yelp_academic_dataset_business.json`
+where review_count > 1000 order by `review_count` desc limit 10;``
+
+ +------------+------------+------------+----------------------------+
+ | name | state | city | review_count |
+ +------------+------------+------------+----------------------------+
+ | Mon Ami Gabi | NV | Las Vegas | 4084 |
+ | Earl of Sandwich | NV | Las Vegas | 3655 |
+ | Wicked Spoon | NV | Las Vegas | 3408 |
+ | The Buffet | NV | Las Vegas | 2791 |
+ | Serendipity 3 | NV | Las Vegas | 2682 |
+ | Bouchon | NV | Las Vegas | 2419 |
+ | The Buffet at Bellagio | NV | Las Vegas | 2404 |
+ | Bacchanal Buffet | NV | Las Vegas | 2369 |
+ | The Cosmopolitan of Las Vegas | NV | Las Vegas | 2253 |
+ | Aria Hotel & Casino | NV | Las Vegas | 2224 |
+ +------------+------------+------------+----------------------------+
+
+#### **Saturday open and close times for a few businesses**
+
+``0: jdbc:drill:zk=local> select b.name, b.hours.Saturday.`open`,
+b.hours.Saturday.`close`
+from
+dfs.`/users/nrentachintala/Downloads/yelp/yelp_academic_dataset_business.json`
+b limit 10;``
+
+ +------------+------------+----------------------------+
+ | name | EXPR$1 | EXPR$2 |
+ +------------+------------+----------------------------+
+ | Eric Goldberg, MD | 08:00 | 17:00 |
+ | Pine Cone Restaurant | null | null |
+ | Deforest Family Restaurant | 06:00 | 22:00 |
+ | Culver's | 10:30 | 22:00 |
+ | Chang Jiang Chinese Kitchen| 11:00 | 22:00 |
+ | Charter Communications | null | null |
+ | Air Quality Systems | null | null |
+ | McFarland Public Library | 09:00 | 20:00 |
+ | Green Lantern Restaurant | 06:00 | 02:00 |
+ | Spartan Animal Hospital | 07:30 | 18:00 |
+ +------------+------------+----------------------------+
+
+** **Note how Drill can traverse and refer through multiple levels of nesting.
+
+### **3\. Get the amenities of each business in the data set**
+
+Note that the attributes column in the Yelp business data set has a different
+element for every row, representing that businesses can have separate
+amenities. Drill makes it easy to quickly access data sets with changing
+schemas.
+
+First, change Drill to work in all text mode (so we can take a look at all of
+the data).
+
+ 0: jdbc:drill:zk=local> alter system set `store.json.all_text_mode` = true;
+ +------------+-----------------------------------+
+ | ok | summary |
+ +------------+-----------------------------------+
+ | true | store.json.all_text_mode updated. |
+ +------------+-----------------------------------+
+
+Then, query the attribute’s data.
+
+ 0: jdbc:drill:zk=local> select attributes from dfs.`/users/nrentachintala/Downloads/yelp/yelp_academic_dataset_business.json` limit 10;
+ +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ | attributes |
+ +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ | {"By Appointment Only":"true","Good For":{},"Ambience":{},"Parking":{},"Music":{},"Hair Types Specialized In":{},"Payment Types":{},"Dietary Restrictions":{}} |
+ | {"Take-out":"true","Good For":{"dessert":"false","latenight":"false","lunch":"true","dinner":"false","breakfast":"false","brunch":"false"},"Caters":"false","Noise Level":"averag |
+ | {"Take-out":"true","Good For":{"dessert":"false","latenight":"false","lunch":"false","dinner":"false","breakfast":"false","brunch":"true"},"Caters":"false","Noise Level":"quiet" |
+ | {"Take-out":"true","Good For":{},"Takes Reservations":"false","Delivery":"false","Ambience":{},"Parking":{"garage":"false","street":"false","validated":"false","lot":"true","val |
+ | {"Take-out":"true","Good For":{},"Ambience":{},"Parking":{},"Has TV":"false","Outdoor Seating":"false","Attire":"casual","Music":{},"Hair Types Specialized In":{},"Payment Types |
+ | {"Good For":{},"Ambience":{},"Parking":{},"Music":{},"Hair Types Specialized In":{},"Payment Types":{},"Dietary Restrictions":{}} |
+ | {"Good For":{},"Ambience":{},"Parking":{},"Music":{},"Hair Types Specialized In":{},"Payment Types":{},"Dietary Restrictions":{}} |
+ | {"Good For":{},"Ambience":{},"Parking":{},"Wi-Fi":"free","Music":{},"Hair Types Specialized In":{},"Payment Types":{},"Dietary Restrictions":{}} |
+ | {"Take-out":"true","Good For":{"dessert":"false","latenight":"false","lunch":"false","dinner":"true","breakfast":"false","brunch":"false"},"Noise Level":"average","Takes Reserva |
+ | {"Good For":{},"Ambience":{},"Parking":{},"Music":{},"Hair Types Specialized In":{},"Payment Types":{},"Dietary Restrictions":{}} |
+ +------------+
+
+Turn off the all text mode so we can continue to perform arithmetic operations
+on data.
+
+ 0: jdbc:drill:zk=local> alter system set `store.json.all_text_mode` = false;
+ +------------+------------+
+ | ok | summary |
+ +------------+------------+
+ | true | store.json.all_text_mode updated. |
+
+**4\. Explore the restaurant businesses in the data set**
+
+#### **Number of restaurants in the data set**** **
+
+ 0: jdbc:drill:zk=local> select count(*) as TotalRestaurants from dfs.`/users/nrentachintala/Downloads/yelp/yelp_academic_dataset_business.json` where true=repeated_contains(categories,'Restaurants');
+ +------------------+
+ | TotalRestaurants |
+ +------------------+
+ | 14303 |
+ +------------------+
+
+#### **Top restaurants in number of reviews**
+
+ 0: jdbc:drill:zk=local> select name,state,city,`review_count` from dfs.`/users/nrentachintala/Downloads/yelp/yelp_academic_dataset_business.json` where true=repeated_contains(categories,'Restaurants') order by `review_count` desc limit 10
+ . . . . . . . . . . . > ;
+ +------------+------------+------------+--------------+
+ | name | state | city | review_count |
+ +------------+------------+------------+--------------+
+ | Mon Ami Gabi | NV | Las Vegas | 4084 |
+ | Earl of Sandwich | NV | Las Vegas | 3655 |
+ | Wicked Spoon | NV | Las Vegas | 3408 |
+ | The Buffet | NV | Las Vegas | 2791 |
+ | Serendipity 3 | NV | Las Vegas | 2682 |
+ | Bouchon | NV | Las Vegas | 2419 |
+ | The Buffet at Bellagio | NV | Las Vegas | 2404 |
+ | Bacchanal Buffet | NV | Las Vegas | 2369 |
+ | Hash House A Go Go | NV | Las Vegas | 2201 |
+ | Mesa Grill | NV | Las Vegas | 2004 |
+ +------------+------------+------------+--------------+
+
+**Top restaurants in number of listed categories**
+
+ 0: jdbc:drill:zk=local> select name,repeated_count(categories) as categorycount, categories from dfs.`/users/nrentachintala/Downloads/yelp/yelp_academic_dataset_business.json` where true=repeated_contains(categories,'Restaurants') order by repeated_count(categories) desc limit 10;
+ +------------+---------------+------------+
+ | name | categorycount | categories |
+ +------------+---------------+------------+
+ | Binion's Hotel & Casino | 10 | ["Arts & Entertainment","Restaurants","Bars","Casinos","Event Planning & Services","Lounges","Nightlife","Hotels & Travel","American (N |
+ | Stage Deli | 10 | ["Arts & Entertainment","Food","Hotels","Desserts","Delis","Casinos","Sandwiches","Hotels & Travel","Restaurants","Event Planning & Services"] |
+ | Jillian's | 9 | ["Arts & Entertainment","American (Traditional)","Music Venues","Bars","Dance Clubs","Nightlife","Bowling","Active Life","Restaurants"] |
+ | Hotel Chocolat | 9 | ["Coffee & Tea","Food","Cafes","Chocolatiers & Shops","Specialty Food","Event Planning & Services","Hotels & Travel","Hotels","Restaurants"] |
+ | Hotel du Vin & Bistro Edinburgh | 9 | ["Modern European","Bars","French","Wine Bars","Event Planning & Services","Nightlife","Hotels & Travel","Hotels","Restaurants" |
+ | Elixir | 9 | ["Arts & Entertainment","American (Traditional)","Music Venues","Bars","Cocktail Bars","Nightlife","American (New)","Local Flavor","Restaurants"] |
+ | Tocasierra Spa and Fitness | 8 | ["Beauty & Spas","Gyms","Medical Spas","Health & Medical","Fitness & Instruction","Active Life","Day Spas","Restaurants"] |
+ | Costa Del Sol At Sunset Station | 8 | ["Steakhouses","Mexican","Seafood","Event Planning & Services","Hotels & Travel","Italian","Restaurants","Hotels"] |
+ | Scottsdale Silverado Golf Club | 8 | ["Fashion","Shopping","Sporting Goods","Active Life","Golf","American (New)","Sports Wear","Restaurants"] |
+ | House of Blues | 8 | ["Arts & Entertainment","Music Venues","Restaurants","Hotels","Event Planning & Services","Hotels & Travel","American (New)","Nightlife"] |
+ +------------+---------------+------------+
+
+#### **Top first categories in number of review counts**
+
+ 0: jdbc:drill:zk=local> select categories[0], count(categories[0]) as categorycount from dfs.`/users/nrentachintala/Downloads/yelp_dataset_challenge_academic_dataset/yelp_academic_dataset_business.json` group by categories[0]
+ order by count(categories[0]) desc limit 10;
+ +------------+---------------+
+ | EXPR$0 | categorycount |
+ +------------+---------------+
+ | Food | 4294 |
+ | Shopping | 1885 |
+ | Active Life | 1676 |
+ | Bars | 1366 |
+ | Local Services | 1351 |
+ | Mexican | 1284 |
+ | Hotels & Travel | 1283 |
+ | Fast Food | 963 |
+ | Arts & Entertainment | 906 |
+ | Hair Salons | 901 |
+ +------------+---------------+
+
+**5\. Explore the Yelp reviews dataset and combine with the businesses.**** **
+
+#### **Take a look at the contents of the Yelp reviews dataset.**** **
+
+ 0: jdbc:drill:zk=local> select * from dfs.`/users/nrentachintala/Downloads/yelp/yelp_academic_dataset_review.json` limit 1;
+ +------------+------------+------------+------------+------------+------------+------------+-------------+
+ | votes | user_id | review_id | stars | date | text | type | business_id |
+ +------------+------------+------------+------------+------------+------------+------------+-------------+
+ | {"funny":0,"useful":2,"cool":1} | Xqd0DzHaiyRqVH3WRG7hzg | 15SdjuK7DmYqUAj6rjGowg | 5 | 2007-05-17 | dr. goldberg offers everything i look for in a general practitioner. he's nice and easy to talk to without being patronizing; he's always on time in seeing his patients; he's affiliated with a top-notch hospital (nyu) which my parents have explained to me is very important in case something happens and you need surgery; and you can get referrals to see specialists without having to see him first. really, what more do you need? i'm sitting here trying to think of any complaints i have about him, but i'm really drawing a blank. | review | vcNAWiLM4dR7D2nwwJ7nCA |
+ +------------+------------+------------+------------+------------+------------+------------+-------------+
+
+#### **Top businesses with cool rated reviews**
+
+Note that we are combining the Yelp business data set that has the overall
+review_count to the Yelp review data, which holds additional details on each
+of the reviews themselves.
+
+ 0: jdbc:drill:zk=local> Select b.name from dfs.`/users/nrentachintala/Downloads/yelp/yelp_academic_dataset_business.json` b where b.business_id in (SELECT r.business_id FROM dfs.`/users/nrentachintala/Downloads/yelp/yelp_academic_dataset_review.json` r
+ GROUP BY r.business_id having sum(r.votes.cool) > 2000 order by sum(r.votes.cool) desc);
+ +------------+
+ | name |
+ +------------+
+ | Earl of Sandwich |
+ | XS Nightclub |
+ | The Cosmopolitan of Las Vegas |
+ | Wicked Spoon |
+ +------------+
+
+**Create a view with the combined business and reviews data sets**
+
+Note that Drill views are lightweight, and can just be created in the local
+file system. Drill in standalone mode comes with a dfs.tmp workspace, which we
+can use to create views (or you can can define your own workspaces on a local
+or distributed file system). If you want to persist the data physically
+instead of in a logical view, you can use CREATE TABLE AS SELECT syntax.
+
+ 0: jdbc:drill:zk=local> create or replace view dfs.tmp.businessreviews as Select b.name,b.stars,b.state,b.city,r.votes.funny,r.votes.useful,r.votes.cool, r.`date` from dfs.`/users/nrentachintala/Downloads/yelp/yelp_academic_dataset_business.json` b , dfs.`/users/nrentachintala/Downloads/yelp/yelp_academic_dataset_review.json` r where r.business_id=b.business_id
+ +------------+------------+
+ | ok | summary |
+ +------------+------------+
+ | true | View 'businessreviews' created successfully in 'dfs.tmp' schema |
+ +------------+------------+
+
+Let’s get the total number of records from the view.
+
+ 0: jdbc:drill:zk=local> select count(*) as Total from dfs.tmp.businessreviews;
+ +------------+
+ | Total |
+ +------------+
+ | 1125458 |
+ +------------+
+
+In addition to these queries, you can get many more deeper insights using
+Drill’s [SQL functionality](https://cwiki.apache.org/confluence/display/DRILL/
+SQL+Reference). If you are not comfortable with writing queries manually, you
+can use a BI/Analytics tools such as Tableau/MicroStrategy to query raw
+files/Hive/HBase data or Drill-created views directly using Drill ODBC/JDBC
+drivers.
+
+The goal of Apache Drill is to provide the freedom and flexibility in
+exploring data in ways we have never seen before with SQL technologies. The
+community is working on more exciting features around nested data and
+supporting data with changing schemas in upcoming releases.
+
+As an example, a new FLATTEN function is in development (an upcoming feature
+in 0.7). This function can be used to dynamically rationalize semi-structured
+data so you can apply even deeper SQL functionality. Here is a sample query:
+
+#### **Get a flattened list of categories for each business**
+
+ 0: jdbc:drill:zk=local> select name, flatten(categories) as category from dfs.`/users/nrentachintala/Downloads/yelp/yelp_academic_dataset_business.json` limit 20;
+ +------------+------------+
+ | name | category |
+ +------------+------------+
+ | Eric Goldberg, MD | Doctors |
+ | Eric Goldberg, MD | Health & Medical |
+ | Pine Cone Restaurant | Restaurants |
+ | Deforest Family Restaurant | American (Traditional) |
+ | Deforest Family Restaurant | Restaurants |
+ | Culver's | Food |
+ | Culver's | Ice Cream & Frozen Yogurt |
+ | Culver's | Fast Food |
+ | Culver's | Restaurants |
+ | Chang Jiang Chinese Kitchen | Chinese |
+ | Chang Jiang Chinese Kitchen | Restaurants |
+ | Charter Communications | Television Stations |
+ | Charter Communications | Mass Media |
+ | Air Quality Systems | Home Services |
+ | Air Quality Systems | Heating & Air Conditioning/HVAC |
+ | McFarland Public Library | Libraries |
+ | McFarland Public Library | Public Services & Government |
+ | Green Lantern Restaurant | American (Traditional) |
+ | Green Lantern Restaurant | Restaurants |
+ | Spartan Animal Hospital | Veterinarians |
+ +------------+------------+
+
+**Top categories used in business reviews**
+
+ 0: jdbc:drill:zk=local> select celltbl.catl, count(celltbl.catl) categorycnt from (select flatten(categories) catl from dfs.`/users/nrentachintala/Downloads/yelp_dataset_challenge_academic_dataset/yelp_academic_dataset_business.json` ) celltbl group by celltbl.catl order by count(celltbl.catl) desc limit 10 ;
+ +------------+-------------+
+ | catl | categorycnt |
+ +------------+-------------+
+ | Restaurants | 14303 |
+ | Shopping | 6428 |
+ | Food | 5209 |
+ | Beauty & Spas | 3421 |
+ | Nightlife | 2870 |
+ | Bars | 2378 |
+ | Health & Medical | 2351 |
+ | Automotive | 2241 |
+ | Home Services | 1957 |
+ | Fashion | 1897 |
+ +------------+-------------+
+
+Stay tuned for more features and upcoming activities in the Drill community.
+
+To learn more about Drill, please refer to the following resources:
+
+ * Download Drill here:<http://incubator.apache.org/drill/download/>
+ * 10 reasons we think Drill is cool:<http://incubator.apache.org/drill/why-drill/>
+ * A simple 10-minute tutorial:<https://cwiki.apache.org/confluence/display/DRILL/Apache+Drill+in+10+Minutes>
+ * A more comprehensive tutorial:<https://cwiki.apache.org/confluence/display/DRILL/Apache+Drill+Tutorial>
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/004-install.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/004-install.md b/_docs/drill-docs/004-install.md
new file mode 100644
index 0000000..fe7578c
--- /dev/null
+++ b/_docs/drill-docs/004-install.md
@@ -0,0 +1,20 @@
+---
+title: "Install Drill"
+parent: "Apache Drill Documentation"
+---
+You can install Drill in embedded mode or in distributed mode. Installing
+Drill in embedded mode does not require any configuration, which means that
+you can quickly get started with Drill. If you want to use Drill in a
+clustered Hadoop environment, you can install Drill in distributed mode.
+Installing in distributed mode requires some configuration, however once you
+install you can connect Drill to your Hive, HBase, or distributed file system
+data sources and run queries on them.
+
+Click on any of the following links for more information about how to install
+Drill in embedded or distributed mode:
+
+ * [Apache Drill in 10 Minutes](/confluence/display/DRILL/Apache+Drill+in+10+Minutes)
+ * [Deploying Apache Drill in a Clustered Environment](/confluence/display/DRILL/Deploying+Apache+Drill+in+a+Clustered+Environment)
+ * [Installing Drill in Embedded Mode](/confluence/display/DRILL/Installing+Drill+in+Embedded+Mode)
+ * [Installing Drill in Distributed Mode](/confluence/display/DRILL/Installing+Drill+in+Distributed+Mode)
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/005-connect.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/005-connect.md b/_docs/drill-docs/005-connect.md
new file mode 100644
index 0000000..039fc78
--- /dev/null
+++ b/_docs/drill-docs/005-connect.md
@@ -0,0 +1,49 @@
+---
+title: "Connect to Data Sources"
+parent: "Apache Drill Documentation"
+---
+Apache Drill serves as a query layer that connects to data sources through
+storage plugins. Drill uses the storage plugins to interact with data sources.
+You can think of a storage plugin as a connection between Drill and a data
+source.
+
+The following image represents the storage plugin layer between Drill and a
+data source:
+
+
+
+Storage plugins provide the following information to Drill:
+
+ * Metadata available in the underlying data source
+ * Location of data
+ * Interfaces that Drill can use to read from and write to data sources
+ * A set of storage plugin optimization rules that assist with efficient and faster execution of Drill queries, such as pushdowns, statistics, and partition awareness
+
+Storage plugins perform scanner and writer functions, and inform the metadata
+repository of any known metadata, such as:
+
+ * Schema
+ * File size
+ * Data ordering
+ * Secondary indices
+ * Number of blocks
+
+Storage plugins inform the execution engine of any native capabilities, such
+as predicate pushdown, joins, and SQL.
+
+Drill provides storage plugins for files and HBase/M7. Drill also integrates
+with Hive through a storage plugin. Hive provides a metadata abstraction layer
+on top of files and HBase/M7.
+
+When you run Drill to query files in HBase/M7, Drill can perform direct
+queries on the data or go through Hive, if you have metadata defined there.
+Drill integrates with the Hive metastore for metadata and also uses a Hive
+SerDe for the deserialization of records. Drill does not invoke the Hive
+execution engine for any requests.
+
+For information about how to connect Drill to your data sources, refer to
+storage plugin registration:
+
+ * [Storage Plugin Registration](/confluence/display/DRILL/Storage+Plugin+Registration)
+ * [MongoDB Plugin for Apache Drill](/confluence/display/DRILL/MongoDB+Plugin+for+Apache+Drill)
+ * [MapR-DB Plugin for Apache Drill](/confluence/display/DRILL/MapR-DB+Plugin+for+Apache+Drill)
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/006-query.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/006-query.md b/_docs/drill-docs/006-query.md
new file mode 100644
index 0000000..4b4fda0
--- /dev/null
+++ b/_docs/drill-docs/006-query.md
@@ -0,0 +1,57 @@
+---
+title: "Query Data"
+parent: "Apache Drill Documentation"
+---
+You can query local and distributed file systems, Hive, and HBase data sources
+registered with Drill. If you connected directly to a particular schema when
+you invoked SQLLine, you can issue SQL queries against that schema. If you did
+not indicate a schema when you invoked SQLLine, you can issue the `USE
+<schema>` statement to run your queries against a particular schema. After you
+issue the `USE` statement, you can use absolute notation, such as
+`schema.table.column`.
+
+Click on any of the following links for information about various data source
+queries and examples:
+
+ * [Querying a File System](/confluence/display/DRILL/Querying+a+File+System)
+ * [Querying HBase](/confluence/display/DRILL/Querying+HBase)
+ * [Querying Hive](/confluence/display/DRILL/Querying+Hive)
+ * [Querying Complex Data](/confluence/display/DRILL/Querying+Complex+Data)
+ * [Querying the INFORMATION_SCHEMA](/confluence/display/DRILL/Querying+the+INFORMATION_SCHEMA)
+ * [Querying System Tables](/confluence/display/DRILL/Querying+System+Tables)
+ * [Drill Interfaces](/confluence/display/DRILL/Drill+Interfaces)
+
+You may need to use casting functions in some queries. For example, you may
+have to cast a string `"100"` to an integer in order to apply a math function
+or an aggregate function.
+
+You can use the EXPLAIN command to analyze errors and troubleshoot queries
+that do not run. For example, if you run into a casting error, the query plan
+text may help you isolate the problem.
+
+ 0: jdbc:drill:zk=local> !set maxwidth 10000
+ 0: jdbc:drill:zk=local> explain plan for select ... ;
+
+The set command increases the default text display (number of characters). By
+default, most of the plan output is hidden.
+
+You may see errors if you try to use non-standard or unsupported SQL syntax in
+a query.
+
+Remember the following tips when querying data with Drill:
+
+ * Include a semicolon at the end of SQL statements, except when you issue a command with an exclamation point `(!).
+`Example: `!set maxwidth 10000`
+
+ * Use backticks around file and directory names that contain special characters and also around reserved words when you query a file system .
+The following special characters require backticks:
+
+ * . (period)
+ * / (forward slash)
+ * _ (underscore)
+
+Example: ``SELECT * FROM dfs.default.`sample_data/my_sample.json`; ``
+
+ * `CAST` data to `VARCHAR` if an expression in a query returns `VARBINARY` as the result type in order to view the `VARBINARY` types as readable data. If you do not use the `CAST` function, Drill returns the results as byte data.
+Example: `CAST (VARBINARY_expr as VARCHAR(50))`
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/006-sql-ref.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/006-sql-ref.md b/_docs/drill-docs/006-sql-ref.md
new file mode 100644
index 0000000..8818ca3
--- /dev/null
+++ b/_docs/drill-docs/006-sql-ref.md
@@ -0,0 +1,25 @@
+---
+title: "Develop Custom Functions"
+parent: "Apache Drill Documentation"
+---
+Drill supports the ANSI standard for SQL. You can use SQL to query your Hive,
+HBase, and distributed file system data sources. Drill can discover the form
+of the data when you submit a query. You can query text files and nested data
+formats, such as JSON and Parquet. Drill provides special operators and
+functions that you can use to _drill down _into nested data formats.
+
+Drill queries do not require information about the data that you are trying to
+access, regardless of its source system or its schema and data types. The
+sweet spot for Apache Drill is a SQL query workload against "complex data":
+data made up of various types of records and fields, rather than data in a
+recognizable relational form (discrete rows and columns).
+
+Refer to the following SQL reference pages for more information:
+
+ * [Data Types](/confluence/display/DRILL/Data+Types)
+ * [Operators](/confluence/display/DRILL/Operators)
+ * [SQL Functions](/confluence/display/DRILL/SQL+Functions)
+ * [Nested Data Functions](/confluence/display/DRILL/Nested+Data+Functions)
+ * [SQL Commands Summary](/confluence/display/DRILL/SQL+Commands+Summary)
+ * [Reserved Keywords](/confluence/display/DRILL/Reserved+Keywords)
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/007-dev-custom-func.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/007-dev-custom-func.md b/_docs/drill-docs/007-dev-custom-func.md
new file mode 100644
index 0000000..9bc8e65
--- /dev/null
+++ b/_docs/drill-docs/007-dev-custom-func.md
@@ -0,0 +1,47 @@
+---
+title: "Develop Custom Functions"
+parent: "Apache Drill Documentation"
+---
+
+Drill provides a high performance Java API with interfaces that you can
+implement to develop simple and aggregate custom functions. Custom functions
+are reusable SQL functions that you develop in Java to encapsulate code that
+processes column values during a query. Custom functions can perform
+calculations and transformations that built-in SQL operators and functions do
+not provide. Custom functions are called from within a SQL statement, like a
+regular function, and return a single value.
+
+### Simple Function
+
+A simple function operates on a single row and produces a single row as the
+output. When you include a simple function in a query, the function is called
+once for each row in the result set. Mathematical and string functions are
+examples of simple functions.
+
+### Aggregate Function
+
+Aggregate functions differ from simple functions in the number of rows that
+they accept as input. An aggregate function operates on multiple input rows
+and produces a single row as output. The COUNT(), MAX(), SUM(), and AVG()
+functions are examples of aggregate functions. You can use an aggregate
+function in a query with a GROUP BY clause to produce a result set with a
+separate aggregate value for each combination of values from the GROUP BY
+clause.
+
+### Process
+
+To develop custom functions that you can use in your Drill queries, you must
+complete the following tasks:
+
+ 1. Create a Java program that implements Drill’s simple or aggregate interface, and compile a sources and a classes JAR file.
+ 2. Add the sources and classes JAR files to Drill’s classpath.
+ 3. Add the name of the package that contains the classes to Drill’s main configuration file, drill-override.conf.
+
+Click on one of the following links to learn how to create custom functions
+for Drill:
+
+ * [Developing a Simple Function](/confluence/display/DRILL/Developing+a+Simple+Function)
+ * [Developing an Aggregate Function](/confluence/display/DRILL/Developing+an+Aggregate+Function)
+ * [Adding Custom Functions to Drill](/confluence/display/DRILL/Adding+Custom+Functions+to+Drill)
+ * [Using Custom Functions in Queries](/confluence/display/DRILL/Using+Custom+Functions+in+Queries)
+ * [Custom Function Interfaces](/confluence/display/DRILL/Custom+Function+Interfaces)
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/008-manage.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/008-manage.md b/_docs/drill-docs/008-manage.md
new file mode 100644
index 0000000..e629b20
--- /dev/null
+++ b/_docs/drill-docs/008-manage.md
@@ -0,0 +1,23 @@
+---
+title: "Manage Drill"
+parent: "Apache Drill Documentation"
+---
+When using Drill, you may need to stop and restart a Drillbit on a node, or
+modify various options. For example, the default storage format for CTAS
+statements is Parquet. You can modify the default setting so that output data
+is stored in CSV or JSON format.
+
+You can use certain SQL commands to manage Drill from within the Drill shell
+(SQLLine). You can also modify Drill configuration options, such as memory
+allocation, in Drill's configuration files.
+
+Refer to the following documentation for information about managing Drill in
+your cluster:
+
+ * [Configuration Options](/confluence/display/DRILL/Configuration+Options)
+ * [Starting/Stopping Drill](/confluence/pages/viewpage.action?pageId=44994063)
+ * [Ports Used by Drill](/confluence/display/DRILL/Ports+Used+by+Drill)
+ * [Partition Pruning](/confluence/display/DRILL/Partition+Pruning)
+ * [Monitoring and Canceling Queries in the Drill Web UI](/confluence/display/DRILL/Monitoring+and+Canceling+Queries+in+the+Drill+Web+UI)
+
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/009-develop.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/009-develop.md b/_docs/drill-docs/009-develop.md
new file mode 100644
index 0000000..d95f986
--- /dev/null
+++ b/_docs/drill-docs/009-develop.md
@@ -0,0 +1,16 @@
+---
+title: "Develop Drill"
+parent: "Apache Drill Documentation"
+---
+To develop Drill, you compile Drill from source code and then set up a project
+in Eclipse for use as your development environment. To review or contribute to
+Drill code, you must complete the steps required to install and use the Drill
+patch review tool.
+
+For information about contributing to the Apache Drill project, you can refer
+to the following pages:
+
+ * [Compiling Drill from Source](/confluence/display/DRILL/Compiling+Drill+from+Source)
+ * [Setting Up Your Development Environment](/confluence/display/DRILL/Setting+Up+Your+Development+Environment)
+ * [Drill Patch Review Tool](/confluence/display/DRILL/Drill+Patch+Review+Tool)
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/010-rn.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/010-rn.md b/_docs/drill-docs/010-rn.md
new file mode 100644
index 0000000..f196714
--- /dev/null
+++ b/_docs/drill-docs/010-rn.md
@@ -0,0 +1,192 @@
+---
+title: "Release Notes"
+parent: "Apache Drill Documentation"
+---
+## Apache Drill 0.7.0 Release Notes
+
+Apache Drill 0.7.0, the third beta release for Drill, is designed to help
+enthusiasts start working and experimenting with Drill. It also continues the
+Drill monthly release cycle as we drive towards general availability.
+
+This release is available as
+[binary](http://www.apache.org/dyn/closer.cgi/drill/drill-0.7.0/apache-
+drill-0.7.0.tar.gz) and
+[source](http://www.apache.org/dyn/closer.cgi/drill/drill-0.7.0/apache-
+drill-0.7.0-src.tar.gz) tarballs that are compiled against Apache Hadoop.
+Drill has been tested against MapR, Cloudera, and Hortonworks Hadoop
+distributions. There are associated build profiles and JIRAs that can help you
+run Drill against your preferred distribution
+
+Apache Drill 0.7.0 Key Features
+
+ * No more dependency on UDP/Multicast - Making it possible for Drill to work well in the following scenarios:
+
+ * UDP multicast not enabled (as in EC2)
+
+ * Cluster spans multiple subnets
+
+ * Cluster has multihome configuration
+
+ * New functions to natively work with nested data - KVGen and Flatten
+
+ * Support for Hive 0.13 (Hive 0.12 with Drill is not supported any more)
+
+ * Improved performance when querying Hive tables and File system through partition pruning
+
+ * Improved performance for HBase with LIKE operator pushdown
+
+ * Improved memory management
+
+ * Drill web UI monitoring and query profile improvements
+
+ * Ability to parse files without explicit extensions using default storage format specification
+
+ * Fixes for dealing with complex/nested data objects in Parquet/JSON
+
+ * Fast schema return - Improved experience working with BI/query tools by returning metadata quickly
+
+ * Several hang related fixes
+
+ * Parquet writer fixes for handling large datasets
+
+ * Stability improvements in ODBC and JDBC drivers
+
+Apache Drill 0.7.0 Key Notes and Limitations
+
+ * The current release supports in-memory and beyond-memory execution. However, you must disable memory-intensive hash aggregate and hash join operations to leverage this functionality.
+ * While the Drill execution engine supports dynamic schema changes during the course of a query, some operators have yet to implement support for this behavior, such as Sort. Other operations, such as streaming aggregate, may have partial support that leads to unexpected results.
+
+## Apache Drill 0.6.0 Release Notes
+
+Apache Drill 0.6.0, the second beta release for Drill, is designed to help
+enthusiasts start working and experimenting with Drill. It also continues the
+Drill monthly release cycle as we drive towards general availability.
+
+This release is available as [binary](http://www.apache.org/dyn/closer.cgi/inc
+ubator/drill/drill-0.5.0-incubating/apache-drill-0.5.0-incubating.tar.gz) and
+[source](http://www.apache.org/dyn/closer.cgi/incubator/drill/drill-0.5.0-incu
+bating/apache-drill-0.5.0-incubating-src.tar.gz) tarballs that are compiled
+against Apache Hadoop. Drill has been tested against MapR, Cloudera, and
+Hortonworks Hadoop distributions. There are associated build profiles and
+JIRAs that can help you run Drill against your preferred distribution.
+
+Apache Drill 0.6.0 Key Features
+
+This release is primarily a bug fix release, with [more than 30 JIRAs closed](
+https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313820&vers
+ion=12327472), but there are some notable features:
+
+ * Direct ANSI SQL access to MongoDB, using the latest [MongoDB Plugin for Apache Drill](/confluence/display/DRILL/MongoDB+Plugin+for+Apache+Drill)
+ * Filesystem query performance improvements with partition pruning
+ * Ability to use the file system as a persistent store for query profiles and diagnostic information
+ * Window function support (alpha)
+
+Apache Drill 0.6.0 Key Notes and Limitations
+
+ * The current release supports in-memory and beyond-memory execution. However, you must disable memory-intensive hash aggregate and hash join operations to leverage this functionality.
+ * While the Drill execution engine supports dynamic schema changes during the course of a query, some operators have yet to implement support for this behavior, such as Sort. Other operations, such as streaming aggregate, may have partial support that leads to unexpected results.
+
+## Apache Drill 0.5.0 Release Notes
+
+Apache Drill 0.5.0, the first beta release for Drill, is designed to help
+enthusiasts start working and experimenting with Drill. It also continues the
+Drill monthly release cycle as we drive towards general availability.
+
+The 0.5.0 release is primarily a bug fix release, with [more than 100 JIRAs](h
+ttps://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313820&versi
+on=12324880) closed, but there are some notable features. For information
+about the features, see the [Apache Drill Blog for the 0.5.0
+release](https://blogs.apache.org/drill/entry/apache_drill_beta_release_see).
+
+This release is available as [binary](http://www.apache.org/dyn/closer.cgi/inc
+ubator/drill/drill-0.5.0-incubating/apache-drill-0.5.0-incubating.tar.gz) and
+[source](http://www.apache.org/dyn/closer.cgi/incubator/drill/drill-0.5.0-incu
+bating/apache-drill-0.5.0-incubating-src.tar.gz) tarballs that are compiled
+against Apache Hadoop. Drill has been tested against MapR, Cloudera, and
+Hortonworks Hadoop distributions. There are associated build profiles and
+JIRAs that can help you run Drill against your preferred distribution.
+
+Apache Drill 0.5.0 Key Notes and Limitations
+
+ * The current release supports in memory and beyond memory execution. However, you must disable memory-intensive hash aggregate and hash join operations to leverage this functionality.
+ * While the Drill execution engine supports dynamic schema changes during the course of a query, some operators have yet to implement support for this behavior, such as Sort. Others operations, such as streaming aggregate, may have partial support that leads to unexpected results.
+ * There are known issues with joining text files without using an intervening view. See [DRILL-1401](https://issues.apache.org/jira/browse/DRILL-1401) for more information.
+
+## Apache Drill 0.4.0 Release Notes
+
+The 0.4.0 release is a developer preview release, designed to help enthusiasts
+start to work with and experiment with Drill. It is the first Drill release
+that provides distributed query execution.
+
+This release is built upon [more than 800
+JIRAs](https://issues.apache.org/jira/browse/DRILL/fixforversion/12324963/).
+It is a pre-beta release on the way towards Drill. As a developer snapshot,
+the release contains a large number of outstanding bugs that will make some
+use cases challenging. Feel free to consult outstanding issues [targeted for
+the 0.5.0
+release](https://issues.apache.org/jira/browse/DRILL/fixforversion/12324880/)
+to see whether your use case is affected.
+
+To read more about this release and new features introduced, please view the
+[0.4.0 announcement blog
+entry](https://blogs.apache.org/drill/entry/announcing_apache_drill_0_4).
+
+The release is available as both [binary](http://www.apache.org/dyn/closer.cgi
+/incubator/drill/drill-0.4.0-incubating/apache-drill-0.4.0-incubating.tar.gz)
+and [source](http://www.apache.org/dyn/closer.cgi/incubator/drill/drill-0.4.0-
+incubating/apache-drill-0.4.0-incubating-src.tar.gz) tarballs. In both cases,
+these are compiled against Apache Hadoop. Drill has also been tested against
+MapR, Cloudera and Hortonworks Hadoop distributions and there are associated
+build profiles or JIRAs that can help you run against your preferred
+distribution.
+
+Some Key Notes & Limitations
+
+ * The current release supports in memory and beyond memory execution. However, users must disable memory-intensive hash aggregate and hash join operations to leverage this functionality.
+ * In many cases,merge join operations return incorrect results.
+ * Use of a local filter in a join “on” clause when using left, right or full outer joins may result in incorrect results.
+ * Because of known memory leaks and memory overrun issues you may need more memory and you may need to restart the system in some cases.
+ * Some types of complex expressions, especially those involving empty arrays may fail or return incorrect results.
+ * While the Drill execution engine supports dynamic schema changes during the course of a query, some operators have yet to implement support for this behavior (such as Sort). Others operations (such as streaming aggregate) may have partial support that leads to unexpected results.
+ * Protobuf, UDF, query plan interfaces and all interfaces are subject to change in incompatible ways.
+ * Multiplication of some types of DECIMAL(28+,*) will return incorrect result.
+
+## Apache Drill M1 -- Release Notes (Apache Drill Alpha)
+
+### Milestone 1 Goals
+
+The first release of Apache Drill is designed as a technology preview for
+people to better understand the architecture and vision. It is a functional
+release tying to piece together the key components of a next generation MPP
+query engine. It is designed to allow milestone 2 (M2) to focus on
+architectural analysis and performance optimization.
+
+ * Provide a new optimistic DAG execution engine for data analysis
+ * Build a new columnar shredded in-memory format and execution model that minimizes data serialization/deserialization costs and operator complexity
+ * Provide a model for runtime generated functions and relational operators that minimizes complexity and maximizes performance
+ * Support queries against columnar on disk format (Parquet) and JSON
+ * Support the most common set of standard SQL read-only phrases using ANSI standards. Includes: SELECT, FROM, WHERE, HAVING, ORDER, GROUP BY, IN, DISTINCT, LEFT JOIN, RIGHT JOIN, INNER JOIN
+ * Support schema-on-read querying and execution
+ * Build a set of columnar operation primitives including Merge Join, Sort, Streaming Aggregate, Filter, Selection Vector removal.
+ * Support unlimited level of subqueries and correlated subqueries
+ * Provided an extensible query-language agnostic JSON-base logical data flow syntax.
+ * Support complex data type manipulation via logical plan operations
+
+### Known Issues
+
+SQL Parsing
+Because Apache Drill is built to support late-bound changing schemas while SQL
+is statically typed, there are couple of special requirements that are
+required writing SQL queries. These are limited to the current release and
+will be correct in a future milestone release.
+
+ * All tables are exposed as a single map field that contains
+ * Drill Alpha doesn't support implicit or explicit casts outside those required above.
+ * Drill Alpha does not include, there are currently a couple of differences for how to write a query in In order to query against
+
+UDFs
+
+ * Drill currently supports simple and aggregate functions using scalar, repeated and
+ * Nested data support incomplete. Drill Alpha supports nested data structures as well repeated fields. However,
+ * asd
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/011-contribute.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/011-contribute.md b/_docs/drill-docs/011-contribute.md
new file mode 100644
index 0000000..282ab8a
--- /dev/null
+++ b/_docs/drill-docs/011-contribute.md
@@ -0,0 +1,11 @@
+---
+title: "Contribute to Drill"
+parent: "Apache Drill Documentation"
+---
+The Apache Drill community welcomes your support. Please read [Apache Drill
+Contribution Guidelines](https://cwiki.apache.org/confluence/display/DRILL/Apa
+che+Drill+Contribution+Guidelines) for information about how to contribute to
+the project. If you would like to contribute to the project and need some
+ideas for what to do, please read [Apache Drill Contribution
+Ideas](/confluence/display/DRILL/Apache+Drill+Contribution+Ideas).
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/012-sample-ds.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/012-sample-ds.md b/_docs/drill-docs/012-sample-ds.md
new file mode 100644
index 0000000..fe63f6b
--- /dev/null
+++ b/_docs/drill-docs/012-sample-ds.md
@@ -0,0 +1,11 @@
+---
+title: "Sample Datasets"
+parent: "Apache Drill Documentation"
+---
+Use any of the following sample datasets provided to test Drill:
+
+ * [AOL Search](/confluence/display/DRILL/AOL+Search)
+ * [Enron Emails](/confluence/display/DRILL/Enron+Emails)
+ * [Wikipedia Edit History](/confluence/display/DRILL/Wikipedia+Edit+History)
+
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/013-design.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/013-design.md b/_docs/drill-docs/013-design.md
new file mode 100644
index 0000000..57d73c1
--- /dev/null
+++ b/_docs/drill-docs/013-design.md
@@ -0,0 +1,14 @@
+---
+title: "Design Docs"
+parent: "Apache Drill Documentation"
+---
+Review the Apache Drill design docs for early descriptions of Apache Drill
+functionality, terms, and goals, and reference the research articles to learn
+about Apache Drill's history:
+
+ * [Drill Plan Syntax](/confluence/display/DRILL/Drill+Plan+Syntax)
+ * [RPC Overview](/confluence/display/DRILL/RPC+Overview)
+ * [Query Stages](/confluence/display/DRILL/Query+Stages)
+ * [Useful Research](/confluence/display/DRILL/Useful+Research)
+ * [Value Vectors](/confluence/display/DRILL/Value+Vectors)
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/014-progress.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/014-progress.md b/_docs/drill-docs/014-progress.md
new file mode 100644
index 0000000..2a1538c
--- /dev/null
+++ b/_docs/drill-docs/014-progress.md
@@ -0,0 +1,9 @@
+---
+title: "Progress Reports"
+parent: "Apache Drill Documentation"
+---
+Review the following Apache Drill progress reports for a summary of issues,
+progression of the project, summary of mailing list discussions, and events:
+
+ * [2014 Q1 Drill Report](/confluence/display/DRILL/2014+Q1+Drill+Report)
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/015-archived-pages.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/015-archived-pages.md b/_docs/drill-docs/015-archived-pages.md
new file mode 100644
index 0000000..b2a29c3
--- /dev/null
+++ b/_docs/drill-docs/015-archived-pages.md
@@ -0,0 +1,9 @@
+---
+title: "Archived Pages"
+parent: "Apache Drill Documentation"
+---
+The following pages have been archived:
+
+* How to Run Drill with Sample Data
+* Meet Apache Drill
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/016-bylaws.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/016-bylaws.md b/_docs/drill-docs/016-bylaws.md
new file mode 100644
index 0000000..6f2604f
--- /dev/null
+++ b/_docs/drill-docs/016-bylaws.md
@@ -0,0 +1,171 @@
+---
+title: "Project Bylaws"
+parent: "Apache Drill Documentation"
+---
+# Introduction
+
+This document defines the bylaws under which the Apache Drill project
+operates. It defines the roles and responsibilities of the project, who may
+vote, how voting works, how conflicts are resolved, etc.
+
+Drill is a project of the [Apache Software
+Foundation](http://www.apache.org/foundation/). The foundation holds the
+copyright on Apache code including the code in the Drill codebase. The
+[foundation FAQ](http://www.apache.org/foundation/faq.html) explains the
+operation and background of the foundation.
+
+Drill is typical of Apache projects in that it operates under a set of
+principles, known collectively as the _Apache Way_. If you are new to Apache
+development, please refer to the [Incubator
+project](http://incubator.apache.org/) for more information on how Apache
+projects operate.
+
+# Roles and Responsibilities
+
+Apache projects define a set of roles with associated rights and
+responsibilities. These roles govern what tasks an individual may perform
+within the project. The roles are defined in the following sections.
+
+## Users
+
+The most important participants in the project are people who use our
+software. The majority of our contributors start out as users and guide their
+development efforts from the user's perspective.
+
+Users contribute to the Apache projects by providing feedback to contributors
+in the form of bug reports and feature suggestions. As well, users participate
+in the Apache community by helping other users on mailing lists and user
+support forums.
+
+## Contributors
+
+All of the volunteers who are contributing time, code, documentation, or
+resources to the Drill Project. A contributor that makes sustained, welcome
+contributions to the project may be invited to become a committer, though the
+exact timing of such invitations depends on many factors.
+
+## Committers
+
+The project's committers are responsible for the project's technical
+management. Committers have access to a specified set of subproject's code
+repositories. Committers on subprojects may cast binding votes on any
+technical discussion regarding that subproject.
+
+Committer access is by invitation only and must be approved by lazy consensus
+of the active PMC members. A Committer is considered _emeritus_ by his or her
+own declaration or by not contributing in any form to the project for over six
+months. An emeritus committer may request reinstatement of commit access from
+the PMC which will be sufficient to restore him or her to active committer
+status.
+
+Commit access can be revoked by a unanimous vote of all the active PMC members
+(except the committer in question if he or she is also a PMC member).
+
+All Apache committers are required to have a signed [Contributor License
+Agreement (CLA)](http://www.apache.org/licenses/icla.txt) on file with the
+Apache Software Foundation. There is a [Committer
+FAQ](http://www.apache.org/dev/committers.html) which provides more details on
+the requirements for committers.
+
+A committer who makes a sustained contribution to the project may be invited
+to become a member of the PMC. The form of contribution is not limited to
+code. It can also include code review, helping out users on the mailing lists,
+documentation, etc.
+
+## Project Management Committee
+
+The PMC is responsible to the board and the ASF for the management and
+oversight of the Apache Drill codebase. The responsibilities of the PMC
+include
+
+ * Deciding what is distributed as products of the Apache Drill project. In particular all releases must be approved by the PMC.
+ * Maintaining the project's shared resources, including the codebase repository, mailing lists, websites.
+ * Speaking on behalf of the project.
+ * Resolving license disputes regarding products of the project.
+ * Nominating new PMC members and committers.
+ * Maintaining these bylaws and other guidelines of the project.
+
+Membership of the PMC is by invitation only and must be approved by a lazy
+consensus of active PMC members. A PMC member is considered _emeritus_ by his
+or her own declaration or by not contributing in any form to the project for
+over six months. An emeritus member may request reinstatement to the PMC,
+which will be sufficient to restore him or her to active PMC member.
+
+Membership of the PMC can be revoked by an unanimous vote of all the active
+PMC members other than the member in question.
+
+The chair of the PMC is appointed by the ASF board. The chair is an office
+holder of the Apache Software Foundation (Vice President, Apache Drill) and
+has primary responsibility to the board for the management of the projects
+within the scope of the Drill PMC. The chair reports to the board quarterly on
+developments within the Drill project.
+
+The term of the chair is one year. When the current chair's term is up or if
+the chair resigns before the end of his or her term, the PMC votes to
+recommend a new chair using lazy consensus, but the decision must be ratified
+by the Apache board.
+
+# Decision Making
+
+Within the Drill project, different types of decisions require different forms
+of approval. For example, the previous section describes several decisions
+which require 'lazy consensus' approval. This section defines how voting is
+performed, the types of approvals, and which types of decision require which
+type of approval.
+
+## Voting
+
+Decisions regarding the project are made by votes on the primary project
+development mailing list
+_[dev@drill.apache.org](mailto:dev@drill.apache.org)_. Where necessary, PMC
+voting may take place on the private Drill PMC mailing list
+[private@drill.apache.org](mailto:private@drill.apache.org). Votes are clearly
+indicated by subject line starting with [VOTE]. Votes may contain multiple
+items for approval and these should be clearly separated. Voting is carried
+out by replying to the vote mail. Voting may take four flavors.
+
+ <table class="confluenceTable"><tbody><tr><td valign="top" class="confluenceTd"><p>Vote</p></td><td valign="top" class="confluenceTd"><p> </p></td></tr><tr><td valign="top" class="confluenceTd"><p>+1</p></td><td valign="top" class="confluenceTd"><p>'Yes,' 'Agree,' or 'the action should be performed.' In general, this vote also indicates a willingness on the behalf of the voter in 'making it happen'.</p></td></tr><tr><td valign="top" class="confluenceTd"><p>+0</p></td><td valign="top" class="confluenceTd"><p>This vote indicates a willingness for the action under consideration to go ahead. The voter, however will not be able to help.</p></td></tr><tr><td valign="top" class="confluenceTd"><p>-0</p></td><td valign="top" class="confluenceTd"><p>This vote indicates that the voter does not, in general, agree with the proposed action but is not concerned enough to prevent the action going ahead.</p></td></tr><tr><td valign="top" class="confluenceTd"><p>-1</p></td><td valign="top" class="co
nfluenceTd"><p>This is a negative vote. On issues where consensus is required, this vote counts as a <strong>veto</strong>. All vetoes must contain an explanation of why the veto is appropriate. Vetoes with no explanation are void. It may also be appropriate for a -1 vote to include an alternative course of action.</p></td></tr></tbody></table>
+
+All participants in the Drill project are encouraged to show their agreement
+with or against a particular action by voting. For technical decisions, only
+the votes of active committers are binding. Non binding votes are still useful
+for those with binding votes to understand the perception of an action in the
+wider Drill community. For PMC decisions, only the votes of PMC members are
+binding.
+
+Voting can also be applied to changes already made to the Drill codebase.
+These typically take the form of a veto (-1) in reply to the commit message
+sent when the commit is made. Note that this should be a rare occurrence. All
+efforts should be made to discuss issues when they are still patches before
+the code is committed.
+
+## Approvals
+
+These are the types of approvals that can be sought. Different actions require
+different types of approvals.
+
+<table class="confluenceTable"><tbody><tr><td valign="top" class="confluenceTd"><p>Approval Type</p></td><td valign="top" class="confluenceTd"><p> </p></td></tr><tr><td valign="top" class="confluenceTd"><p>Consensus</p></td><td valign="top" class="confluenceTd"><p>For this to pass, all voters with binding votes must vote and there can be no binding vetoes (-1). Consensus votes are rarely required due to the impracticality of getting all eligible voters to cast a vote.</p></td></tr><tr><td valign="top" class="confluenceTd"><p>Lazy Consensus</p></td><td valign="top" class="confluenceTd"><p>Lazy consensus requires 3 binding +1 votes and no binding vetoes.</p></td></tr><tr><td valign="top" class="confluenceTd"><p>Lazy Majority</p></td><td valign="top" class="confluenceTd"><p>A lazy majority vote requires 3 binding +1 votes and more binding +1 votes that -1 votes.</p></td></tr><tr><td valign="top" class="confluenceTd"><p>Lazy Approval</p></td><td valign="top" class="confluenceTd"><p>An a
ction with lazy approval is implicitly allowed unless a -1 vote is received, at which time, depending on the type of action, either lazy majority or lazy consensus approval must be obtained.</p></td></tr></tbody></table>
+
+## Vetoes
+
+A valid, binding veto cannot be overruled. If a veto is cast, it must be
+accompanied by a valid reason explaining the reasons for the veto. The
+validity of a veto, if challenged, can be confirmed by anyone who has a
+binding vote. This does not necessarily signify agreement with the veto -
+merely that the veto is valid.
+
+If you disagree with a valid veto, you must lobby the person casting the veto
+to withdraw his or her veto. If a veto is not withdrawn, the action that has
+been vetoed must be reversed in a timely manner.
+
+## Actions
+
+This section describes the various actions which are undertaken within the
+project, the corresponding approval required for that action and those who
+have binding votes over the action. It also specifies the minimum length of
+time that a vote must remain open, measured in business days. In general votes
+should not be called at times when it is known that interested members of the
+project will be unavailable.
+
+<table class="confluenceTable"><tbody><tr><td valign="top" class="confluenceTd"><p>Action</p></td><td valign="top" class="confluenceTd"><p>Description</p></td><td valign="top" class="confluenceTd"><p>Approval</p></td><td valign="top" class="confluenceTd"><p>Binding Votes</p></td><td valign="top" class="confluenceTd"><p>Minimum Length</p></td></tr><tr><td valign="top" class="confluenceTd"><p>Code Change</p></td><td valign="top" class="confluenceTd"><p>A change made to a codebase of the project and committed by a committer. This includes source code, documentation, website content, etc.</p></td><td valign="top" class="confluenceTd"><p>Consensus approval of active committers, with a minimum of one +1. The code can be committed after the first +1</p></td><td valign="top" class="confluenceTd"><p>Active committers</p></td><td valign="top" class="confluenceTd"><p>1</p></td></tr><tr><td valign="top" class="confluenceTd"><p>Release Plan</p></td><td valign="top" class="confluenceTd"><p>Define
s the timetable and actions for a release. The plan also nominates a Release Manager.</p></td><td valign="top" class="confluenceTd"><p>Lazy majority</p></td><td valign="top" class="confluenceTd"><p>Active committers</p></td><td valign="top" class="confluenceTd"><p>3</p></td></tr><tr><td valign="top" class="confluenceTd"><p>Product Release</p></td><td valign="top" class="confluenceTd"><p>When a release of one of the project's products is ready, a vote is required to accept the release as an official release of the project.</p></td><td valign="top" class="confluenceTd"><p>Lazy Majority</p></td><td valign="top" class="confluenceTd"><p>Active PMC members</p></td><td valign="top" class="confluenceTd"><p>3</p></td></tr><tr><td valign="top" class="confluenceTd"><p>Adoption of New Codebase</p></td><td valign="top" class="confluenceTd"><p>When the codebase for an existing, released product is to be replaced with an alternative codebase. If such a vote fails to gain approval, the existing cod
e base will continue. This also covers the creation of new sub-projects within the project.</p></td><td valign="top" class="confluenceTd"><p>2/3 majority</p></td><td valign="top" class="confluenceTd"><p>Active PMC members</p></td><td valign="top" class="confluenceTd"><p>6</p></td></tr><tr><td valign="top" class="confluenceTd"><p>New Committer</p></td><td valign="top" class="confluenceTd"><p>When a new committer is proposed for the project.</p></td><td valign="top" class="confluenceTd"><p>Lazy consensus</p></td><td valign="top" class="confluenceTd"><p>Active PMC members</p></td><td valign="top" class="confluenceTd"><p>3</p></td></tr><tr><td valign="top" class="confluenceTd"><p>New PMC Member</p></td><td valign="top" class="confluenceTd"><p>When a committer is proposed for the PMC.</p></td><td valign="top" class="confluenceTd"><p>Lazy consensus</p></td><td valign="top" class="confluenceTd"><p>Active PMC members</p></td><td valign="top" class="confluenceTd"><p>3</p></td></tr><tr><td va
lign="top" class="confluenceTd"><p>Committer Removal</p></td><td valign="top" class="confluenceTd"><p>When removal of commit privileges is sought. <em>Note: Such actions will also be referred to the ASF board by the PMC chair.</em></p></td><td valign="top" class="confluenceTd"><p>Consensus</p></td><td valign="top" class="confluenceTd"><p>Active PMC members (excluding the committer in question if a member of the PMC).</p></td><td valign="top" class="confluenceTd"><p>6</p></td></tr><tr><td valign="top" class="confluenceTd"><p>PMC Member Removal</p></td><td valign="top" class="confluenceTd"><p>When removal of a PMC member is sought. <em>Note: Such actions will also be referred to the ASF board by the PMC chair.</em></p></td><td valign="top" class="confluenceTd"><p>Consensus</p></td><td valign="top" class="confluenceTd"><p>Active PMC members (excluding the member in question).</p></td><td valign="top" class="confluenceTd"><p>6</p></td></tr><tr><td valign="top" class="confluenceTd"><p>Mo
difying Bylaws</p></td><td valign="top" class="confluenceTd"><p>Modifying this document.</p></td><td valign="top" class="confluenceTd"><p>2/3 majority</p></td><td valign="top" class="confluenceTd"><p>Active PMC members</p></td><td valign="top" class="confluenceTd"><p>6</p></td></tr></tbody></table>
+
[2/6] drill git commit: Added Drill docs
Posted by ts...@apache.org.
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/query/query-fs/004-query-dir.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/query/query-fs/004-query-dir.md b/_docs/drill-docs/query/query-fs/004-query-dir.md
new file mode 100644
index 0000000..47abee9
--- /dev/null
+++ b/_docs/drill-docs/query/query-fs/004-query-dir.md
@@ -0,0 +1,90 @@
+---
+title: "Querying Directories"
+parent: "Querying a File System"
+---
+You can store multiple files in a directory and query them as if they were a
+single entity. You do not have to explicitly join the files. The files must be
+compatible, in the sense that they must have comparable data types and columns
+in the same order. This type of query is not limited to text files; you can
+also query directories of JSON files, for example.
+
+For example, assume that a `testdata` directory contains two files with the
+same structure: `plays.csv` and `moreplays.csv`. The first file contains 7
+records and the second file contains 3 records. The following query returns
+the "union" of the two files, ordered by the first column:
+
+ 0: jdbc:drill:zk=local> select columns[0] as `Year`, columns[1] as Play
+ from dfs.`/Users/brumsby/drill/testdata` order by 1;
+
+ +------------+------------+
+ | Year | Play |
+ +------------+------------+
+ | 1594 | Comedy of Errors |
+ | 1595 | Romeo and Juliet |
+ | 1596 | The Merchant of Venice |
+ | 1599 | As You Like It |
+ | 1599 | Hamlet |
+ | 1601 | Twelfth Night |
+ | 1606 | Macbeth |
+ | 1606 | King Lear |
+ | 1609 | The Winter's Tale |
+ | 1610 | The Tempest |
+ +------------+------------+
+ 10 rows selected (0.296 seconds)
+
+You can drill down further and automatically query subdirectories as well. For
+example, assume that you have a logs directory that contains a subdirectory
+for each year and subdirectories for each month (1 through 12). The month
+directories contain JSON files.
+
+ [root@ip-172-16-1-200 logs]# pwd
+ /mapr/drilldemo/labs/clicks/logs
+ [root@ip-172-16-1-200 logs]# ls
+ 2012 2013 2014
+ [root@ip-172-16-1-200 logs]# cd 2013
+ [root@ip-172-16-1-200 2013]# ls
+ 1 10 11 12 2 3 4 5 6 7 8 9
+
+You can query all of these files, or a subset, by referencing the file system
+once in a Drill query. For example, the following query counts the number of
+records in all of the files inside the `2013` directory:
+
+ 0: jdbc:drill:> select count(*) from MFS.`/mapr/drilldemo/labs/clicks/logs/2013` ;
+ +------------+
+ | EXPR$0 |
+ +------------+
+ | 24000 |
+ +------------+
+ 1 row selected (2.607 seconds)
+
+You can also use variables `dir0`, `dir1`, and so on, to refer to
+subdirectories in your workspace path. For example, assume that `bob.logdata`
+is a workspace that points to the `logs` directory, which contains multiple
+subdirectories: `2012`, `2013`, and `2014`. The following query constrains
+files inside the subdirectory named `2013`. The variable `dir0` refers to the
+first level down from logs, `dir1` to the next level, and so on.
+
+ 0: jdbc:drill:> use bob.logdata;
+ +------------+------------+
+ | ok | summary |
+ +------------+------------+
+ | true | Default schema changed to 'bob.logdata' |
+ +------------+------------+
+ 1 row selected (0.305 seconds)
+
+ 0: jdbc:drill:> select * from logs where dir0='2013' limit 10;
+ +------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+
+ | dir0 | dir1 | trans_id | date | time | cust_id | device | state | camp_id | keywords |
+ +------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+
+ | 2013 | 2 | 12115 | 02/23/2013 | 19:48:24 | 3 | IOS5 | az | 5 | who's |
+ | 2013 | 2 | 12127 | 02/26/2013 | 19:42:03 | 11459 | IOS5 | wa | 10 | for |
+ | 2013 | 2 | 12138 | 02/09/2013 | 05:49:01 | 1 | IOS6 | ca | 7 | minutes |
+ | 2013 | 2 | 12139 | 02/23/2013 | 06:58:20 | 1 | AOS4.4 | ms | 7 | i |
+ | 2013 | 2 | 12145 | 02/10/2013 | 10:14:56 | 10 | IOS5 | mi | 6 | wrong |
+ | 2013 | 2 | 12157 | 02/15/2013 | 02:49:22 | 102 | IOS5 | ny | 5 | want |
+ | 2013 | 2 | 12176 | 02/19/2013 | 08:39:02 | 28 | IOS5 | or | 0 | and |
+ | 2013 | 2 | 12194 | 02/24/2013 | 08:26:17 | 125445 | IOS5 | ar | 0 | say |
+ | 2013 | 2 | 12236 | 02/05/2013 | 01:40:05 | 10 | IOS5 | nj | 2 | sir |
+ | 2013 | 2 | 12249 | 02/03/2013 | 04:45:47 | 21725 | IOS5 | nj | 5 | no |
+ +------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+
+ 10 rows selected (0.583 seconds)
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/rn/001-0.5.0rn.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/rn/001-0.5.0rn.md b/_docs/drill-docs/rn/001-0.5.0rn.md
new file mode 100644
index 0000000..2dff978
--- /dev/null
+++ b/_docs/drill-docs/rn/001-0.5.0rn.md
@@ -0,0 +1,29 @@
+---
+title: "Apache Drill 0.5.0 Release Notes"
+parent: "Release Notes"
+---
+
+Apache Drill 0.5.0, the first beta release for Drill, is designed to help
+enthusiasts start working and experimenting with Drill. It also continues the
+Drill monthly release cycle as we drive towards general availability.
+
+The 0.5.0 release is primarily a bug fix release, with [more than 100 JIRAs](h
+ttps://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313820&versi
+on=12324880) closed, but there are some notable features. For information
+about the features, see the [Apache Drill Blog for the 0.5.0
+release](https://blogs.apache.org/drill/entry/apache_drill_beta_release_see).
+
+This release is available as [binary](http://www.apache.org/dyn/closer.cgi/inc
+ubator/drill/drill-0.5.0-incubating/apache-drill-0.5.0-incubating.tar.gz) and
+[source](http://www.apache.org/dyn/closer.cgi/incubator/drill/drill-0.5.0-incu
+bating/apache-drill-0.5.0-incubating-src.tar.gz) tarballs that are compiled
+against Apache Hadoop. Drill has been tested against MapR, Cloudera, and
+Hortonworks Hadoop distributions. There are associated build profiles and
+JIRAs that can help you run Drill against your preferred distribution.
+
+Apache Drill 0.5.0 Key Notes and Limitations
+
+ * The current release supports in memory and beyond memory execution. However, you must disable memory-intensive hash aggregate and hash join operations to leverage this functionality.
+ * While the Drill execution engine supports dynamic schema changes during the course of a query, some operators have yet to implement support for this behavior, such as Sort. Others operations, such as streaming aggregate, may have partial support that leads to unexpected results.
+ * There are known issues with joining text files without using an intervening view. See [DRILL-1401](https://issues.apache.org/jira/browse/DRILL-1401) for more information.
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/rn/002-0.4.0rn.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/rn/002-0.4.0rn.md b/_docs/drill-docs/rn/002-0.4.0rn.md
new file mode 100644
index 0000000..d0dac30
--- /dev/null
+++ b/_docs/drill-docs/rn/002-0.4.0rn.md
@@ -0,0 +1,42 @@
+---
+title: "Apache Drill 0.4.0 Release Notes"
+parent: "Release Notes"
+---
+The 0.4.0 release is a developer preview release, designed to help enthusiasts
+start to work with and experiment with Drill. It is the first Drill release
+that provides distributed query execution.
+
+This release is built upon [more than 800
+JIRAs](https://issues.apache.org/jira/browse/DRILL/fixforversion/12324963/).
+It is a pre-beta release on the way towards Drill. As a developer snapshot,
+the release contains a large number of outstanding bugs that will make some
+use cases challenging. Feel free to consult outstanding issues [targeted for
+the 0.5.0
+release](https://issues.apache.org/jira/browse/DRILL/fixforversion/12324880/)
+to see whether your use case is affected.
+
+To read more about this release and new features introduced, please view the
+[0.4.0 announcement blog
+entry](https://blogs.apache.org/drill/entry/announcing_apache_drill_0_4).
+
+The release is available as both [binary](http://www.apache.org/dyn/closer.cgi
+/incubator/drill/drill-0.4.0-incubating/apache-drill-0.4.0-incubating.tar.gz)
+and [source](http://www.apache.org/dyn/closer.cgi/incubator/drill/drill-0.4.0-
+incubating/apache-drill-0.4.0-incubating-src.tar.gz) tarballs. In both cases,
+these are compiled against Apache Hadoop. Drill has also been tested against
+MapR, Cloudera and Hortonworks Hadoop distributions and there are associated
+build profiles or JIRAs that can help you run against your preferred
+distribution.
+
+Some Key Notes & Limitations
+
+ * The current release supports in memory and beyond memory execution. However, users must disable memory-intensive hash aggregate and hash join operations to leverage this functionality.
+ * In many cases,merge join operations return incorrect results.
+ * Use of a local filter in a join “on” clause when using left, right or full outer joins may result in incorrect results.
+ * Because of known memory leaks and memory overrun issues you may need more memory and you may need to restart the system in some cases.
+ * Some types of complex expressions, especially those involving empty arrays may fail or return incorrect results.
+ * While the Drill execution engine supports dynamic schema changes during the course of a query, some operators have yet to implement support for this behavior (such as Sort). Others operations (such as streaming aggregate) may have partial support that leads to unexpected results.
+ * Protobuf, UDF, query plan interfaces and all interfaces are subject to change in incompatible ways.
+ * Multiplication of some types of DECIMAL(28+,*) will return incorrect result.
+
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/rn/003-alpha-rn.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/rn/003-alpha-rn.md b/_docs/drill-docs/rn/003-alpha-rn.md
new file mode 100644
index 0000000..15e160c
--- /dev/null
+++ b/_docs/drill-docs/rn/003-alpha-rn.md
@@ -0,0 +1,44 @@
+---
+title: "Apache Drill M1 Release Notes (Apache Drill Alpha)"
+parent: "Release Notes"
+---
+### Milestone 1 Goals
+
+The first release of Apache Drill is designed as a technology preview for
+people to better understand the architecture and vision. It is a functional
+release tying to piece together the key components of a next generation MPP
+query engine. It is designed to allow milestone 2 (M2) to focus on
+architectural analysis and performance optimization.
+
+ * Provide a new optimistic DAG execution engine for data analysis
+ * Build a new columnar shredded in-memory format and execution model that minimizes data serialization/deserialization costs and operator complexity
+ * Provide a model for runtime generated functions and relational operators that minimizes complexity and maximizes performance
+ * Support queries against columnar on disk format (Parquet) and JSON
+ * Support the most common set of standard SQL read-only phrases using ANSI standards. Includes: SELECT, FROM, WHERE, HAVING, ORDER, GROUP BY, IN, DISTINCT, LEFT JOIN, RIGHT JOIN, INNER JOIN
+ * Support schema-on-read querying and execution
+ * Build a set of columnar operation primitives including Merge Join, Sort, Streaming Aggregate, Filter, Selection Vector removal.
+ * Support unlimited level of subqueries and correlated subqueries
+ * Provided an extensible query-language agnostic JSON-base logical data flow syntax.
+ * Support complex data type manipulation via logical plan operations
+
+### Known Issues
+
+SQL Parsing
+Because Apache Drill is built to support late-bound changing schemas while SQL
+is statically typed, there are couple of special requirements that are
+required writing SQL queries. These are limited to the current release and
+will be correct in a future milestone release.
+
+ * All tables are exposed as a single map field that contains
+ * Drill Alpha doesn't support implicit or explicit casts outside those required above.
+ * Drill Alpha does not include, there are currently a couple of differences for how to write a query in In order to query against
+
+UDFs
+
+ * Drill currently supports simple and aggregate functions using scalar, repeated and
+ * Nested data support incomplete. Drill Alpha supports nested data structures as well repeated fields. However,
+ * asd
+
+
+
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/rn/004-0.6.0-rn.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/rn/004-0.6.0-rn.md b/_docs/drill-docs/rn/004-0.6.0-rn.md
new file mode 100644
index 0000000..22aa96b
--- /dev/null
+++ b/_docs/drill-docs/rn/004-0.6.0-rn.md
@@ -0,0 +1,32 @@
+---
+title: "Apache Drill 0.6.0 Release Notes (Apache Drill Alpha)"
+parent: "Release Notes"
+---
+Apache Drill 0.6.0, the second beta release for Drill, is designed to help
+enthusiasts start working and experimenting with Drill. It also continues the
+Drill monthly release cycle as we drive towards general availability.
+
+This release is available as [binary](http://www.apache.org/dyn/closer.cgi/inc
+ubator/drill/drill-0.5.0-incubating/apache-drill-0.5.0-incubating.tar.gz) and
+[source](http://www.apache.org/dyn/closer.cgi/incubator/drill/drill-0.5.0-incu
+bating/apache-drill-0.5.0-incubating-src.tar.gz) tarballs that are compiled
+against Apache Hadoop. Drill has been tested against MapR, Cloudera, and
+Hortonworks Hadoop distributions. There are associated build profiles and
+JIRAs that can help you run Drill against your preferred distribution.
+
+Apache Drill 0.6.0 Key Features
+
+This release is primarily a bug fix release, with [more than 30 JIRAs closed](
+https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313820&vers
+ion=12327472), but there are some notable features:
+
+ * Direct ANSI SQL access to MongoDB, using the latest [MongoDB Plugin for Apache Drill](/confluence/display/DRILL/MongoDB+Plugin+for+Apache+Drill)
+ * Filesystem query performance improvements with partition pruning
+ * Ability to use the file system as a persistent store for query profiles and diagnostic information
+ * Window function support (alpha)
+
+Apache Drill 0.6.0 Key Notes and Limitations
+
+ * The current release supports in-memory and beyond-memory execution. However, you must disable memory-intensive hash aggregate and hash join operations to leverage this functionality.
+ * While the Drill execution engine supports dynamic schema changes during the course of a query, some operators have yet to implement support for this behavior, such as Sort. Other operations, such as streaming aggregate, may have partial support that leads to unexpected results.
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/rn/005-0.7.0-rn.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/rn/005-0.7.0-rn.md b/_docs/drill-docs/rn/005-0.7.0-rn.md
new file mode 100644
index 0000000..e25a58a
--- /dev/null
+++ b/_docs/drill-docs/rn/005-0.7.0-rn.md
@@ -0,0 +1,56 @@
+---
+title: "Apache Drill 0.7.0 Release Notes (Apache Drill Alpha)"
+parent: "Release Notes"
+---
+Apache Drill 0.7.0, the third beta release for Drill, is designed to help
+enthusiasts start working and experimenting with Drill. It also continues the
+Drill monthly release cycle as we drive towards general availability.
+
+This release is available as
+[binary](http://www.apache.org/dyn/closer.cgi/drill/drill-0.7.0/apache-
+drill-0.7.0.tar.gz) and
+[source](http://www.apache.org/dyn/closer.cgi/drill/drill-0.7.0/apache-
+drill-0.7.0-src.tar.gz) tarballs that are compiled against Apache Hadoop.
+Drill has been tested against MapR, Cloudera, and Hortonworks Hadoop
+distributions. There are associated build profiles and JIRAs that can help you
+run Drill against your preferred distribution
+
+Apache Drill 0.7.0 Key Features
+
+ * No more dependency on UDP/Multicast - Making it possible for Drill to work well in the following scenarios:
+
+ * UDP multicast not enabled (as in EC2)
+
+ * Cluster spans multiple subnets
+
+ * Cluster has multihome configuration
+
+ * New functions to natively work with nested data - KVGen and Flatten
+
+ * Support for Hive 0.13 (Hive 0.12 with Drill is not supported any more)
+
+ * Improved performance when querying Hive tables and File system through partition pruning
+
+ * Improved performance for HBase with LIKE operator pushdown
+
+ * Improved memory management
+
+ * Drill web UI monitoring and query profile improvements
+
+ * Ability to parse files without explicit extensions using default storage format specification
+
+ * Fixes for dealing with complex/nested data objects in Parquet/JSON
+
+ * Fast schema return - Improved experience working with BI/query tools by returning metadata quickly
+
+ * Several hang related fixes
+
+ * Parquet writer fixes for handling large datasets
+
+ * Stability improvements in ODBC and JDBC drivers
+
+Apache Drill 0.7.0 Key Notes and Limitations
+
+ * The current release supports in-memory and beyond-memory execution. However, you must disable memory-intensive hash aggregate and hash join operations to leverage this functionality.
+ * While the Drill execution engine supports dynamic schema changes during the course of a query, some operators have yet to implement support for this behavior, such as Sort. Other operations, such as streaming aggregate, may have partial support that leads to unexpected results.
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/sql-ref/001-data-types.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/sql-ref/001-data-types.md b/_docs/drill-docs/sql-ref/001-data-types.md
new file mode 100644
index 0000000..1f0ba0e
--- /dev/null
+++ b/_docs/drill-docs/sql-ref/001-data-types.md
@@ -0,0 +1,96 @@
+---
+title: "Data Types"
+parent: "SQL Reference"
+---
+You can use the following SQL data types in your Drill queries:
+
+#### Binary Types
+
+ * BYTEA
+ * VARBINARY
+
+#### Character Types
+
+ * CHAR
+ * VARCHAR
+ * TEXT
+
+#### Date/Time Data Types
+
+ * DATE
+ * INTERVAL
+ * Interval Year (stores year and month)
+
+ * Interval Day (stores day, hour, minute, seconds, and milliseconds)
+ * TIME
+ * TIMESTAMP
+
+Refer to [Supported Date/Time Data Type formats](https://cwiki.apache.org/conf
+luence/pages/viewpage.action?pageId=44304039).
+
+#### Integer
+
+ * BIGINT
+
+ * INT
+ * SMALLINT
+
+ * TINYINT
+
+#### Numeric Types
+
+ * DECIMAL
+ * FLOAT
+ * DOUBLE PRECISION (FLOAT 8)
+ * REAL (FLOAT 4)
+
+#### Boolean Types
+
+ * FALSE
+ * NULL
+ * TRUE
+
+# Complex Data Types
+
+Drill provides map and array data types to work with complex and nested data
+structures. For analysis of complex data, a more modern JSON-style approach to
+writing queries is more effective than using standard SQL functions.
+
+The following table provides descriptions and examples of the complex data
+types:
+
+<table><tbody>
+ <tr><th>Data Type</th>
+ <th>Description</th>
+ <th>Example</th></tr>
+ <tr>
+ <td valign="top">Map</td>
+ <td valign="top">A map is a set of name/value pairs. </br>
+ A value in an map can be a scalar type, </br>
+ such as string or int, or a map can be a </br>
+ complex type, such as an array or another map.</td>
+ <td valign="top">Map with scalar type values:</br><code> "phoneNumber": { "areaCode": "622", "number": "1567845"}</code></br>Map with complex type value:<code></br> { "citiesLived" : [ { "place" : "Los Angeles",</br>
+ "yearsLived" : [ "1989",</br>
+ "1993",</br>
+ "1998",</br>
+ "2002"</br>
+ ]</br>
+
+ } ] }</code></td>
+ </tr>
+ <tr>
+ <td valign="top">Array</td>
+ <td valign="top">An array is a repeated list of values. </br>
+ A value in an array can be a scalar type, </br>
+ such as string or int, or an array can be a</br>
+ complex type, such as a map or another array.</td>
+ <td valign="top">Array with scalar values:</br><code> "yearsLived": ["1990", "1993", "1998", "2008"]</code></br>Array with complex type values:</br><code> "children":</br> [ { "age" : "10", </br> "gender" : "Male",</br> "name" : "Earl"</br> }, </br> { "age" : "6",</br> "gender" : "Male",</br> "name" : "Sam"</br> },</br> { "age" : "8",</br> "gender" : "Male", </br> "name" : "Kit" </br>  
;}</br> ]</code></td>
+ </tr>
+ </tbody></table>
+
+
+<!--| Data Type | Description | Example |
+| --------- | ----------- | --------|
+| Map | A map is a set of name/value pairs. A value in an map can be a scalar type, such as string or int, or a map can be a complex type, such as an array or another map.| Map with scalar type values: "phoneNumber": { "areaCode": "622", "number": "1567845"} Map with complex type value :{ "citiesLived" : [ { "place" : "Los Angeles", "yearsLived" : [ "1989", "1993", "1998", "2002"] } ] } |
+|Array| An array is a repeated list of values. A value in an array can be a scalar type, such as string or int, or an array can be a complex type, such as a map or another array.| Array with scalar type values: "yearsLived": ["1990"", "1993", "1998", "2008"] Array with complex type value :"children" : [ { "age" : "10", "gender" : "Male", "name": "Earl" }, {"age" : "6", "gender": "Male", "name": "Sam"}, { "age" : "8", "gender" : "Male", "name" : "Kit" }] |
+-->
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/sql-ref/002-operators.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/sql-ref/002-operators.md b/_docs/drill-docs/sql-ref/002-operators.md
new file mode 100644
index 0000000..763b809
--- /dev/null
+++ b/_docs/drill-docs/sql-ref/002-operators.md
@@ -0,0 +1,71 @@
+---
+title: "Operators"
+parent: "SQL Reference"
+---
+You can use various types of operators in your Drill queries to perform
+operations on your data.
+
+## Logical Operators
+
+You can use the following logical operators in your Drill queries:
+
+ * AND
+ * BETWEEN
+ * IN
+ * LIKE
+ * NOT
+ * OR
+
+## Comparison Operators
+
+You can use the following comparison operators in your Drill queries:
+
+ * <
+ * \>
+ * <=
+ * \>=
+ * =
+ * <>
+ * IS NULL
+ * IS NOT NULL
+ * IS FALSE
+ * IS NOT FALSE
+ * IS TRUE
+ * IS NOT TRUE
+
+## Pattern Matching Operators
+
+You can use the following pattern matching operators in your Drill queries:
+
+ * LIKE
+ * NOT LIKE
+ * SIMILAR TO
+ * NOT SIMILAR TO
+
+## Math Operators
+
+You can use the following math operators in your Drill queries:
+
+**Operator**| **Description**
+---|---
++| Addition
+-| Subtraction
+*| Multiplication
+/| Division
+
+## Subquery Operators
+
+You can use the following subquery operators in your Drill queries:
+
+ * EXISTS
+ * IN
+
+See [SELECT Statements](https://cwiki.apache.org/confluence/display/DRILL/SELE
+CT+Statements).
+
+## String Operators
+
+You can use the following string operators in your Drill queries:
+
+ * string || string
+ * string || non-string or non-string || string
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/sql-ref/003-functions.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/sql-ref/003-functions.md b/_docs/drill-docs/sql-ref/003-functions.md
new file mode 100644
index 0000000..2494b0e
--- /dev/null
+++ b/_docs/drill-docs/sql-ref/003-functions.md
@@ -0,0 +1,185 @@
+---
+title: "SQL Functions"
+parent: "SQL Reference"
+---
+You can use the following types of functions in your Drill queries:
+
+ * Scalar Functions
+ * Aggregate Functions
+ * Aggregate Statistics Functions
+ * Convert Functions
+ * Nested Data Functions
+
+## Scalar Functions
+
+### Math
+
+You can use the following scalar math functions in your Drill queries:
+
+ * ABS
+ * CEIL
+ * CEILING
+ * DIV
+ * FLOOR
+ * MOD
+ * POWER
+ * RANDOM
+ * ROUND
+ * SIGN
+ * SQRT
+ * TRUNC
+
+### String Functions
+
+The following table provides the string functions that you can use in your
+Drill queries:
+
+Function| Return Type
+--------|---
+char_length(string) or character_length(string)| int
+concat(str "any" [, str "any" [, ...] ])| text
+convert_from(string bytea, src_encoding name)| text
+convert_to(string text, dest_encoding name)| bytea
+initcap(string)| text
+left(str text, n int)| text
+length(string)| int
+length(string bytes, encoding name )| int
+lower(string)| text
+lpad(string text, length int [, fill text])| text
+ltrim(string text [, characters text])| text
+position(substring in string)| int
+regexp_replace(string text, pattern text, replacement text [, flags text])|text
+replace(string text, from text, to text)| text
+right(str text, n int)| text
+rpad(string text, length int [, fill text])| text
+rtrim(string text [, characters text])| text
+strpos(string, substring)| int
+substr(string, from [, count])| text
+substring(string [from int] [for int])| text
+trim([leading | trailing | both] [characters] from string)| text
+upper(string)| text
+
+
+### Date/Time Functions
+
+The following table provides the date/time functions that you can use in your
+Drill queries:
+
+**Function**| **Return Type**
+---|---
+current_date| date
+current_time| time with time zone
+current_timestamp| timestamp with time zone
+date_add(date,interval expr type)| date/datetime
+date_part(text, timestamp)| double precision
+date_part(text, interval)| double precision
+date_sub(date,INTERVAL expr type)| date/datetime
+extract(field from interval)| double precision
+extract(field from timestamp)| double precision
+localtime| time
+localtimestamp| timestamp
+now()| timestamp with time zone
+timeofday()| text
+
+### Data Type Formatting Functions
+
+The following table provides the data type formatting functions that you can
+use in your Drill queries:
+
+**Function**| **Return Type**
+---|---
+to_char(timestamp, text)| text
+to_char(int, text)| text
+to_char(double precision, text)| text
+to_char(numeric, text)| text
+to_date(text, text)| date
+to_number(text, text)| numeric
+to_timestamp(text, text)| timestamp with time zone
+to_timestamp(double precision)| timestamp with time zone
+
+## Aggregate Functions
+
+The following table provides the aggregate functions that you can use in your
+Drill queries:
+
+**Function** | **Argument Type** | **Return Type**
+ -------- | ------------- | -----------
+avg(expression)| smallint, int, bigint, real, double precision, numeric, or interval| numeric for any integer-type argument, double precision for a floating-point argument, otherwise the same as the argument data type
+count(*)| _-_| bigint
+count([DISTINCT] expression)| any| bigint
+max(expression)| any array, numeric, string, or date/time type| same as argument type
+min(expression)| any array, numeric, string, or date/time type| same as argument type
+sum(expression)| smallint, int, bigint, real, double precision, numeric, or interval| bigint for smallint or int arguments, numeric for bigint arguments, double precision for floating-point arguments, otherwise the same as the argument data type
+
+
+## Aggregate Statistics Functions
+
+The following table provides the aggregate statistics functions that you can use in your Drill queries:
+
+**Function**| **Argument Type**| **Return Type**
+ -------- | ------------- | -----------
+stddev(expression)| smallint, int, bigint, real, double precision, or numeric| double precision for floating-point arguments, otherwise numeric
+stddev_pop(expression)| smallint, int, bigint, real, double precision, or numeric| double precision for floating-point arguments, otherwise numeric
+stddev_samp(expression)| smallint, int, bigint, real, double precision, or numeric| double precision for floating-point arguments, otherwise numeric
+variance(expression)| smallint, int, bigint, real, double precision, or numeric| double precision for floating-point arguments, otherwise numeric
+var_pop(expression)| smallint, int, bigint, real, double precision, or numeric| double precision for floating-point arguments, otherwise numeric
+var_samp(expression)| smallint, int, bigint, real, double precision, or numeric| double precision for floating-point arguments, otherwise numeric
+
+
+## Convert Functions
+
+You can use the CONVERT_TO and CONVERT_FROM functions to encode and decode
+data when you query your data sources with Drill. For example, HBase stores
+data as encoded byte arrays (VARBINARY data). When you issue a query with the
+CONVERT_FROM function on HBase, Drill decodes the data and converts it to the
+specified data type. In instances where Drill sends data back to HBase during
+a query, you can use the CONVERT_TO function to change the data type to bytes.
+
+Although you can achieve the same results by using the CAST function for some
+data types (such as VARBINARY to VARCHAR conversions), in general it is more
+efficient to use CONVERT functions when your data sources return binary data.
+When your data sources return more conventional data types, you can use the
+CAST function.
+
+The following table provides the data types that you use with the CONVERT_TO
+and CONVERT_FROM functions:
+
+**Type**| **Input Type**| **Output Type**
+---|---|---
+BOOLEAN_BYTE| bytes(1)| boolean
+TINYINT_BE| bytes(1)| tinyint
+TINYINT| bytes(1)| tinyint
+SMALLINT_BE| bytes(2)| smallint
+SMALLINT| bytes(2)| smallint
+INT_BE| bytes(4)| int
+INT| bytes(4)| int
+BIGINT_BE| bytes(8)| bigint
+BIGINT| bytes(8)| bigint
+FLOAT| bytes(4)| float (float4)
+DOUBLE| bytes(8)| double (float8)
+INT_HADOOPV| bytes(1-9)| int
+BIGINT_HADOOPV| bytes(1-9)| bigint
+DATE_EPOCH_BE| bytes(8)| date
+DATE_EPOCH| bytes(8)| date
+TIME_EPOCH_BE| bytes(8)| time
+TIME_EPOCH| bytes(8)| time
+UTF8| bytes| varchar
+UTF16| bytes| var16char
+UINT8| bytes(8)| uint8
+
+A common use case for CONVERT_FROM is when a data source embeds complex data
+inside a column. For example, you may have an HBase or MapR-DB table with
+embedded JSON data:
+
+ select CONVERT_FROM(col1, 'JSON')
+ FROM hbase.table1
+ ...
+
+## Nested Data Functions
+
+This section contains descriptions of SQL functions that you can use to
+analyze nested data:
+
+ * [FLATTEN Function](/confluence/display/DRILL/FLATTEN+Function)
+ * [KVGEN Function](/confluence/display/DRILL/KVGEN+Function)
+ * [REPEATED_COUNT Function](/confluence/display/DRILL/REPEATED_COUNT+Function)
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/sql-ref/004-nest-functions.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/sql-ref/004-nest-functions.md b/_docs/drill-docs/sql-ref/004-nest-functions.md
new file mode 100644
index 0000000..a07dd51
--- /dev/null
+++ b/_docs/drill-docs/sql-ref/004-nest-functions.md
@@ -0,0 +1,10 @@
+---
+title: "Nested Data Functions"
+parent: "SQL Reference"
+---
+This section contains descriptions of SQL functions that you can use to
+analyze nested data:
+
+ * [FLATTEN Function](/confluence/display/DRILL/FLATTEN+Function)
+ * [KVGEN Function](/confluence/display/DRILL/KVGEN+Function)
+ * [REPEATED_COUNT Function](/confluence/display/DRILL/REPEATED_COUNT+Function)
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/sql-ref/005-cmd-summary.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/sql-ref/005-cmd-summary.md b/_docs/drill-docs/sql-ref/005-cmd-summary.md
new file mode 100644
index 0000000..bd3c73e
--- /dev/null
+++ b/_docs/drill-docs/sql-ref/005-cmd-summary.md
@@ -0,0 +1,16 @@
+---
+title: "SQL Commands Summary"
+parent: "SQL Reference"
+---
+The following table provides a list of the SQL commands that Drill supports,
+with their descriptions and example syntax:
+
+<div class="table-wrap"><table class="confluenceTable"><tbody><tr><th class="confluenceTh">Command</th><th class="confluenceTh">Description</th><th class="confluenceTh">Syntax</th></tr><tr><td valign="top" class="confluenceTd">ALTER SESSION</td><td valign="top" class="confluenceTd">Changes a system setting for the duration of a session. A session ends when you quit the Drill shell. For a list of Drill options and their descriptions, refer to <a href="https://cwiki.apache.org/confluence/display/DRILL/Planning+and+Execution+Options" rel="nofollow">Planning and Execution Options</a>.</td><td valign="top" class="confluenceTd"><code>ALTER SESSION SET `option_name`='string';<br />ALTER SESSION SET `option_name`=TRUE | FALSE;</code></td></tr><tr><td valign="top" class="confluenceTd">ALTER SYSTEM</td><td valign="top" class="confluenceTd">Permanently changes a system setting. The new settings persist across all sessions. For a list of Drill options and their descriptions, refer to <a href="h
ttps://cwiki.apache.org/confluence/display/DRILL/Planning+and+Execution+Options" rel="nofollow">Planning and Execution Options</a>.</td><td valign="top" class="confluenceTd"><code>ALTER SYSTEM `option_name`='string'<br />ALTER SYSTEM `option_name`=TRUE | FALSE;</code></td></tr><tr><td valign="top" class="confluenceTd"><p><a href="/confluence/display/DRILL/CREATE+TABLE+AS+%28CTAS%29+Command">CREATE TABLE AS<br />(CTAS)</a></p></td><td valign="top" class="confluenceTd"><p>Creates a new table and populates the new table with rows returned from a SELECT query. Use the CREATE TABLE AS (CTAS) statement in place of INSERT INTO. When you issue the CTAS command, you create a directory that contains parquet or CSV files. Each workspace in a file system has a default file type.</p><p>You can specify which writer you want Drill to use when creating a table: parquet, CSV, or JSON (as specified with <span style="line-height: 1.4285715;">the </span><code>store.format</code><span style="line-height
: 1.4285715;"> option<span><span>).</span></span></span></p></td><td valign="top" class="confluenceTd"><code>CREATE TABLE new_table_name AS <query>;</code></td></tr><tr><td valign="top" colspan="1" class="confluenceTd">CREATE VIEW</td><td valign="top" colspan="1" class="confluenceTd"><p>Creates a new view based on the results of a SELECT query.</p></td><td valign="top" colspan="1" class="confluenceTd"><code>CREATE VIEW view_name [(column_list)] AS <query>;</code></td></tr><tr><td valign="top" colspan="1" class="confluenceTd">DROP VIEW</td><td valign="top" colspan="1" class="confluenceTd">Removes one or more views.</td><td valign="top" colspan="1" class="confluenceTd"><code>DROP VIEW view_name [, <em class="replaceable">view_name</em>] ...; </code></td></tr><tr><td valign="top" colspan="1" class="confluenceTd"><a href="https://cwiki.apache.org/confluence/display/DRILL/EXPLAIN+Commands" rel="nofollow">EXPLAIN PLAN FOR</a></td><td valign="top" colspan="1" class="conflue
nceTd">Returns the physical plan for a particular query.</td><td valign="top" colspan="1" class="confluenceTd"><code>EXPLAIN PLAN FOR <query>;</code></td></tr><tr><td valign="top" colspan="1" class="confluenceTd"><a href="https://cwiki.apache.org/confluence/display/DRILL/EXPLAIN+Commands" rel="nofollow">EXPLAIN PLAN WITHOUT IMPLEMENTATION FOR</a></td><td valign="top" colspan="1" class="confluenceTd">Returns the logical plan for a particular query.</td><td valign="top" colspan="1" class="confluenceTd"><code>EXPLAIN PLAN WITHOUT IMPLEMENTATION FOR <query>;</code></td></tr><tr><td valign="top" colspan="1" class="confluenceTd"><a href="https://cwiki.apache.org/confluence/display/DRILL/SELECT+Statements" rel="nofollow">SELECT</a></td><td valign="top" colspan="1" class="confluenceTd">Retrieves data from tables and files.</td><td valign="top" colspan="1" class="confluenceTd"><code>[WITH subquery]<br />SELECT column_list FROM table_name <br />[WHERE clause]<br />[GROUP BY clause
]<br />[HAVING clause]<br />[ORDER BY clause];</code></td></tr><tr><td valign="top" colspan="1" class="confluenceTd">SHOW DATABASES</td><td valign="top" colspan="1" class="confluenceTd">Returns a list of available schemas. Equivalent to SHOW SCHEMAS.</td><td valign="top" colspan="1" class="confluenceTd"><code>SHOW DATABASES;</code></td></tr><tr><td valign="top" colspan="1" class="confluenceTd"><a href="https://cwiki.apache.org/confluence/display/DRILL/SHOW+FILES+Command" rel="nofollow">SHOW FILES</a></td><td valign="top" colspan="1" class="confluenceTd">Returns a list of files in a file system schema.</td><td valign="top" colspan="1" class="confluenceTd"><code>SHOW FILES IN filesystem.`schema_name`;<br />SHOW FILES FROM filesystem.`schema_name`;</code></td></tr><tr><td valign="top" colspan="1" class="confluenceTd">SHOW SCHEMAS</td><td valign="top" colspan="1" class="confluenceTd">Returns a list of available schemas. Equivalent to SHOW DATABASES.</td><td valign="top" colspan="1" clas
s="confluenceTd"><code>SHOW SCHEMAS;</code></td></tr><tr><td valign="top" colspan="1" class="confluenceTd">SHOW TABLES</td><td valign="top" colspan="1" class="confluenceTd">Returns a list of tables for all schemas. Optionally, you can first issue the <code>USE </code>command to identify the schema for which you want to view tables.<br />For example, the following <code>USE</code> statement tells Drill that you only want information from the <code>hive.default</code> schema:<br /><code>USE hive.`default`;</code></td><td valign="top" colspan="1" class="confluenceTd"><code>SHOW TABLES;</code></td></tr><tr><td valign="top" colspan="1" class="confluenceTd">USE</td><td valign="top" colspan="1" class="confluenceTd">Change to a particular schema. When you opt to use a particular schema, Drill issues queries on that schema only.</td><td valign="top" colspan="1" class="confluenceTd"><code>USE schema_name;</code></td></tr></tbody></table></div>
+
+Click on any of the following links for more information about the various
+commands:
+
+ * [CREATE TABLE AS (CTAS) Command](/confluence/display/DRILL/CREATE+TABLE+AS+%28CTAS%29+Command)
+ * [EXPLAIN Commands](/confluence/display/DRILL/EXPLAIN+Commands)
+ * [SELECT Statements](/confluence/display/DRILL/SELECT+Statements)
+ * [SHOW FILES Command](/confluence/display/DRILL/SHOW+FILES+Command)
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/sql-ref/006-reserved-wds.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/sql-ref/006-reserved-wds.md b/_docs/drill-docs/sql-ref/006-reserved-wds.md
new file mode 100644
index 0000000..6df4360
--- /dev/null
+++ b/_docs/drill-docs/sql-ref/006-reserved-wds.md
@@ -0,0 +1,16 @@
+---
+title: "Reserved Keyswords"
+parent: "SQL Reference"
+---
+When you use a reserved keyword in a Drill query, enclose the word in
+backticks. For example, if you issue the following query to Drill,
+you must include backticks around the word TABLES because TABLES is a reserved
+keyword:
+
+``SELECT * FROM INFORMATION_SCHEMA.`TABLES`;``
+
+The following table provides the Drill reserved keywords that require back
+ticks:
+
+<div class="table-wrap"><table class="confluenceTable"><tbody><tr><td valign="top" class="confluenceTd"><h1 id="ReservedKeywords-A">A</h1><p>ABS<br />ALL<br />ALLOCATE<br />ALLOW<br />ALTER<br />AND<br />ANY<br />ARE<br />ARRAY<br />AS<br />ASENSITIVE<br />ASYMMETRIC<br />AT<br />ATOMIC<br />AUTHORIZATION<br />AVG</p><h1 id="ReservedKeywords-B">B</h1><p>BEGIN<br />BETWEEN<br />BIGINT<br />BINARY<br />BIT<br />BLOB<br />BOOLEAN<br />BOTH<br />BY</p><h1 id="ReservedKeywords-C">C</h1><p>CALL<br />CALLED<br />CARDINALITY<br />CASCADED<br />CASE<br />CAST<br />CEIL<br />CEILING<br />CHAR<br />CHARACTER<br />CHARACTER_LENGTH<br />CHAR_LENGTH<br />CHECK<br />CLOB<br />CLOSE<br />COALESCE<br />COLLATE<br />COLLECT<br />COLUMN<br />COMMIT<br />CONDITION<br />CONNECT<br />CONSTRAINT<br />CONVERT<br />CORR<br />CORRESPONDING<br />COUNT<br />COVAR_POP<br />COVAR_SAMP<br />CREATE<br />CROSS<br />CUBE<br />CUME_DIST<br />CURRENT<br />CURRENT_CATALOG<br />CURRENT_DATE<br />CURRENT_DEFAULT_TRANSFOR
M_GROUP<br />CURRENT_PATH<br />CURRENT_ROLE<br />CURRENT_SCHEMA<br />CURRENT_TIME<br />CURRENT_TIMESTAMP<br />CURRENT_TRANSFORM_GROUP_FOR_TYPE<br />CURRENT_USER<br />CURSOR<br />CYCLE</p></td><td valign="top" class="confluenceTd"><h1 id="ReservedKeywords-D">D</h1><p>DATABASES<br />DATE<br />DAY<br />DEALLOCATE<br />DEC<br />DECIMAL<br />DECLARE<br />DEFAULT<br />DEFAULT_KW<br />DELETE<br />DENSE_RANK<br />DEREF<br />DESCRIBE<br />DETERMINISTIC<br />DISALLOW<br />DISCONNECT<br />DISTINCT<br />DOUBLE<br />DROP<br />DYNAMIC</p><h1 id="ReservedKeywords-E">E</h1><p>EACH<br />ELEMENT<br />ELSE<br />END<br />END_EXEC<br />ESCAPE<br />EVERY<br />EXCEPT<br />EXEC<br />EXECUTE<br />EXISTS<br />EXP<br />EXPLAIN<br />EXTERNAL<br />EXTRACT</p><h1 id="ReservedKeywords-F">F</h1><p>FALSE<br />FETCH<br />FILES<br />FILTER<br />FIRST_VALUE<br />FLOAT<br />FLOOR<br />FOR<br />FOREIGN<br />FREE<br />FROM<br />FULL<br />FUNCTION<br />FUSION</p><h1 id="ReservedKeywords-G">G</h1><p>GET<br />GLOBAL<br />GR
ANT<br />GROUP<br />GROUPING</p><h1 id="ReservedKeywords-H">H</h1><p>HAVING<br />HOLD<br />HOUR</p></td><td valign="top" class="confluenceTd"><h1 id="ReservedKeywords-I">I</h1><p>IDENTITY<br />IMPORT<br />IN<br />INDICATOR<br />INNER<br />INOUT<br />INSENSITIVE<br />INSERT<br />INT<br />INTEGER<br />INTERSECT<br />INTERSECTION<br />INTERVAL<br />INTO<br />IS</p><h1 id="ReservedKeywords-J">J</h1><p>JOIN</p><h1 id="ReservedKeywords-L">L</h1><p>LANGUAGE<br />LARGE<br />LAST_VALUE<br />LATERAL<br />LEADING<br />LEFT<br />LIKE<br />LIMIT<br />LN<br />LOCAL<br />LOCALTIME<br />LOCALTIMESTAMP<br />LOWER</p><h1 id="ReservedKeywords-M">M</h1><p>MATCH<br />MAX<br />MEMBER<br />MERGE<br />METHOD<br />MIN<br />MINUTE<br />MOD<br />MODIFIES<br />MODULE<br />MONTH<br />MULTISET</p><h1 id="ReservedKeywords-N">N</h1><p>NATIONAL<br />NATURAL<br />NCHAR<br />NCLOB<br />NEW<br />NO<br />NONE<br />NORMALIZE<br />NOT<br />NULL<br />NULLIF<br />NUMERIC</p><h1 id="ReservedKeywords-O">O</h1><p>OCTET_LENGTH
<br />OF<br />OFFSET<br />OLD<br />ON<br />ONLY<br />OPEN<br />OR<br />ORDER<br />OUT<br />OUTER<br />OVER<br />OVERLAPS<br />OVERLAY</p></td><td valign="top" colspan="1" class="confluenceTd"><h1 id="ReservedKeywords-P">P</h1><p>PARAMETER<br />PARTITION<br />PERCENTILE_CONT<br />PERCENTILE_DISC<br />PERCENT_RANK<br />POSITION<br />POWER<br />PRECISION<br />PREPARE<br />PRIMARY<br />PROCEDURE</p><h1 id="ReservedKeywords-R">R</h1><p>RANGE<br />RANK<br />READS<br />REAL<br />RECURSIVE<br />REF<br />REFERENCES<br />REFERENCING<br />REGR_AVGX<br />REGR_AVGY<br />REGR_COUNT<br />REGR_INTERCEPT<br />REGR_R2<br />REGR_SLOPE<br />REGR_SXX<br />REGR_SXY<br />RELEASE<br />REPLACE<br />RESULT<br />RETURN<br />RETURNS<br />REVOKE<br />RIGHT<br />ROLLBACK<br />ROLLUP<br />ROW<br />ROWS<br />ROW_NUMBER</p><h1 id="ReservedKeywords-S">S</h1><p>SAVEPOINT<br />SCHEMAS<br />SCOPE<br />SCROLL<br />SEARCH<br />SECOND<br />SELECT<br />SENSITIVE<br />SESSION_USER<br />SET<br />SHOW<br />SIMILAR<br />SMALLI
NT<br />SOME<br />SPECIFIC<br />SPECIFICTYPE<br />SQL<br />SQLEXCEPTION<br />SQLSTATE<br />SQLWARNING<br />SQRT<br />START<br />STATIC<br />STDDEV_POP<br />STDDEV_SAMP<br />SUBMULTISET<br />SUBSTRING<br />SUM<br />SYMMETRIC<br />SYSTEM<br />SYSTEM_USER</p></td><td valign="top" colspan="1" class="confluenceTd"><h1 id="ReservedKeywords-T">T</h1><p>TABLE<br />TABLES<br />TABLESAMPLE<br />THEN<br />TIME<br />TIMESTAMP<br />TIMEZONE_HOUR<br />TIMEZONE_MINUTE<br />TINYINT<br />TO<br />TRAILING<br />TRANSLATE<br />TRANSLATION<br />TREAT<br />TRIGGER<br />TRIM<br />TRUE</p><h1 id="ReservedKeywords-U">U</h1><p>UESCAPE<br />UNION<br />UNIQUE<br />UNKNOWN<br />UNNEST<br />UPDATE<br />UPPER<br />USE<br />USER<br />USING</p><h1 id="ReservedKeywords-V">V</h1><p>VALUE<br />VALUES<br />VARBINARY<br />VARCHAR<br />VARYING<br />VAR_POP<br />VAR_SAMP</p><h1 id="ReservedKeywords-W">W</h1><p>WHEN<br />WHENEVER<br />WHERE<br />WIDTH_BUCKET<br />WINDOW<br />WITH<br />WITHIN<br />WITHOUT</p><h1 id="Reserve
dKeywords-Y">Y</h1><p>YEAR</p></td></tr></tbody></table></div>
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/sql-ref/cmd-summary/001-create-table-as.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/sql-ref/cmd-summary/001-create-table-as.md b/_docs/drill-docs/sql-ref/cmd-summary/001-create-table-as.md
new file mode 100644
index 0000000..573c295
--- /dev/null
+++ b/_docs/drill-docs/sql-ref/cmd-summary/001-create-table-as.md
@@ -0,0 +1,134 @@
+---
+title: "CREATE TABLE AS (CTAS) command"
+parent: "SQL Commands Summary"
+---
+You can create tables in Drill by using the CTAS command:
+
+ CREATE TABLE new_table_name AS <query>;
+
+where query is any valid Drill query. Each table you create must have a unique
+name. You can include an optional column list for the new table. For example:
+
+ create table logtable(transid, prodid) as select transaction_id, product_id from ...
+
+You can store table data in one of three formats:
+
+ * csv
+ * parquet
+ * json
+
+The parquet and json formats can be used to store complex data.
+
+To set the output format for a Drill table, set the `store.format` option with
+the ALTER SYSTEM or ALTER SESSION command. For example:
+
+ alter session set `store.format`='json';
+
+Table data is stored in the location specified by the workspace that is in use
+when you run the CTAS statement. By default, a directory is created, using the
+exact table name specified in the CTAS statement. A .json, .csv, or .parquet
+file inside that directory contains the data.
+
+You can only create new tables in workspaces. You cannot create tables in
+other storage plugins such as Hive and HBase.
+
+You must use a writable (mutable) workspace when creating Drill tables. For
+example:
+
+ "tmp": {
+ "location": "/tmp",
+ "writable": true,
+ }
+
+## Example
+
+The following query returns one row from a JSON file:
+
+ 0: jdbc:drill:zk=local> select id, type, name, ppu
+ from dfs.`/Users/brumsby/drill/donuts.json`;
+ +------------+------------+------------+------------+
+ | id | type | name | ppu |
+ +------------+------------+------------+------------+
+ | 0001 | donut | Cake | 0.55 |
+ +------------+------------+------------+------------+
+ 1 row selected (0.248 seconds)
+
+To create and verify the contents of a table that contains this row:
+
+ 1. Set the workspace to a writable workspace.
+ 2. Set the `store.format` option appropriately.
+ 3. Run a CTAS statement that contains the query.
+ 4. Go to the directory where the table is stored and check the contents of the file.
+ 5. Run a query against the new table.
+
+The following sqlline output captures this sequence of steps.
+
+### Workspace Definition
+
+ "tmp": {
+ "location": "/tmp",
+ "writable": true,
+ }
+
+### ALTER SESSION Command
+
+ alter session set `store.format`='json';
+
+### USE Command
+
+ 0: jdbc:drill:zk=local> use dfs.tmp;
+ +------------+------------+
+ | ok | summary |
+ +------------+------------+
+ | true | Default schema changed to 'dfs.tmp' |
+ +------------+------------+
+ 1 row selected (0.03 seconds)
+
+### CTAS Command
+
+ 0: jdbc:drill:zk=local> create table donuts_json as
+ select id, type, name, ppu from dfs.`/Users/brumsby/drill/donuts.json`;
+ +------------+---------------------------+
+ | Fragment | Number of records written |
+ +------------+---------------------------+
+ | 0_0 | 1 |
+ +------------+---------------------------+
+ 1 row selected (0.107 seconds)
+
+### File Contents
+
+ administorsmbp7:tmp brumsby$ pwd
+ /tmp
+ administorsmbp7:tmp brumsby$ cd donuts_json
+ administorsmbp7:donuts_json brumsby$ more 0_0_0.json
+ {
+ "id" : "0001",
+ "type" : "donut",
+ "name" : "Cake",
+ "ppu" : 0.55
+ }
+
+### Query Against New Table
+
+ 0: jdbc:drill:zk=local> select * from donuts_json;
+ +------------+------------+------------+------------+
+ | id | type | name | ppu |
+ +------------+------------+------------+------------+
+ | 0001 | donut | Cake | 0.55 |
+ +------------+------------+------------+------------+
+ 1 row selected (0.053 seconds)
+
+### Use a Different Output Format
+
+You can run the same sequence again with a different storage format set for
+the system or session (csv or parquet). For example, if the format is set to
+csv, and you name the table donuts_csv, the resulting file would look like
+this:
+
+ administorsmbp7:tmp brumsby$ cd donuts_csv
+ administorsmbp7:donuts_csv brumsby$ ls
+ 0_0_0.csv
+ administorsmbp7:donuts_csv brumsby$ more 0_0_0.csv
+ id,type,name,ppu
+ 0001,donut,Cake,0.55
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/sql-ref/cmd-summary/002-explain.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/sql-ref/cmd-summary/002-explain.md b/_docs/drill-docs/sql-ref/cmd-summary/002-explain.md
new file mode 100644
index 0000000..4ca0870
--- /dev/null
+++ b/_docs/drill-docs/sql-ref/cmd-summary/002-explain.md
@@ -0,0 +1,166 @@
+---
+title: "EXPLAIN commands"
+parent: "SQL Commands Summary"
+---
+EXPLAIN is a useful tool for examining the steps that a query goes through
+when it is executed. You can use the EXPLAIN output to gain a deeper
+understanding of the parallel processing that Drill queries exploit. You can
+also look at costing information, troubleshoot performance issues, and
+diagnose routine errors that may occur when you run queries.
+
+Drill provides two variations on the EXPLAIN command, one that returns the
+physical plan and one that returns the logical plan. A logical plan takes the
+SQL query (as written by the user and accepted by the parser) and translates
+it into a logical series of operations that correspond to SQL language
+constructs (without defining the specific algorithms that will be implemented
+to run the query). A physical plan translates the logical plan into a specific
+series of steps that will be used when the query runs. For example, a logical
+plan may indicate a join step in general and classify it as inner or outer,
+but the corresponding physical plan will indicate the specific type of join
+operator that will run, such as a merge join or a hash join. The physical plan
+is operational and reveals the specific _access methods_ that will be used for
+the query.
+
+An EXPLAIN command for a query that is run repeatedly under the exact same
+conditions against the same data will return the same plan. However, if you
+change a configuration option, for example, or update the tables or files that
+you are selecting from, you are likely to see plan changes.
+
+# EXPLAIN Syntax
+
+The EXPLAIN command supports the following syntax:
+
+ explain plan [ including all attributes ] [ with implementation | without implementation ] for <query> ;
+
+where `query` is any valid SELECT statement supported by Drill.
+
+##### INCLUDING ALL ATTRIBUTES
+
+This option returns costing information. You can use this option for both
+physical and logical plans.
+
+#### WITH IMPLEMENTATION | WITHOUT IMPLEMENTATION
+
+These options return the physical and logical plan information, respectively.
+The default is physical (WITH IMPLEMENTATION).
+
+# EXPLAIN for Physical Plans
+
+The EXPLAIN PLAN FOR <query> command returns the chosen physical execution
+plan for a query statement without running the query. You can use this command
+to see what kind of execution operators Drill implements. For example, you can
+find out what kind of join algorithm is chosen when tables or files are
+joined. You can also use this command to analyze errors and troubleshoot
+queries that do not run. For example, if you run into a casting error, the
+query plan text may help you isolate the problem.
+
+Use the following syntax:
+
+ explain plan for <query> ;
+ explain plan with implementation for <query> ;
+
+The following set command increases the default text display (number of
+characters). By default, most of the plan output is not displayed.
+
+ 0: jdbc:drill:zk=local> !set maxwidth 10000
+
+Do not use a semicolon to terminate set commands.
+
+For example, here is the top portion of the explain output for a
+COUNT(DISTINCT) query on a JSON file:
+
+ 0: jdbc:drill:zk=local> !set maxwidth 10000
+
+ 0: jdbc:drill:zk=local> explain plan for
+ select type t, count(distinct id)
+ from dfs.`/Users/brumsby/drill/donuts.json`
+ where type='donut' group by type;
+
+ +------------+------------+
+ | text | json |
+ +------------+------------+
+ | 00-00 Screen
+ 00-01 Project(t=[$0], EXPR$1=[$1])
+ 00-02 Project(t=[$0], EXPR$1=[$1])
+ 00-03 HashAgg(group=[{0}], EXPR$1=[COUNT($1)])
+ 00-04 HashAgg(group=[{0, 1}])
+ 00-05 SelectionVectorRemover
+ 00-06 Filter(condition=[=(CAST($0):CHAR(5) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary", 'donut')])
+ 00-07 Project(type=[$1], id=[$2])
+ 00-08 ProducerConsumer
+ 00-09 Scan(groupscan=[EasyGroupScan [selectionRoot=/Users/brumsby/drill/donuts.json, columns = null]])
+ ...
+
+Read the text output from bottom to top to understand the sequence of
+operators that will execute the query. Note that the physical plan starts with
+a scan of the JSON file that is being queried. The selected columns are
+projected and filtered, then the aggregate function is applied.
+
+The EXPLAIN text output is followed by detailed JSON output, which is reusable
+for submitting the query via Drill APIs.
+
+ | {
+ "head" : {
+ "version" : 1,
+ "generator" : {
+ "type" : "ExplainHandler",
+ "info" : ""
+ },
+ "type" : "APACHE_DRILL_PHYSICAL",
+ "options" : [ ],
+ "queue" : 0,
+ "resultMode" : "EXEC"
+ },
+ ....
+
+# Costing Information
+
+Add the INCLUDING ALL ATTRIBUTES option to the EXPLAIN command to see cost
+estimates for the query plan. For example:
+
+ 0: jdbc:drill:zk=local> !set maxwidth 10000
+ 0: jdbc:drill:zk=local> explain plan including all attributes for
+ select * from dfs.`/Users/brumsby/drill/donuts.json` where type='donut';
+
+ +------------+------------+
+ | text | json |
+ +------------+------------+
+
+ | 00-00 Screen: rowcount = 1.0, cumulative cost = {4.1 rows, 14.1 cpu, 0.0 io, 0.0 network}, id = 3110
+ 00-01 Project(*=[$0], type=[$1]): rowcount = 1.0, cumulative cost = {4.0 rows, 14.0 cpu, 0.0 io, 0.0 network}, id = 3109
+ 00-02 SelectionVectorRemover: rowcount = 1.0, cumulative cost = {3.0 rows, 6.0 cpu, 0.0 io, 0.0 network}, id = 3108
+ 00-03 Filter(condition=[=(CAST($1):CHAR(5) CHARACTER SET "ISO-8859-1" COLLATE "ISO-8859-1$en_US$primary", 'donut')]): rowcount = 1.0, cumulative cost = {2.0 rows, 5.0 cpu, 0.0 io, 0.0 network}, id = 3107
+ 00-04 ProducerConsumer: rowcount = 1.0, cumulative cost = {1.0 rows, 1.0 cpu, 0.0 io, 0.0 network}, id = 3106
+ 00-05 Scan(groupscan=[EasyGroupScan [selectionRoot=/Users/brumsby/drill/donuts.json, columns = null]]): rowcount = 1.0, cumulative cost = {0.0 rows, 0.0 cpu, 0.0 io, 0.0 network}, id = 3101
+
+# EXPLAIN for Logical Plans
+
+To return the logical plan for a query (again, without actually running the
+query), use the EXPLAIN PLAN WITHOUT IMPLEMENTATION syntax:
+
+ explain plan without implementation for <query> ;
+
+For example:
+
+ 0: jdbc:drill:zk=local> explain plan without implementation for
+ select a.id
+ from dfs.`/Users/brumsby/drill/donuts.json` a, dfs.`/Users/brumsby/drill/moredonuts.json` b
+ where a.id=b.id;
+
+ +------------+------------+
+ | text | json |
+ +------------+------------+
+ | DrillScreenRel
+ DrillProjectRel(id=[$1])
+ DrillJoinRel(condition=[=($1, $3)], joinType=[inner])
+ DrillScanRel(table=[[dfs, /Users/brumsby/drill/donuts.json]], groupscan=[EasyGroupScan [selectionRoot=/Users/brumsby/drill/donuts.json, columns = null]])
+ DrillScanRel(table=[[dfs, /Users/brumsby/drill/moredonuts.json]], groupscan=[EasyGroupScan [selectionRoot=/Users/brumsby/drill/moredonuts.json, columns = null]])
+ | {
+ "head" : {
+ "version" : 1,
+ "generator" : {
+ "type" : "org.apache.drill.exec.planner.logical.DrillImplementor",
+ "info" : ""
+ },
+ ...
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/sql-ref/cmd-summary/003-select.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/sql-ref/cmd-summary/003-select.md b/_docs/drill-docs/sql-ref/cmd-summary/003-select.md
new file mode 100644
index 0000000..57fc0cf
--- /dev/null
+++ b/_docs/drill-docs/sql-ref/cmd-summary/003-select.md
@@ -0,0 +1,85 @@
+---
+title: "SELECT Statements"
+parent: "SQL Commands Summary"
+---
+Drill supports the following ANSI standard clauses in the SELECT statement:
+
+ * WITH clause
+ * SELECT list
+ * FROM clause
+ * WHERE clause
+ * GROUP BY clause
+ * HAVING clause
+ * ORDER BY clause (with an optional LIMIT clause)
+
+You can use the same SELECT syntax in the following commands:
+
+ * CREATE TABLE AS (CTAS)
+ * CREATE VIEW
+
+INSERT INTO SELECT is not yet supported.
+
+# Column Aliases
+
+You can use named column aliases in the SELECT list to provide meaningful
+names for regular columns and computed columns, such as the results of
+aggregate functions. See the section on running queries for examples.
+
+You cannot reference column aliases in the following clauses:
+
+ * WHERE
+ * GROUP BY
+ * HAVING
+
+Because Drill works with schema-less data sources, you cannot use positional
+aliases (1, 2, etc.) to refer to SELECT list columns, except in the ORDER BY
+clause.
+
+# UNION ALL Set Operator
+
+Drill supports the UNION ALL set operator to combine two result sets. The
+distinct UNION operator is not yet supported.
+
+The EXCEPT, EXCEPT ALL, INTERSECT, and INTERSECT ALL operators are not yet
+supported.
+
+# Joins
+
+Drill supports ANSI standard joins in the FROM and WHERE clauses:
+
+ * Inner joins
+ * Left, full, and right outer joins
+
+The following types of join syntax are supported:
+
+Join type| Syntax
+---|---
+Join condition in WHERE clause|FROM table1, table 2 WHERE table1.col1=table2.col1
+USING join in FROM clause|FROM table1 JOIN table2 USING(col1, ...)
+ON join in FROM clause|FROM table1 JOIN table2 ON table1.col1=table2.col1
+NATURAL JOIN in FROM clause|FROM table 1 NATURAL JOIN table 2
+
+Cross-joins are not yet supported. You must specify a join condition when more
+than one table is listed in the FROM clause.
+
+Non-equijoins are supported if the join also contains an equality condition on
+the same two tables as part of a conjunction:
+
+ table1.col1 = table2.col1 AND table1.c2 < table2.c2
+
+This restriction applies to both inner and outer joins.
+
+# Subqueries
+
+You can use the following subquery operators in Drill queries. These operators
+all return Boolean results.
+
+ * ALL
+ * ANY
+ * EXISTS
+ * IN
+ * SOME
+
+In general, correlated subqueries are supported. EXISTS and NOT EXISTS
+subqueries that do not contain a correlation join are not yet supported.
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/sql-ref/cmd-summary/004-show-files.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/sql-ref/cmd-summary/004-show-files.md b/_docs/drill-docs/sql-ref/cmd-summary/004-show-files.md
new file mode 100644
index 0000000..d0f5a83
--- /dev/null
+++ b/_docs/drill-docs/sql-ref/cmd-summary/004-show-files.md
@@ -0,0 +1,65 @@
+---
+title: "SHOW FILES Command"
+parent: "SQL Commands Summary"
+---
+The SHOW FILES command provides a quick report of the file systems that are
+visible to Drill for query purposes. This command is unique to Apache Drill.
+
+# Syntax
+
+The SHOW FILES command supports the following syntax.
+
+ SHOW FILES [ FROM filesystem.directory_name | IN filesystem.directory_name ];
+
+The FROM or IN clause is required if you do not specify a default file system
+first. You can do this with the USE command. FROM and IN are synonyms.
+
+The directory name is optional. (If the directory name is a Drill reserved
+word, you must use back ticks around the name.)
+
+The command returns standard Linux `stat` information for each file or
+directory, such as permissions, owner, and group values. This information is
+not specific to Drill.
+
+# Examples
+
+The following example returns information about directories and files in the
+local (`dfs`) file system.
+
+ 0: jdbc:drill:> use dfs;
+
+ +------------+------------+
+ | ok | summary |
+ +------------+------------+
+ | true | Default schema changed to 'dfs' |
+ +------------+------------+
+ 1 row selected (0.318 seconds)
+
+ 0: jdbc:drill:> show files;
+ +------------+-------------+------------+------------+------------+------------+-------------+------------+------------------+
+ | name | isDirectory | isFile | length | owner | group | permissions | accessTime | modificationTime |
+ +------------+-------------+------------+------------+------------+------------+-------------+------------+------------------+
+ | user | true | false | 1 | mapr | mapr | rwxr-xr-x | 2014-07-30 21:37:06.0 | 2014-07-31 22:15:53.193 |
+ | backup.tgz | false | true | 36272 | root | root | rw-r--r-- | 2014-07-31 22:09:13.0 | 2014-07-31 22:09:13.211 |
+ | JSON | true | false | 1 | root | root | rwxr-xr-x | 2014-07-31 15:22:42.0 | 2014-08-04 15:43:07.083 |
+ | scripts | true | false | 3 | root | root | rwxr-xr-x | 2014-07-31 22:10:51.0 | 2014-08-04 18:23:09.236 |
+ | temp | true | false | 2 | root | root | rwxr-xr-x | 2014-08-01 20:07:37.0 | 2014-08-01 20:09:42.595 |
+ | hbase | true | false | 10 | mapr | mapr | rwxr-xr-x | 2014-07-30 21:36:08.0 | 2014-08-04 18:31:13.778 |
+ | tables | true | false | 0 | root | root | rwxrwxrwx | 2014-07-31 22:14:35.0 | 2014-08-04 15:42:43.415 |
+ | CSV | true | false | 4 | root | root | rwxrwxrwx | 2014-07-31 17:34:53.0 | 2014-08-04
+ ...
+
+The following example shows the files in a specific directory in the `dfs`
+file system:
+
+ 0: jdbc:drill:> show files in dfs.CSV;
+
+ +------------+-------------+------------+------------+------------+------------+-------------+------------+------------------+
+ | name | isDirectory | isFile | length | owner | group | permissions | accessTime | modificationTime |
+ +------------+-------------+------------+------------+------------+------------+-------------+------------+------------------+
+ | customers.csv | false | true | 62011 | root | root | rw-r--r-- | 2014-08-04 18:30:39.0 | 2014-08-04 18:30:39.314 |
+ | products.csv.small | false | true | 34972 | root | root | rw-r--r-- | 2014-07-31 23:58:42.0 | 2014-07-31 23:59:16.849 |
+ | products.csv | false | true | 34972 | root | root | rw-r--r-- | 2014-08-01 06:39:34.0 | 2014-08-04 15:58:09.325 |
+ | products.csv.bad | false | true | 62307 | root | root | rw-r--r-- | 2014-08-04 15:58:02.0 | 2014-08-04 15:58:02.612 |
+ +------------+-------------+------------+------------+------------+------------+-------------+------------+------------------+
+ 4 rows selected (0.165 seconds)
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/sql-ref/data-types/001-date.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/sql-ref/data-types/001-date.md b/_docs/drill-docs/sql-ref/data-types/001-date.md
new file mode 100644
index 0000000..8bf4230
--- /dev/null
+++ b/_docs/drill-docs/sql-ref/data-types/001-date.md
@@ -0,0 +1,148 @@
+---
+title: "Supported Date/Time Data Type Formats"
+parent: "Data Types"
+---
+You must use supported `date` and `time` formats when you `SELECT` date and
+time literals or when you `CAST()` from `VARCHAR `to `date` and `time` data
+types. Apache Drill currently supports specific formats for the following
+`date` and `time` data types:
+
+ * Date
+ * Timestamp
+ * Time
+ * Interval
+ * Interval Year
+ * Interval Day
+ * Literal
+
+The following query provides an example of how to `SELECT` a few of the
+supported date and time formats as literals:
+
+ select date '2008-2-23', timestamp '2008-1-23 14:24:23', time '10:20:30' from dfs.`/tmp/input.json`;
+
+The following query provides an example where `VARCHAR` data in a file is
+`CAST()` to supported `date `and `time` formats:
+
+ select cast(col_A as date), cast(col_B as timestamp), cast(col_C as time) from dfs.`/tmp/dates.json`;
+
+`Date`, t`imestamp`, and` time` data types store values in `UTC`. Currently,
+Apache Drill does not support `timestamp` with time zone.
+
+## Date
+
+Drill supports the `date` data type in the following format:
+
+ YYYY-MM-DD (year-month-date)
+
+The following table provides some examples for the `date` data type:
+
+ | Use | Example |
+ | --- | ------- |
+ |Literal| `select date ‘2008-2-23’ from dfs.`/tmp/input.json`;`|
+ |`JSON` input | `{"date_col" : "2008-2-23"}
+ | `CAST` from `VARCHAR`| `` select CAST(date_col as date) as CAST_DATE from dfs.`/tmp/input.json`; ``|
+
+## Timestamp
+
+Drill supports the `timestamp` data type in the following format:
+
+ yyyy-MM-dd HH:mm:ss.SSS (year-month-date hour:minute:sec.milliseconds)
+
+The following table provides some examples for the `timestamp` data type:
+
+<table>
+ <tbody>
+ <tr>
+ <th>Use</th>
+ <th>CAST Example</th>
+ </tr>
+ <tr>
+ <td valign="top">Literal</td>
+ <td valign="top"><code><span style="color: rgb(0,0,0);">select timestamp ‘2008-2-23 10:20:30.345’, timestamp ‘2008-2-23 10:20:30’ from dfs.`/tmp/input.json`;</span></code>
+ </td></tr>
+ <tr>
+ <td colspan="1" valign="top"><code>JSON</code> Input</td>
+ <td colspan="1" valign="top"><code><span style="color: rgb(0,0,0);">{“timestamp_col”: “2008-2-23 15:20:30.345”}<br /></span><span style="color: rgb(0,0,0);">{“timestamp_col”: “2008-2-23 10:20:30”}</span></code><span style="color: rgb(0,0,0);">The fractional millisecond component is optional.</span></td>
+ </tr>
+ <tr>
+ <td colspan="1" valign="top"><code>CAST</code> from <code>VARCHAR</code></td>
+ <td colspan="1" valign="top"><code><span style="color: rgb(0,0,0);">select cast(timestamp_col as timestamp) from dfs.`/tmp/input.json`; </span></code></td>
+ </tr>
+ </tbody>
+ </table>
+
+## Time
+
+Drill supports the `time` data type in the following format:
+
+ HH:mm:ss.SSS (hour:minute:sec.milliseconds)
+
+The following table provides some examples for the` time` data type:
+
+<table><tbody><tr>
+ <th>Use</th>
+ <th>Example</th>
+ </tr>
+ <tr>
+ <td valign="top">Literal</td>
+ <td valign="top"><code><span style="color: rgb(0,0,0);">select time ‘15:20:30’, time ‘10:20:30.123’ from dfs.`/tmp/input.json`;</span></code></td>
+ </tr>
+ <tr>
+ <td colspan="1" valign="top"><code>JSON</code> Input</td>
+ <td colspan="1" valign="top"><code><span style="color: rgb(0,0,0);">{“time_col” : “10:20:30.999”}<br /></span><span style="color: rgb(0,0,0);">{“time_col”: “10:20:30”}</span></code></td>
+ </tr>
+ <tr>
+ <td colspan="1" valign="top"><code>CAST</code> from <code>VARCHAR</code></td>
+ <td colspan="1" valign="top"><code><span style="color: rgb(0,0,0);">select cast(time_col as time) from dfs.`/tmp/input.json`;</span></code></td>
+</tr></tbody>
+</table>
+
+## Interval
+
+Drill supports the `interval year` and `interval day` data types.
+
+### Interval Year
+
+The `interval year` data type stores time duration in years and months. Drill
+supports the `interval` data type in the following format:
+
+ P [qty] Y [qty] M
+
+The following table provides examples for `interval year `data type:
+
+<table ><tbody><tr>
+<th>Use</th>
+<th>Example</th></tr>
+ <tr>
+ <td valign="top">Literals</td>
+ <td valign="top"><code><span style="color: rgb(0,0,0);">select interval ‘1-2’ year to month from dfs.`/tmp/input.json`;<br /></span><span style="color: rgb(0,0,0);">select interval ‘1’ year from dfs.`/tmp/input.json`;<br /></span><span style="color: rgb(0,0,0);">select interval '13’ month from dfs.`/tmp/input.json`;</span></code></td></tr><tr>
+ <td colspan="1" valign="top"><code>JSON</code> Input</td>
+ <td colspan="1" valign="top"><code><span style="color: rgb(0,0,0);">{“col” : “P1Y2M”}<br /></span><span style="color: rgb(0,0,0);">{“col” : “P-1Y2M”}<br /></span><span style="color: rgb(0,0,0);">{“col” : “P-1Y-2M”}<br /></span><span style="color: rgb(0,0,0);">{“col”: “P10M”}<br /></span><span style="color: rgb(0,0,0);">{“col”: “P5Y”}</span></code></td>
+ </tr>
+ <tr>
+ <td colspan="1" valign="top"><code>CAST</code> from <code>VARCHAR</code></td>
+ <td colspan="1" valign="top"><code><span style="color: rgb(0,0,0);">select cast(col as interval year) from dfs.`/tmp/input.json`;</span></code></td>
+ </tr>
+ </tbody></table>
+
+### Interval Day
+
+The `interval day` data type stores time duration in days, hours, minutes, and
+seconds. You do not need to specify all fields in a given interval. Drill
+supports the `interval day` data type in the following format:
+
+ P [qty] D T [qty] H [qty] M [qty] S
+
+The following table provides examples for `interval day` data type:
+
+<div class="table-wrap"><table class="confluenceTable"><tbody><tr><th class="confluenceTh">Use</th><th class="confluenceTh">Example</th></tr><tr><td valign="top">Literal</td><td valign="top"><code><span style="color: rgb(0,0,0);">select interval '1 10:20:30.123' day to second from dfs.`/tmp/input.json`;<br /></span><span style="color: rgb(0,0,0);">select interval '1 10' day to hour from dfs.`/tmp/input.json`;<br /></span><span style="color: rgb(0,0,0);">select interval '10' day from dfs.`/tmp/input.json`;<br /></span><span style="color: rgb(0,0,0);">select interval '10' hour from dfs.`/tmp/input.json`;</span></code><code><span style="color: rgb(0,0,0);">select interval '10.999' second from dfs.`/tmp/input.json`;</span></code></td></tr><tr><td colspan="1" valign="top"><code>JSON</code> Input</td><td colspan="1" valign="top"><code><span style="color: rgb(0,0,0);">{"col" : "P1DT10H20M30S"}<br /></span><span style="color: rgb(0,0,0);">{"col" : "P1DT
10H20M30.123S"}<br /></span><span style="color: rgb(0,0,0);">{"col" : "P1D"}<br /></span><span style="color: rgb(0,0,0);">{"col" : "PT10H"}<br /></span><span style="color: rgb(0,0,0);">{"col" : "PT10.10S"}<br /></span><span style="color: rgb(0,0,0);">{"col" : "PT20S"}<br /></span><span style="color: rgb(0,0,0);">{"col" : "PT10H10S"}</span></code></td></tr><tr><td colspan="1" valign="top"><code>CAST</code> from <code>VARCHAR</code></td><td colspan="1" valign="top"><code><span style="color: rgb(0,0,0);">select cast(col as interval day) from dfs.`/tmp/input.json`;</span></code></td></tr></tbody></table></div>
+
+## Literal
+
+The following table provides a list of` date/time` literals that Drill
+supports with examples of each:
+
+<div class="table-wrap"><table class="confluenceTable"><tbody><tr><th class="confluenceTh">Format</th><th colspan="1" class="confluenceTh">Interpretation</th><th class="confluenceTh">Example</th></tr><tr><td colspan="1" valign="top"><code><span style="color: rgb(0,0,0);">interval '1 10:20:30.123' day to second</span></code></td><td colspan="1" valign="top"><code>1 day, 10 hours, 20 minutes, 30 seconds, and 123 thousandths of a second</code></td><td colspan="1" valign="top"><code><span style="color: rgb(0,0,0);">select interval '1 10:20:30.123' day to second from dfs.`/tmp/input.json`;</span></code></td></tr><tr><td colspan="1" valign="top"><code><span style="color: rgb(0,0,0);">interval '1 10' day to hour</span></code></td><td colspan="1" valign="top"><code>1 day 10 hours</code></td><td colspan="1" valign="top"><code><span style="color: rgb(0,0,0);">select interval '1 10' day to hour from dfs.`/tmp/input.json`;</span></code></td></tr><tr><td colspan="1" valign="top"><code><span styl
e="color: rgb(0,0,0);">interval '10' day</span></code></td><td colspan="1" valign="top"><code>10 days</code></td><td colspan="1" valign="top"><code><span style="color: rgb(0,0,0);">select interval '10' day from dfs.`/tmp/input.json`;</span></code></td></tr><tr><td colspan="1" valign="top"><code><span style="color: rgb(0,0,0);">interval '10' hour</span></code></td><td colspan="1" valign="top"><code>10 hours</code></td><td colspan="1" valign="top"><code><span style="color: rgb(0,0,0);">select interval '10' hour from dfs.`/tmp/input.json`;</span></code></td></tr><tr><td colspan="1" valign="top"><code><span style="color: rgb(0,0,0);">interval '10.999' second</span></code></td><td colspan="1" valign="top"><code>10.999 seconds</code></td><td colspan="1" valign="top"><code><span style="color: rgb(0,0,0);">select interval '10.999' second from dfs.`/tmp/input.json`; </span></code></td></tr></tbody></table></div>
+
+
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/sql-ref/nested/001-flatten.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/sql-ref/nested/001-flatten.md b/_docs/drill-docs/sql-ref/nested/001-flatten.md
new file mode 100644
index 0000000..124db91
--- /dev/null
+++ b/_docs/drill-docs/sql-ref/nested/001-flatten.md
@@ -0,0 +1,89 @@
+---
+title: "FLATTEN Function"
+parent: "Nested Data Functions"
+---
+The FLATTEN function is useful for flexible exploration of repeated data.
+FLATTEN separates the elements in a repeated field into individual records. To
+maintain the association between each flattened value and the other fields in
+the record, all of the other columns are copied into each new record. A very
+simple example would turn this data (one record):
+
+ {
+ "x" : 5,
+ "y" : "a string",
+ "z" : [ 1,2,3]
+ }
+
+into three distinct records:
+
+ select flatten(z) from table;
+ | x | y | z |
+ +-------------+----------------+-----------+
+ | 5 | "a string" | 1 |
+ | 5 | "a string" | 2 |
+ | 5 | "a string" | 3 |
+
+The function takes a single argument, which must be an array (the `z` column
+in this example).
+
+
+
+For a more interesting example, consider the JSON data in the publicly
+available [Yelp](https://www.yelp.com/dataset_challenge/dataset) data set. The
+first query below returns three columns from the
+`yelp_academic_dataset_business.json` file: `name`, `hours`, and `categories`.
+The query is restricted to distinct rows where the name is `z``pizza`. The
+query returns only one row that meets those criteria; however, note that this
+row contains an array of four categories:
+
+ 0: jdbc:drill:zk=local> select distinct name, hours, categories
+ from dfs.yelp.`yelp_academic_dataset_business.json`
+ where name ='zpizza';
+ +------------+------------+------------+
+ | name | hours | categories |
+ +------------+------------+------------+
+ | zpizza | {"Tuesday":{"close":"22:00","open":"10:00"},"Friday":{"close":"23:00","open":"10:00"},"Monday":{"close":"22:00","open":"10:00"},"Wednesday":{"close":"22:00","open":"10:00"},"Thursday":{"close":"22:00","open":"10:00"},"Sunday":{"close":"22:00","open":"10:00"},"Saturday":{"close":"23:00","open":"10:00"}} | ["Gluten-Free","Pizza","Vegan","Restaurants"] |
+
+The FLATTEN function can operate on this single row and return multiple rows,
+one for each category:
+
+ 0: jdbc:drill:zk=local> select distinct name, flatten(categories) as categories
+ from dfs.yelp.`yelp_academic_dataset_business.json`
+ where name ='zpizza' order by 2;
+ +------------+-------------+
+ | name | categories |
+ +------------+-------------+
+ | zpizza | Gluten-Free |
+ | zpizza | Pizza |
+ | zpizza | Restaurants |
+ | zpizza | Vegan |
+ +------------+-------------+
+ 4 rows selected (2.797 seconds)
+
+Having used the FLATTEN function to break down arrays into distinct rows, you
+can run queries that do deeper analysis on the flattened result set. For
+example, you can use FLATTEN in a subquery, then apply WHERE clause
+constraints or aggregate functions to the results in the outer query.
+
+The following query uses the same data file as the previous query to flatten
+the categories array, then run a COUNT function on the flattened result:
+
+ select celltbl.catl, count(celltbl.catl) catcount
+ from (select flatten(categories) catl
+ from dfs.yelp.`yelp_academic_dataset_business.json`) celltbl
+ group by celltbl.catl
+ order by count(celltbl.catl) desc limit 5;
+
+ +---------------+------------+
+ | catl | catcount |
+ +---------------+------------+
+ | Restaurants | 14303 |
+ | Shopping | 6428 |
+ | Food | 5209 |
+ | Beauty & Spas | 3421 |
+ | Nightlife | 2870 |
+ +---------------|------------+
+
+A common use case for FLATTEN is its use in conjunction with the
+[KVGEN](/confluence/display/DRILL/KVGEN+Function) function.
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/sql-ref/nested/002-kvgen.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/sql-ref/nested/002-kvgen.md b/_docs/drill-docs/sql-ref/nested/002-kvgen.md
new file mode 100644
index 0000000..a27a781
--- /dev/null
+++ b/_docs/drill-docs/sql-ref/nested/002-kvgen.md
@@ -0,0 +1,150 @@
+---
+title: "KVGEN Function"
+parent: "Nested Data Functions"
+---
+KVGEN stands for _key-value generation_. This function is useful when complex
+data files contain arbitrary maps that consist of relatively "unknown" column
+names. Instead of having to specify columns in the map to access the data, you
+can use KVGEN to return a list of the keys that exist in the map. KVGEN turns
+a map with a wide set of columns into an array of key-value pairs.
+
+In turn, you can write analytic queries that return a subset of the generated
+keys or constrain the keys in some way. For example, you can use the
+[FLATTEN](/confluence/display/DRILL/FLATTEN+Function) function to break the
+array down into multiple distinct rows and further query those rows.
+
+
+
+For example, assume that a JSON file contains this data:
+
+ {"a": "valA", "b": "valB"}
+ {"c": "valC", "d": "valD"}
+
+
+KVGEN would operate on this data to generate:
+
+ [{"key": "a", "value": "valA"}, {"key": "b", "value": "valB"}]
+ [{"key": "c", "value": "valC"}, {"key": "d", "value": "valD"}]
+
+Applying the [FLATTEN](/confluence/display/DRILL/FLATTEN+Function) function to
+this data would return:
+
+ {"key": "a", "value": "valA"}
+ {"key": "b", "value": "valB"}
+ {"key": "c", "value": "valC"}
+ {"key": "d", "value": "valD"}
+
+Assume that a JSON file called `kvgendata.json` includes multiple records that
+look like this one:
+
+ {
+ "rownum": 1,
+ "bigintegercol": {
+ "int_1": 1,
+ "int_2": 2,
+ "int_3": 3
+ },
+ "varcharcol": {
+ "varchar_1": "abc",
+ "varchar_2": "def",
+ "varchar_3": "xyz"
+ },
+ "boolcol": {
+ "boolean_1": true,
+ "boolean_2": false,
+ "boolean_3": true
+ },
+ "float8col": {
+ "f8_1": 1.1,
+ "f8_2": 2.2
+ },
+ "complex": [
+ {
+ "col1": 3
+ },
+ {
+ "col2": 2,
+ "col3": 1
+ },
+ {
+ "col1": 7
+ }
+ ]
+ }
+
+ {
+ "rownum": 3,
+ "bigintegercol": {
+ "int_1": 1,
+ "int_3": 3
+ },
+ "varcharcol": {
+ "varchar_1": "abcde",
+ "varchar_2": null,
+ "varchar_3": "xyz",
+ "varchar_4": "xyz2"
+ },
+ "boolcol": {
+ "boolean_1": true,
+ "boolean_2": false
+ },
+ "float8col": {
+ "f8_1": 1.1,
+ "f8_3": 6.6
+ },
+ "complex": [
+ {
+ "col1": 2,
+ "col3": 1
+ }
+ ]
+ }
+ ...
+
+
+A SELECT * query against this specific record returns the following row:
+
+ 0: jdbc:drill:zk=local> select * from dfs.yelp.`kvgendata.json` where rownum=1;
+
+ +------------+---------------+------------+------------+------------+------------+
+ | rownum | bigintegercol | varcharcol | boolcol | float8col | complex |
+ +------------+---------------+------------+------------+------------+------------+
+ | 1 | {"int_1":1,"int_2":2,"int_3":3} | {"varchar_1":"abc","varchar_2":"def","varchar_3":"xyz"} | {"boolean_1":true,"boolean_2":false,"boolean_3":true} | {"f8_1":1.1,"f8_2":2.2} | [{"col1":3},{"col2":2,"col3":1},{"col1":7}] |
+ +------------+---------------+------------+------------+------------+------------+
+ 1 row selected (0.122 seconds)
+
+You can use the KVGEN function to turn the maps in this data into key-value
+pairs. For example:
+
+ 0: jdbc:drill:zk=local> select kvgen(varcharcol) from dfs.yelp.`kvgendata.json`;
+ +------------+
+ | EXPR$0 |
+ +------------+
+ | [{"key":"varchar_1","value":"abc"},{"key":"varchar_2","value":"def"},{"key":"varchar_3","value":"xyz"}] |
+ | [{"key":"varchar_1","value":"abcd"}] |
+ | [{"key":"varchar_1","value":"abcde"},{"key":"varchar_3","value":"xyz"},{"key":"varchar_4","value":"xyz2"}] |
+ | [{"key":"varchar_1","value":"abc"},{"key":"varchar_2","value":"def"}] |
+ +------------+
+ 4 rows selected (0.091 seconds)
+
+Now you can apply the FLATTEN function to break out the key-value pairs into
+distinct rows:
+
+ 0: jdbc:drill:zk=local> select flatten(kvgen(varcharcol)) from dfs.yelp.`kvgendata.json`;
+ +------------+
+ | EXPR$0 |
+ +------------+
+ | {"key":"varchar_1","value":"abc"} |
+ | {"key":"varchar_2","value":"def"} |
+ | {"key":"varchar_3","value":"xyz"} |
+ | {"key":"varchar_1","value":"abcd"} |
+ | {"key":"varchar_1","value":"abcde"} |
+ | {"key":"varchar_3","value":"xyz"} |
+ | {"key":"varchar_4","value":"xyz2"} |
+ | {"key":"varchar_1","value":"abc"} |
+ | {"key":"varchar_2","value":"def"} |
+ +------------+
+ 9 rows selected (0.151 seconds)
+
+See the description of [FLATTEN](/confluence/display/DRILL/FLATTEN+Function)
+for an example of a query against the flattened data.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/sql-ref/nested/003-repeated-cnt.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/sql-ref/nested/003-repeated-cnt.md b/_docs/drill-docs/sql-ref/nested/003-repeated-cnt.md
new file mode 100644
index 0000000..a66075c
--- /dev/null
+++ b/_docs/drill-docs/sql-ref/nested/003-repeated-cnt.md
@@ -0,0 +1,34 @@
+---
+title: "REPEATED_COUNT Function"
+parent: "Nested Data Functions"
+---
+This function counts the values in an array. The following example returns the
+counts for the `categories` array in the `yelp_academic_dataset_business.json`
+file. The counts are restricted to rows that contain the string `pizza`.
+
+ SELECT name, REPEATED_COUNT(categories)
+ FROM dfs.yelp.`yelp_academic_dataset_business.json`
+ WHERE name LIKE '%pizza%';
+
+ +---------------+------------+
+ | name | EXPR$1 |
+ +---------------+------------+
+ | Villapizza | 2 |
+ | zpizza | 4 |
+ | zpizza | 4 |
+ | Luckys pizza | 2 |
+ | Zpizza | 2 |
+ | S2pizzabar | 4 |
+ | Dominos pizza | 5 |
+ +---------------+------------+
+
+ 7 rows selected (2.03 seconds)
+
+The function requires a single argument, which must be an array. Note that
+this function is not a standard SQL aggregate function and does not require
+the count to be grouped by other columns in the select list (such as `name` in
+this example).
+
+For another example of this function, see the following lesson in the Apache
+Drill Tutorial for Hadoop: [Lesson 3: Run Queries on Complex Data Types](/conf
+luence/display/DRILL/Lesson+3%3A+Run+Queries+on+Complex+Data+Types).
\ No newline at end of file
[3/6] drill git commit: Added Drill docs
Posted by ts...@apache.org.
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/manage/conf/003-plan-exec.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/manage/conf/003-plan-exec.md b/_docs/drill-docs/manage/conf/003-plan-exec.md
new file mode 100644
index 0000000..c045e8f
--- /dev/null
+++ b/_docs/drill-docs/manage/conf/003-plan-exec.md
@@ -0,0 +1,37 @@
+---
+title: "Planning and Execution Options"
+parent: "Configuration Options"
+---
+You can set Drill query planning and execution options per cluster, at the
+system or session level. Options set at the session level only apply to
+queries that you run during the current Drill connection. Options set at the
+system level affect the entire system and persist between restarts. Session
+level settings override system level settings.
+
+#### Querying Planning and Execution Options
+
+You can run the following query to see a list of the system and session
+planning and execution options:
+
+ SELECT name FROM sys.options WHERE type in ('SYSTEM','SESSION');
+
+#### Configuring Planning and Execution Options
+
+Use the` ALTER SYSTEM` or `ALTER SESSION` commands to set options. Typically,
+you set the options at the session level unless you want the setting to
+persist across all sessions.
+
+The following table contains planning and execution options that you can set
+at the system or session level:
+
+<div class="table-wrap"><table class="confluenceTable"><tbody><tr><th class="confluenceTh">Option name</th><th class="confluenceTh">Default value</th><th class="confluenceTh">Description</th></tr><tr><td valign="top" colspan="1" class="confluenceTd">exec.errors.verbose</td><td valign="top" colspan="1" class="confluenceTd"><p>false</p></td><td valign="top" colspan="1" class="confluenceTd"><p>This option enables or disables the verbose message that Drill returns when a query fails. When enabled, Drill provides additional information about failed queries.</p></td></tr><tr><td valign="top" colspan="1" class="confluenceTd"><span>exec.max_hash_table_size</span></td><td valign="top" colspan="1" class="confluenceTd">1073741824</td><td valign="top" colspan="1" class="confluenceTd"><span>The default maximum size for hash tables.</span></td></tr><tr><td valign="top" colspan="1" class="confluenceTd">exec.min_hash_table_size</td><td valign="top" colspan="1" class="confluenceTd">65536</td><td val
ign="top" colspan="1" class="confluenceTd">The default starting size for hash tables. Increasing this size is useful for very large aggregations or joins when you have large amounts of memory for Drill to use. Drill can spend a lot of time resizing the hash table as it finds new data. If you have large data sets, you can increase this hash table size to increase performance.</td></tr><tr><td valign="top" colspan="1" class="confluenceTd">planner.add_producer_consumer</td><td valign="top" colspan="1" class="confluenceTd"><p>false</p><p> </p></td><td valign="top" colspan="1" class="confluenceTd"><p>This option enables or disables a secondary reading thread that works out of band of the rest of the scanning fragment to prefetch data from disk. <span style="line-height: 1.4285715;background-color: transparent;">If you interact with a certain type of storage medium that is slow or does not prefetch much data, this option tells Drill to add a producer consumer reading thread to the operati
on. Drill can then assign one thread that focuses on a single reading fragment. </span></p><p>If Drill is using memory, you can disable this option to get better performance. If Drill is using disk space, you should enable this option and set a reasonable queue size for the planner.producer_consumer_queue_size option.</p></td></tr><tr><td valign="top" colspan="1" class="confluenceTd">planner.broadcast_threshold</td><td valign="top" colspan="1" class="confluenceTd">1000000</td><td valign="top" colspan="1" class="confluenceTd"><span style="color: rgb(34,34,34);">Threshold, in terms of a number of rows, that determines whether a broadcast join is chosen for a query. Regardless of the setting of the broadcast_join option (enabled or disabled), a broadcast join is not chosen unless the right side of the join is estimated to contain fewer rows than this threshold. The intent of this option is to avoid broadcasting too many rows for join purposes. Broadcasting involves sending data across
nodes and is a network-intensive operation. (The "right side" of the join, which may itself be a join or simply a table, is determined by cost-based optimizations and heuristics during physical planning.)</span></td></tr><tr><td valign="top" colspan="1" class="confluenceTd"><p>planner.enable_broadcast_join<br />planner.enable_hashagg<br />planner.enable_hashjoin<br />planner.enable_mergejoin<br />planner.enable_multiphase_agg<br />planner.enable_streamagg</p></td><td valign="top" colspan="1" class="confluenceTd">true</td><td valign="top" colspan="1" class="confluenceTd"><p>These options enable or disable specific aggregation and join operators for queries. These operators are all enabled by default and in general should not be disabled.</p><p>Hash aggregation and hash join are hash-based operations. Streaming aggregation and merge join are sort-based operations. Both hash-based and sort-based operations consume memory; however, currently, hash-based operations do not spill
to disk as needed, but the sort-based operations do. If large hash operations do not fit in memory on your system, you may need to disable these operations. Queries will continue to run, using alternative plans.</p></td></tr><tr><td valign="top" colspan="1" class="confluenceTd">planner.producer_consumer_queue_size</td><td valign="top" colspan="1" class="confluenceTd">10</td><td valign="top" colspan="1" class="confluenceTd">Determines how much data to prefetch from disk (in record batches) out of band of query execution. The larger the queue size, the greater the amount of memory that the queue and overall query execution consumes.</td></tr><tr><td valign="top" colspan="1" class="confluenceTd">planner.slice_target</td><td valign="top" colspan="1" class="confluenceTd">100000</td><td valign="top" colspan="1" class="confluenceTd">The number of records manipulated within a fragment before Drill parallelizes them.</td></tr><tr><td valign="top" colspan="1" class="confluenceTd"><p>planner.
width.max_per_node</p><p> </p></td><td valign="top" colspan="1" class="confluenceTd"><p>The default depends on the number of cores on each node.</p></td><td valign="top" colspan="1" class="confluenceTd"><p>In this context "width" refers to fanout or distribution potential: the ability to run a query in parallel across the cores on a node and the nodes on a cluster.</p><p><span>A physical plan consists of intermediate operations, known as query "fragments," that run concurrently, yielding opportunities for parallelism above and below each exchange operator in the plan. An exchange operator represents a breakpoint in the execution flow where processing can be distributed. For example, a single-process scan of a file may flow into an exchange operator, followed by a multi-process aggregation fragment.</span><span> </span></p><p>The maximum width per node defines the maximum degree of parallelism for any fragment of a query, but the setting applies at the level of a
single node in the cluster.</p><p>The <em>default</em> maximum degree of parallelism per node is calculated as follows, with the theoretical maximum automatically scaled back (and rounded down) so that only 70% of the actual available capacity is taken into account:</p><div class="code panel pdl" style="border-width: 1px;"><div class="codeContent panelContent pdl">
+<script type="syntaxhighlighter" class="theme: Default; brush: java; gutter: false"><![CDATA[number of active drillbits (typically one per node)
+* number of cores per node
+* 0.7]]></script>
+</div></div><p>For example, on a single-node test system with 2 cores and hyper-threading enabled:</p><div class="code panel pdl" style="border-width: 1px;"><div class="codeContent panelContent pdl">
+<script type="syntaxhighlighter" class="theme: Default; brush: java; gutter: false"><![CDATA[1 * 4 * 0.7 = 3]]></script>
+</div></div><p>When you modify the default setting, you can supply any meaningful number. The system does not automatically scale down your setting.</p></td></tr><tr><td valign="top" colspan="1" class="confluenceTd">planner.width.max_per_query</td><td valign="top" colspan="1" class="confluenceTd">1000</td><td valign="top" colspan="1" class="confluenceTd"><p>The max_per_query value also sets the maximum degree of parallelism for any given stage of a query, but the setting applies to the query as executed by the whole cluster (multiple nodes). In effect, the actual maximum width per query is the <em>minimum of two values</em>:</p><div class="code panel pdl" style="border-width: 1px;"><div class="codeContent panelContent pdl">
+<script type="syntaxhighlighter" class="theme: Default; brush: java; gutter: false"><![CDATA[min((number of nodes * width.max_per_node), width.max_per_query)]]></script>
+</div></div><p>For example, on a 4-node cluster where <span><code>width.max_per_node</code> is set to 6 and </span><span><code>width.max_per_query</code> is set to 30:</span></p><div class="code panel pdl" style="border-width: 1px;"><div class="codeContent panelContent pdl">
+<script type="syntaxhighlighter" class="theme: Default; brush: java; gutter: false"><![CDATA[min((4 * 6), 30) = 24]]></script>
+</div></div><p>In this case, the effective maximum width per query is 24, not 30.</p></td></tr><tr><td valign="top" colspan="1" class="confluenceTd">store.format</td><td valign="top" colspan="1" class="confluenceTd"> </td><td valign="top" colspan="1" class="confluenceTd">Output format for data that is written to tables with the CREATE TABLE AS (CTAS) command.</td></tr><tr><td valign="top" colspan="1" class="confluenceTd">store.json.all_text_mode</td><td valign="top" colspan="1" class="confluenceTd"><p>false</p></td><td valign="top" colspan="1" class="confluenceTd"><p>This option enables or disables text mode. When enabled, Drill reads everything in JSON as a text object instead of trying to interpret data types. This allows complicated JSON to be read using CASE and CAST.</p></td></tr><tr><td valign="top" class="confluenceTd">store.parquet.block-size</td><td valign="top" class="confluenceTd"><p>536870912</p></td><td valign="top" class="confluenceTd">T<span style="color: rgb(34,34,34
);">arget size for a parquet row group, which should be equal to or less than the configured HDFS block size. </span></td></tr></tbody></table></div>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/manage/conf/004-persist-conf.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/manage/conf/004-persist-conf.md b/_docs/drill-docs/manage/conf/004-persist-conf.md
new file mode 100644
index 0000000..99c22b5
--- /dev/null
+++ b/_docs/drill-docs/manage/conf/004-persist-conf.md
@@ -0,0 +1,93 @@
+---
+title: "Persistent Configuration Storage"
+parent: "Configuration Options"
+---
+Drill stores persistent configuration data in a persistent configuration store
+(PStore). This data is encoded in JSON or Protobuf format. Drill can use the
+local file system, ZooKeeper, HBase, or MapR-DB to store this data. The data
+stored in a PStore includes state information for storage plugins, query
+profiles, and ALTER SYSTEM settings. The default type of PStore configured
+depends on the Drill installation mode.
+
+The following table provides the persistent storage mode for each of the Drill
+modes:
+
+<div class="table-wrap"><table class="confluenceTable"><tbody><tr><th class="confluenceTh">Mode</th><th class="confluenceTh">Description</th></tr><tr><td valign="top" class="confluenceTd">Embedded</td><td valign="top" class="confluenceTd">Drill stores persistent data in the local file system. <br />You cannot modify the PStore location for Drill in embedded mode.</td></tr><tr><td valign="top" class="confluenceTd">Distributed</td><td valign="top" class="confluenceTd">Drill stores persistent data in ZooKeeper, by default. <br />You can modify where ZooKeeper offloads data, <br />or you can change the persistent storage mode to HBase or MapR-DB.</td></tr></tbody></table></div>
+
+**Note:** Switching between storage modes does not migrate configuration data.
+
+## ZooKeeper for Persistent Configuration Storage
+
+To make Drill installation and configuration simple, Drill uses ZooKeeper to
+store persistent configuration data. The ZooKeeper PStore provider stores all
+of the persistent configuration data in ZooKeeper except for query profile
+data.
+
+The ZooKeeper PStore provider offloads query profile data to the
+${DRILL_LOG_DIR:-/var/log/drill} directory on Drill nodes. If you want the
+query profile data stored in a specific location, you can configure where
+ZooKeeper offloads the data.
+
+To modify where the ZooKeeper PStore provider offloads query profile data,
+configure the `sys.store.provider.zk.blobroot` property in the `drill.exec`
+block in `<drill_installation_directory>/conf/drill-override.conf` on each
+Drill node and then restart the Drillbit service.
+
+**Example**
+
+ drill.exec: {
+ cluster-id: "my_cluster_com-drillbits",
+ zk.connect: "<zkhostname>:<port>",
+ sys.store.provider.zk.blobroot: "maprfs://<directory to store pstore data>/"
+ }
+
+Issue the following command to restart the Drillbit on all Drill nodes:
+
+ maprcli node services -name drill-bits -action restart -nodes <node IP addresses separated by a space>
+
+## HBase for Persistent Configuration Storage
+
+To change the persistent storage mode for Drill, add or modify the
+`sys.store.provider` block in `<drill_installation_directory>/conf/drill-
+override.conf.`
+
+**Example**
+
+ sys.store.provider: {
+ class: "org.apache.drill.exec.store.hbase.config.HBasePStoreProvider",
+ hbase: {
+ table : "drill_store",
+ config: {
+ "hbase.zookeeper.quorum": "<ip_address>,<ip_address>,<ip_address >,<ip_address>",
+ "hbase.zookeeper.property.clientPort": "2181"
+ }
+ }
+ },
+
+## MapR-DB for Persistent Configuration Storage
+
+The MapR-DB plugin will be released soon. You can [compile Drill from
+source](/confluence/display/DRILL/Compiling+Drill+from+Source) to try out this
+new feature.
+
+If you have MapR-DB in your cluster, you can use MapR-DB for persistent
+configuration storage. Using MapR-DB to store persistent configuration data
+can prevent memory strain on ZooKeeper in clusters running heavy workloads.
+
+To change the persistent storage mode to MapR-DB, add or modify the
+`sys.store.provider` block in `<drill_installation_directory>/conf/drill-
+override.conf` on each Drill node and then restart the Drillbit service.
+
+**Example**
+
+ sys.store.provider: {
+ class: "org.apache.drill.exec.store.hbase.config.HBasePStoreProvider",
+ hbase: {
+ table : "/tables/drill_store",
+ }
+ },
+
+Issue the following command to restart the Drillbit on all Drill nodes:
+
+ maprcli node services -name drill-bits -action restart -nodes <node IP addresses separated by a space>
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/progress/001-2014-q1.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/progress/001-2014-q1.md b/_docs/drill-docs/progress/001-2014-q1.md
new file mode 100644
index 0000000..1909753
--- /dev/null
+++ b/_docs/drill-docs/progress/001-2014-q1.md
@@ -0,0 +1,204 @@
+---
+title: "2014 Q1 Drill Report"
+parent: "Progress Reports"
+---
+
+Apache: Project Drill
+
+Description:
+
+Apache Drill is a distributed system for interactive analysis of large-scale
+datasets that is based on Google's Dremel. Its goal is to efficiently process
+nested data, scale to 10,000 servers or
+
+more and to be able to process petabyes of data and trillions of records in
+seconds.
+
+Drill has been incubating since 2012-08-11.
+
+Three Issues to Address in Move to Graduation:
+
+1\. Continue to attract new developers and and early users with a variety of
+skills and viewpoints
+
+2\. Continue to develop deeper community skills and knowledge by building
+additional releases
+
+3\. Demonstrate community robustness by rotating project tasks among multiple
+project members
+
+Issues to Call to Attention of PMC or ASF Board:
+
+None
+
+How community has developed since last report:
+
+Community awareness and participation were strengthened through a meeting of
+the Bay Area Apache Drill User Group in San Jose
+
+sponsored by Yahoo! This event expanded participation to include many new to
+Drill and particularly those interested as potential users (analysts rather
+than developers).
+
+Speakers included Drill project mentor Ted Dunning from MapR, Data Scientist
+Will Ford from Alpine Data Labs, new Drill committer Julian Hyde from
+HortonWorks and Aman Sinha, MapR Drill engineer.
+
+Additional events include:
+
+• Two new Drill committers accepted appointment:
+
+Julian Hyde (HortonWorks) and Tim Chen (Microsoft).
+
+• Drill has a new project mentor, Sebastian Schelter.
+
+Mailing list discussions:
+
+Subscriptions to the Drill mailing lists have risen to 399 on dev list and 308
+on the user list and 508 uniques across both lists.
+
+There has been active and increasing participation in discussions on the
+developer mailing list, including new participants and
+
+developers. Participation on the user list is growing although still small;
+mainly activity takes place on developer mailing list.
+
+Activity summary for the user mailing list:
+
+<http://mail-archives.apache.org/mod_mbox/incubator-drill-user/>
+
+February to date 02/26/2014: 25
+
+January 2014, 12
+
+December 2013, 62
+
+Topics in discussion on the user mailing list included but not limited to:
+
+ * Feb 2014: Connecting Drill to HBase, Support for Distinct/Count
+ * Jan 2014: Loading Data into Drill, Data Locality
+ * December 2013: Loading Data into Drill, Setting Drill with HDFS and other Storage engines
+
+Activity summary for the dev mailing list:
+
+<http://mail-archives.apache.org/mod_mbox/incubator-drill-dev/>
+
+February to date 02/26/2014: 250 (jira; discussion; review
+
+requests)
+
+January2014, 156(jira, focused discussions)
+
+December 2013, 51 (jira; focused discussions)
+
+Topics in discussion on the dev mailing list included but not
+
+limited to:
+
+• February to date 02/26/2014: How to contribute to Drill;
+
+review requests for Drill 357, 346, 366, 364; status of
+
+Drill functions including Hash functions; support operators
+
++,- for date and interval arithmetic
+
+• January: Sql Options discussions, Casting discussions, Multiplex Data
+Channel feedbacks
+
+• December: Guide for new comers contribution, Aggregate functions code gen
+feedback
+
+Code
+
+For details of code commits, see <http://bit.ly/14YPXN9>
+
+There has been continued activity in code commits
+
+19 contributors have participated in GitHUB code activity; there
+
+have been 116 forks.
+
+February code commits include but not limited to: Support for
+Information_schema, Hive storage and metastore integration, Optiq JDBC
+thinning and refactoring, Math functions rework to use codegen, Column pruning
+for Parquet/Json, Moving Sql parsing into Drillbit server side, TravisCI setup
+
+January code commits include but not limited to: Implicit and explicit casting
+support, Broadcast Sender exchange, add TPC-H test queries, Refactor memory
+allocation to use hierarchical memory allocation and freeing.
+
+Community Interactions
+
+Weekly Drill hangout continues, conducted remotely through Google hangouts
+Tuesday mornings 9am Pacific Time to keep
+
+core developers in contact in realtime despite geographical separation.
+
+Community stays in touch through @ApacheDrill Twitter ID, and by postings on
+various blogs including Apache Drill User <http://drill-user.org/> which has
+had several updates and through
+
+international presentations at conferences.
+
+Viability of community is also apparent through active participation in the
+Bay Area Apache Drill User group meeting in early November, which has grown to
+440 members.
+
+Sample presentations:
+
+• “How to Use Drill” by Ted Dunning and Will Ford, Bay Area Apache Drill Meet-
+up 24 February
+
+• “How Drill Addresses Dynamic Typing” by Julian Hyde, Bay Area Apache Drill
+Meet-up 24 February
+
+• “New Features and Infrastructure Improvements” by Aman Sinha, Bay Area
+Apache Drill Meet-up 24 February
+
+Articles
+
+Examples of articles or reports on Apache Drill since last report
+
+include:
+
+• Drill blog post by Ellen Friedman at Apache Drill User updating community on
+how people will use Drill and inviting comments/ questions from remote
+participants as
+
+part of the Drill User Group <http://bit.ly/1p1Qvgn>
+
+• Drill blog post by Ellen Friedman at Apache Drill User reports on
+appointment of new Drill committers and new mentor <http://bit.ly/JIcwQe>
+
+Social Networking
+
+@ApacheDrill Twitter entity is active and has grown substantially
+
+by 19%, to 744 followers.
+
+How project has developed since last report:
+
+1\. Significant progress is being made on execution engine and sql front end
+to support more functionality, also more integrations with storage engines.
+
+2\. Work on ODBC driver has begun with a new group led by George Chow in
+Vancouver.
+
+3\. Significant code drops have been checked in from a number of contributors
+and commiters
+
+4\. Work toward 2nd milestone is progressing substantially.
+
+Please check this [ ] when you have filled in the report for Drill.
+
+Signed-off-by:
+
+Ted Dunning: [](drill)
+
+Grant Ingersoll: [ ](drill)
+
+Isabel Drost: [ ](drill)
+
+Sebastian Schelter: [ ](drill)
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/query/001-query-fs.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/query/001-query-fs.md b/_docs/drill-docs/query/001-query-fs.md
new file mode 100644
index 0000000..bd69c17
--- /dev/null
+++ b/_docs/drill-docs/query/001-query-fs.md
@@ -0,0 +1,44 @@
+---
+title: "Querying a File System"
+parent: "Query"
+---
+Files and directories are like standard SQL tables to Drill. You can specify a
+file system "database" as a prefix in queries when you refer to objects across
+databases. In Drill, a file system database consists of a storage plugin name
+followed by an optional workspace name, for example <storage
+plugin>.<workspace> or hdfs.logs.
+
+
+
+The following example shows a query on a file system database in a Hadoop
+distributed file system:
+
+``SELECT * FROM hdfs.logs.`AppServerLogs/20104/Jan/01/part0001.txt`;``
+
+The default `dfs` storage plugin instance registered with Drill has a
+`default` workspace. If you query data in the `default` workspace, you do not
+need to include the workspace in the query. Refer to
+[Workspaces](https://cwiki.apache.org/confluence/display/DRILL/Workspaces) for
+more information.
+
+Drill supports the following file types:
+
+ * Plain text files, including:
+ * Comma-separated values (CSV, type: text)
+ * Tab-separated values (TSV, type: text)
+ * Pipe-separated values (PSV, type: text)
+ * Structured data files:
+ * JSON (type: json)
+ * Parquet (type: parquet)
+
+The extensions for these file types must match the configuration settings for
+your registered storage plugins. For example, PSV files may be defined with a
+`.tbl` extension, while CSV files are defined with a `.csv` extension.
+
+Click on any of the following links for more information about querying
+various file types:
+
+ * [Querying JSON Files](/confluence/display/DRILL/Querying+JSON+Files)
+ * [Querying Parquet Files](/confluence/display/DRILL/Querying+Parquet+Files)
+ * [Querying Plain Text Files](/confluence/display/DRILL/Querying+Plain+Text+Files)
+ * [Querying Directories](/confluence/display/DRILL/Querying+Directories)
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/query/002-query-hbase.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/query/002-query-hbase.md b/_docs/drill-docs/query/002-query-hbase.md
new file mode 100644
index 0000000..2fb0a10
--- /dev/null
+++ b/_docs/drill-docs/query/002-query-hbase.md
@@ -0,0 +1,177 @@
+---
+title: "Querying HBase"
+parent: "Query"
+---
+This is a simple exercise that provides steps for creating a “students” table
+and a “clicks” table in HBase that you can query with Drill.
+
+To create the HBase tables and query them with Drill, complete the following
+steps:
+
+ 1. Issue the following command to start the HBase shell:
+
+ hbase shell
+
+ 2. Issue the following commands to create a ‘students’ table and a ‘clicks’ table with column families in HBase:
+
+
+ ``echo "create 'students','account','address'" | hbase shell``
+
+ ``echo "create 'clicks','clickinfo','iteminfo'" | hbase shell``
+
+ 3. Issue the following command with the provided data to create a `testdata.txt` file:
+
+ `cat > testdata.txt`
+
+ **Sample Data**
+
+ put 'students','student1','account:name','Alice'
+ put 'students','student1','address:street','123 Ballmer Av'
+ put 'students','student1','address:zipcode','12345'
+ put 'students','student1','address:state','CA'
+ put 'students','student2','account:name','Bob'
+ put 'students','student2','address:street','1 Infinite Loop'
+ put 'students','student2','address:zipcode','12345'
+ put 'students','student2','address:state','CA'
+ put 'students','student3','account:name','Frank'
+ put 'students','student3','address:street','435 Walker Ct'
+ put 'students','student3','address:zipcode','12345'
+ put 'students','student3','address:state','CA'
+ put 'students','student4','account:name','Mary'
+ put 'students','student4','address:street','56 Southern Pkwy'
+ put 'students','student4','address:zipcode','12345'
+ put 'students','student4','address:state','CA'
+ put 'clicks','click1','clickinfo:studentid','student1'
+ put 'clicks','click1','clickinfo:url','http://www.google.com'
+ put 'clicks','click1','clickinfo:time','2014-01-01 12:01:01.0001'
+ put 'clicks','click1','iteminfo:itemtype','image'
+ put 'clicks','click1','iteminfo:quantity','1'
+ put 'clicks','click2','clickinfo:studentid','student1'
+ put 'clicks','click2','clickinfo:url','http://www.amazon.com'
+ put 'clicks','click2','clickinfo:time','2014-01-01 01:01:01.0001'
+ put 'clicks','click2','iteminfo:itemtype','image'
+ put 'clicks','click2','iteminfo:quantity','1'
+ put 'clicks','click3','clickinfo:studentid','student2'
+ put 'clicks','click3','clickinfo:url','http://www.google.com'
+ put 'clicks','click3','clickinfo:time','2014-01-01 01:02:01.0001'
+ put 'clicks','click3','iteminfo:itemtype','text'
+ put 'clicks','click3','iteminfo:quantity','2'
+ put 'clicks','click4','clickinfo:studentid','student2'
+ put 'clicks','click4','clickinfo:url','http://www.ask.com'
+ put 'clicks','click4','clickinfo:time','2013-02-01 12:01:01.0001'
+ put 'clicks','click4','iteminfo:itemtype','text'
+ put 'clicks','click4','iteminfo:quantity','5'
+ put 'clicks','click5','clickinfo:studentid','student2'
+ put 'clicks','click5','clickinfo:url','http://www.reuters.com'
+ put 'clicks','click5','clickinfo:time','2013-02-01 12:01:01.0001'
+ put 'clicks','click5','iteminfo:itemtype','text'
+ put 'clicks','click5','iteminfo:quantity','100'
+ put 'clicks','click6','clickinfo:studentid','student3'
+ put 'clicks','click6','clickinfo:url','http://www.google.com'
+ put 'clicks','click6','clickinfo:time','2013-02-01 12:01:01.0001'
+ put 'clicks','click6','iteminfo:itemtype','image'
+ put 'clicks','click6','iteminfo:quantity','1'
+ put 'clicks','click7','clickinfo:studentid','student3'
+ put 'clicks','click7','clickinfo:url','http://www.ask.com'
+ put 'clicks','click7','clickinfo:time','2013-02-01 12:45:01.0001'
+ put 'clicks','click7','iteminfo:itemtype','image'
+ put 'clicks','click7','iteminfo:quantity','10'
+ put 'clicks','click8','clickinfo:studentid','student4'
+ put 'clicks','click8','clickinfo:url','http://www.amazon.com'
+ put 'clicks','click8','clickinfo:time','2013-02-01 22:01:01.0001'
+ put 'clicks','click8','iteminfo:itemtype','image'
+ put 'clicks','click8','iteminfo:quantity','1'
+ put 'clicks','click9','clickinfo:studentid','student4'
+ put 'clicks','click9','clickinfo:url','http://www.amazon.com'
+ put 'clicks','click9','clickinfo:time','2013-02-01 22:01:01.0001'
+ put 'clicks','click9','iteminfo:itemtype','image'
+ put 'clicks','click9','iteminfo:quantity','10'
+
+ 4. Issue the following command to verify that the data is in the `testdata.txt` file:
+
+ `cat testdata.txt | hbase shell`
+
+ 5. Issue `exit` to leave the `hbase shell`.
+ 6. Start Drill. Refer to [Starting/Stopping Drill](/confluence/pages/viewpage.action?pageId=44994063) for instructions.
+ 7. Use Drill to issue the following SQL queries on the “students” and “clicks” tables:
+ a. Issue the following query to see the data in the “students” table:
+
+ ``SELECT * FROM hbase.`students`;``
+
+ The query returns binary results:
+
+
+ `Query finished, fetching results ...`
+
+ `+----------+----------+----------+-----------+----------+----------+----------+-----------+`
+
+ `|id | name | state | street | zipcode |`
+
+ `+----------+----------+----------+-----------+----------+-----------+----------+-----------`
+
+ `| [B@1ee37126 | [B@661985a1 | [B@15944165 | [B@385158f4 | [B@3e08d131 |`
+
+ `| [B@64a7180e | [B@161c72c2 | [B@25b229e5 | [B@53dc8cb8 |[B@1d11c878 |`
+
+ `| [B@349aaf0b | [B@175a1628 | [B@1b64a812 | [B@6d5643ca |[B@147db06f |`
+
+ `| [B@3a7cbada | [B@52cf5c35 | [B@2baec60c | [B@5f4c543b |[B@2ec515d6 |`
+
+ Since Drill does not require metadata, you must use the SQL `CAST` function in
+some queries to get readable query results.
+
+ b. Issue the following query, that includes the `CAST` function, to see the data in the “`students`” table:
+
+ `SELECT CAST(students.clickinfo.studentid as VarChar(20)),
+CAST(students.account.name as VarChar(20)), CAST (students.address.state as
+VarChar(20)), CAST (students.address.street as VarChar(20)), CAST
+(students.address.zipcode as VarChar(20)), FROM hbase.students;`
+
+ **Note:** Use the following format when you query a column in an HBase table:
+ `tablename.columnfamilyname.columnname`
+ For more information about column families, refer to [5.6. Column
+Family](http://hbase.apache.org/book/columnfamily.html).
+
+ The query returns the data:
+
+
+ `Query finished, fetching results ...`
+
+ `+----------+-------+-------+------------------+---------+`
+
+ `| studentid | name | state | street | zipcode |`
+
+ `+----------+-------+-------+------------------+---------+`
+
+ `| student1 | Alice | CA | 123 Ballmer Av | 12345 |`
+
+ `| student2 | Bob | CA | 1 Infinite Loop | 12345 |`
+
+ `| student3 | Frank | CA | 435 Walker Ct | 12345 |`
+
+ `| student4 | Mary | CA | 56 Southern Pkwy | 12345 |`
+
+ `+----------+-------+-------+------------------+---------+`
+
+ c. Issue the following query on the “clicks” table to find out which students clicked on google.com:
+
+ ``SELECT CAST(clicks.clickinfo.studentid as VarChar(200)), CAST(clicks.clickinfo.url as VarChar(200)) FROM hbase.`clicks` WHERE URL LIKE '%google%';``
+
+ The query returns the data:
+
+
+ `Query finished, fetching results ...`
+
+ `+---------+-----------+-------------------------------+-----------------------+----------+----------+`
+
+ `| clickid | studentid | time | url | itemtype | quantity |`
+
+ `+---------+-----------+-------------------------------+-----------------------+----------+----------+`
+
+ `| click1 | student1 | 2014-01-01 12:01:01.000100000 | http://www.google.com | image | 1 |`
+
+ `| click3 | student2 | 2014-01-01 01:02:01.000100000 | http://www.google.com | text | 2 |`
+
+ `| click6 | student3 | 2013-02-01 12:01:01.000100000 | http://www.google.com | image | 1 |`
+
+ `+---------+-----------+-------------------------------+-----------------------+----------+----------+`
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/query/003-query-hive.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/query/003-query-hive.md b/_docs/drill-docs/query/003-query-hive.md
new file mode 100644
index 0000000..7af069d
--- /dev/null
+++ b/_docs/drill-docs/query/003-query-hive.md
@@ -0,0 +1,67 @@
+---
+title: "Querying Hive"
+parent: "Query"
+---
+This is a simple exercise that provides steps for creating a Hive table and
+inserting data that you can query using Drill. Before you perform the steps,
+download the [customers.csv](http://doc.mapr.com/download/attachments/22906623
+/customers.csv?api=v2) file.
+
+To create a Hive table and query it with Drill, complete the following steps:
+
+ 1. Issue the following command to start the Hive shell:
+
+ hive
+
+ 2. Issue the following command from the Hive shell to import the `customers.csv` file and create a table:
+
+ hive> create table customers(FirstName string,
+ LastName string,Company string,Address string,
+ City string,County string,State string,Zip string,
+ Phone string,Fax string,Email string,Web string)
+ row format delimited fields terminated by ',' stored as textfile;
+
+ 3. Issue the following command to load the customer data into the customers table:
+
+ `Hive> load data local inpath '/<directory path>/customers.csv' overwrite into table customers;`
+
+ 4. Issue `quit` or `exit` to leave the Hive shell.
+ 5. Start Drill. Refer to [Starting/Stopping Drill](/confluence/pages/viewpage.action?pageId=44994063) for instructions.
+ 6. Issue the following query to Drill to get the first and last names of the first ten customers in the Hive table:
+
+ `0: jdbc:drill:schema=hiveremote> SELECT firstname,lastname FROM hiveremote.`customers` limit 10;`
+
+ The query returns the following results:
+
+ `+------------+------------+`
+
+ `| firstname | lastname |`
+
+ `+------------+------------+`
+
+ `| Essie | Vaill |`
+
+ `| Cruz | Roudabush |`
+
+ `| Billie | Tinnes |`
+
+ `| Zackary | Mockus |`
+
+ `| Rosemarie | Fifield |`
+
+ `| Bernard | Laboy |`
+
+ `| Sue | Haakinson |`
+
+ `| Valerie | Pou |`
+
+ `| Lashawn | Hasty |`
+
+ `| Marianne | Earman |`
+
+ `+------------+------------+`
+
+ `10 rows selected (1.5 seconds)`
+
+ `0: jdbc:drill:schema=hiveremote>`
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/query/004-query-complex.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/query/004-query-complex.md b/_docs/drill-docs/query/004-query-complex.md
new file mode 100644
index 0000000..1685d62
--- /dev/null
+++ b/_docs/drill-docs/query/004-query-complex.md
@@ -0,0 +1,63 @@
+---
+title: "Querying Complex Data"
+parent: "Query"
+---
+Apache Drill queries do not require prior knowledge of the actual data you are
+trying to access, regardless of its source system or its schema and data
+types. The sweet spot for Apache Drill is a SQL query workload against
+"complex data": data made up of various types of records and fields, rather
+than data in a recognizable relational form (discrete rows and columns). Drill
+is capable of discovering the form of the data when you submit the query.
+Nested data formats such as JSON (JavaScript Object Notation) files and
+Parquet files are not only _accessible_: Drill provides special operators and
+functions that you can use to _drill down _into these files and ask
+interesting analytic questions.
+
+These operators and functions include:
+
+ * References to nested data values
+ * Access to repeating values in arrays and arrays within arrays (array indexes)
+
+The SQL query developer needs to know the data well enough to write queries
+that identify values of interest in the target file. For example, the writer
+needs to know what a record consists of, and its data types, in order to
+reliably request the right "columns" in the select list. Although these data
+values do not manifest themselves as columns in the source file, Drill will
+return them in the result set as if they had the predictable form of columns
+in a table. Drill also optimizes queries by treating the data as "columnar"
+rather than reading and analyzing complete records. (Drill uses similar
+parallel execution and optimization capabilities to commercial columnar MPP
+databases.)
+
+Given a basic knowledge of the input file, the developer needs to know how to
+use the SQL extensions that Drill provides and how to use them to "reach into"
+the nested data. The following examples show how to write both simple queries
+against JSON files and interesting queries that unpack the nested data. The
+examples show how to use the Drill extensions in the context of standard SQL
+SELECT statements. For the most part, the extensions use standard JavaScript
+notation for referencing data elements in a hierarchy.
+
+### Before You Begin
+
+The examples in this section operate on JSON data files. In order to write
+your own queries, you need to be aware of the basic data types in these files:
+
+ * string (all data inside double quotes), such as `"0001"` or `"Cake"`
+ * numeric types: integers, decimals, and floats, such as `0.55` or `10`
+ * null values
+ * boolean values: true, false
+
+Check that you have the following configuration setting for JSON files in the
+Drill Web UI (`dfs` storage plugin configuration):
+
+ "json" : {
+ "type" : "json"
+ }
+
+Click on any of the following links to see examples of complex queries:
+
+ * [Sample Data: Donuts](/confluence/display/DRILL/Sample+Data%3A+Donuts)
+ * [Query 1: Selecting Flat Data](/confluence/display/DRILL/Query+1%3A+Selecting+Flat+Data)
+ * [Query 2: Using Standard SQL Functions, Clauses, and Joins](/confluence/display/DRILL/Query+2%3A+Using+Standard+SQL+Functions%2C+Clauses%2C+and+Joins)
+ * [Query 3: Selecting Nested Data for a Column](/confluence/display/DRILL/Query+3%3A+Selecting+Nested+Data+for+a+Column)
+ * [Query 4: Selecting Multiple Columns Within Nested Data](/confluence/display/DRILL/Query+4%3A+Selecting+Multiple+Columns+Within+Nested+Data)
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/query/005-query-info-skema.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/query/005-query-info-skema.md b/_docs/drill-docs/query/005-query-info-skema.md
new file mode 100644
index 0000000..5de4d4e
--- /dev/null
+++ b/_docs/drill-docs/query/005-query-info-skema.md
@@ -0,0 +1,109 @@
+---
+title: "Querying the INFORMATION SCHEMA"
+parent: "Query"
+---
+When you are using Drill to connect to multiple data sources, you need a
+simple mechanism to discover what each data source contains. The information
+schema is an ANSI standard set of metadata tables that you can query to return
+information about all of your Drill data sources (or schemas). Data sources
+may be databases or file systems; they are all known as "schemas" in this
+context. You can query the following INFORMATION_SCHEMA tables:
+
+ * SCHEMATA
+ * CATALOGS
+ * TABLES
+ * COLUMNS
+ * VIEWS
+
+## SCHEMATA
+
+The SCHEMATA table contains the CATALOG_NAME and SCHEMA_NAME columns. To allow
+maximum flexibility inside BI tools, the only catalog that Drill supports is
+`DRILL`.
+
+ 0: jdbc:drill:zk=local> select CATALOG_NAME, SCHEMA_NAME as all_my_data_sources from INFORMATION_SCHEMA.SCHEMATA order by SCHEMA_NAME;
+ +--------------+---------------------+
+ | CATALOG_NAME | all_my_data_sources |
+ +--------------+---------------------+
+ | DRILL | INFORMATION_SCHEMA |
+ | DRILL | cp.default |
+ | DRILL | dfs.default |
+ | DRILL | dfs.root |
+ | DRILL | dfs.tmp |
+ | DRILL | HiveTest.SalesDB |
+ | DRILL | maprfs.logs |
+ | DRILL | sys |
+ +--------------+---------------------+
+
+The INFORMATION_SCHEMA name and associated keywords are case-sensitive. You
+can also return a list of schemas by running the SHOW DATABASES command:
+
+ 0: jdbc:drill:zk=local> show databases;
+ +-------------+
+ | SCHEMA_NAME |
+ +-------------+
+ | dfs.default |
+ | dfs.root |
+ | dfs.tmp |
+ ...
+
+## CATALOGS
+
+The CATALOGS table returns only one row, with the hardcoded DRILL catalog name
+and description.
+
+## TABLES
+
+The TABLES table returns the table name and type for each table or view in
+your databases. (Type means TABLE or VIEW.) Note that Drill does not return
+files available for querying in file-based data sources. Instead, use SHOW
+FILES to explore these data sources.
+
+## COLUMNS
+
+The COLUMNS table returns the column name and other metadata (such as the data
+type) for each column in each table or view.
+
+## VIEWS
+
+The VIEWS table returns the name and definition for each view in your
+databases. Note that file schemas are the canonical repository for views in
+Drill. Depending on how you create a view, the may only be displayed in Drill
+after it has been used.
+
+## Useful Queries
+
+Run an ``INFORMATION_SCHEMA.`TABLES` ``query to view all of the tables and views
+within a database. TABLES is a reserved word in Drill and requires back ticks
+(`).
+
+For example, the following query identifies all of the tables and views that
+Drill can access:
+
+ SELECT TABLE_SCHEMA, TABLE_NAME, TABLE_TYPE
+ FROM INFORMATION_SCHEMA.`TABLES`
+ ORDER BY TABLE_NAME DESC;
+ ----------------------------------------------------------------
+ TABLE_SCHEMA TABLE_NAME TABLE_TYPE
+ ----------------------------------------------------------------
+ HiveTest.CustomersDB Customers TABLE
+ HiveTest.SalesDB Orders TABLE
+ HiveTest.SalesDB OrderLines TABLE
+ HiveTest.SalesDB USOrders VIEW
+ dfs.default CustomerSocialProfile VIEW
+ ----------------------------------------------------------------
+
+**Note:** Currently, Drill only supports querying Drill views; Hive views are not yet supported.
+
+You can run a similar query to identify columns in tables and the data types
+of those columns:
+
+ SELECT COLUMN_NAME, DATA_TYPE
+ FROM INFORMATION_SCHEMA.COLUMNS
+ WHERE TABLE_NAME = 'Orders' AND TABLE_SCHEMA = 'HiveTest.SalesDB' AND COLUMN_NAME LIKE '%Total';
+ +-------------+------------+
+ | COLUMN_NAME | DATA_TYPE |
+ +-------------+------------+
+ | OrderTotal | Decimal |
+ +-------------+------------+
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/query/006-query-sys-tbl.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/query/006-query-sys-tbl.md b/_docs/drill-docs/query/006-query-sys-tbl.md
new file mode 100644
index 0000000..1cfd3bd
--- /dev/null
+++ b/_docs/drill-docs/query/006-query-sys-tbl.md
@@ -0,0 +1,176 @@
+---
+title: "Querying System Tables"
+parent: "Query"
+---
+Drill has a sys database that contains system tables. You can query the system
+tables for information about Drill, including Drill ports, the Drill version
+running on the system, and available Drill options. View the databases in
+Drill to identify the sys database, and then use the sys database to view
+system tables that you can query.
+
+## View Drill Databases
+
+Issue the `SHOW DATABASES` command to view Drill databases.
+
+ 0: jdbc:drill:zk=10.10.100.113:5181> show databases;
+ +-------------+
+ | SCHEMA_NAME |
+ +-------------+
+ | M7 |
+ | hive.default|
+ | dfs.default |
+ | dfs.root |
+ | dfs.views |
+ | dfs.tmp |
+ | dfs.tpcds |
+ | sys |
+ | cp.default |
+ | hbase |
+ | INFORMATION_SCHEMA |
+ +-------------+
+ 11 rows selected (0.162 seconds)
+
+Drill returns `sys` in the database results.
+
+## Use the Sys Database
+
+Issue the `USE` command to select the sys database for subsequent SQL
+requests.
+
+ 0: jdbc:drill:zk=10.10.100.113:5181> use sys;
+ +------------+--------------------------------+
+ | ok | summary |
+ +------------+--------------------------------+
+ | true | Default schema changed to 'sys' |
+ +------------+--------------------------------+
+ 1 row selected (0.101 seconds)
+
+## View Tables
+
+Issue the `SHOW TABLES` command to view the tables in the sys database.
+
+ 0: jdbc:drill:zk=10.10.100.113:5181> show tables;
+ +--------------+------------+
+ | TABLE_SCHEMA | TABLE_NAME |
+ +--------------+------------+
+ | sys | drillbits |
+ | sys | version |
+ | sys | options |
+ +--------------+------------+
+ 3 rows selected (0.934 seconds)
+ 0: jdbc:drill:zk=10.10.100.113:5181>
+
+## Query System Tables
+
+Query the drillbits, version, and options tables in the sys database.
+
+###### Query the drillbits table.
+
+ 0: jdbc:drill:zk=10.10.100.113:5181> select * from drillbits;
+ +------------------+------------+--------------+------------+---------+
+ | host | user_port | control_port | data_port | current|
+ +-------------------+------------+--------------+------------+--------+
+ | qa-node115.qa.lab | 31010 | 31011 | 31012 | true |
+ | qa-node114.qa.lab | 31010 | 31011 | 31012 | false |
+ | qa-node116.qa.lab | 31010 | 31011 | 31012 | false |
+ +------------+------------+--------------+------------+---------------+
+ 3 rows selected (0.146 seconds)
+
+ * host
+The name of the node running the Drillbit service.
+
+ * user-port
+The user port address, used between nodes in a cluster for connecting to
+external clients and for the Drill Web UI.
+
+ * control_port
+The control port address, used between nodes for multi-node installation of
+Apache Drill.
+
+ * data_port
+The data port address, used between nodes for multi-node installation of
+Apache Drill.
+
+ * current
+True means the Drillbit is connected to the session or client running the
+query. This Drillbit is the Foreman for the current session.
+
+###### Query the version table.
+
+ 0: jdbc:drill:zk=10.10.100.113:5181> select * from version;
+ +------------+----------------+-------------+-------------+------------+
+ | commit_id | commit_message | commit_time | build_email | build_time |
+ +------------+----------------+-------------+-------------+------------+
+ | 108d29fce3d8465d619d45db5f6f433ca3d97619 | DRILL-1635: Additional fix for validation exceptions. | 14.11.2014 @ 02:32:47 UTC | Unknown | 14.11.2014 @ 03:56:07 UTC |
+ +------------+----------------+-------------+-------------+------------+
+ 1 row selected (0.144 seconds)
+
+ * commit_id
+The github id of the release you are running. For example, <https://github.com
+/apache/drill/commit/e3ab2c1760ad34bda80141e2c3108f7eda7c9104>
+
+ * commit_message
+The message explaining the change.
+
+ * commit_time
+The date and time of the change.
+
+ * build_email
+The email address of the person who made the change, which is unknown in this
+example.
+
+ * build_time
+The time that the release was built.
+
+###### Query the options table.
+
+Drill provides system, session, and boot options that you can query.
+
+The following example shows a query on the system options:
+
+ 0: jdbc:drill:zk=10.10.100.113:5181> select * from options where type='SYSTEM' limit 10;
+ +------------+------------+------------+------------+------------+------------+------------+
+ | name | kind | type | num_val | string_val | bool_val | float_val |
+ +------------+------------+------------+------------+------------+------------+------------+
+ | exec.max_hash_table_size | LONG | SYSTEM | 1073741824 | null | null | null |
+ | planner.memory.max_query_memory_per_node | LONG | SYSTEM | 2048 | null | null | null |
+ | planner.join.row_count_estimate_factor | DOUBLE | SYSTEM | null | null | null | 1.0 |
+ | planner.affinity_factor | DOUBLE | SYSTEM | null | null | null | 1.2 |
+ | exec.errors.verbose | BOOLEAN | SYSTEM | null | null | false | null |
+ | planner.disable_exchanges | BOOLEAN | SYSTEM | null | null | false | null |
+ | exec.java_compiler_debug | BOOLEAN | SYSTEM | null | null | true | null |
+ | exec.min_hash_table_size | LONG | SYSTEM | 65536 | null | null | null |
+ | exec.java_compiler_janino_maxsize | LONG | SYSTEM | 262144 | null | null | null |
+ | planner.enable_mergejoin | BOOLEAN | SYSTEM | null | null | true | null |
+ +------------+------------+------------+------------+------------+------------+------------+
+ 10 rows selected (0.334 seconds)
+
+ * name
+The name of the option.
+
+ * kind
+The data type of the option value.
+
+ * type
+The type of options in the output: system, session, or boot.
+
+ * num_val
+The default value, which is of the long or int data type; otherwise, null.
+
+ * string_val
+The default value, which is a string; otherwise, null.
+
+ * bool_val
+The default value, which is true or false; otherwise, null.
+
+ * float_val
+The default value, which is of the double, float, or long double data type;
+otherwise, null.
+
+For information about how to configure Drill system and session options, see[
+Planning and Execution Options](https://cwiki.apache.org/confluence/display/DR
+ILL/Planning+and+Execution+Options).
+
+For information about how to configure Drill start-up options, see[ Start-Up
+Options](https://cwiki.apache.org/confluence/display/DRILL/Start-Up+Options).
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/query/007-interfaces.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/query/007-interfaces.md b/_docs/drill-docs/query/007-interfaces.md
new file mode 100644
index 0000000..5dc69c4
--- /dev/null
+++ b/_docs/drill-docs/query/007-interfaces.md
@@ -0,0 +1,16 @@
+---
+title: "Drill Interfaces"
+parent: "Query"
+---
+You can connect to Apache Drill through the following interfaces:
+
+ * Drill shell (SQLLine)
+ * Drill Web UI
+ * ODBC*
+ * [JDBC](/confluence/display/DRILL/Using+JDBC+to+Access+Apache+Drill+from+SQuirreL)
+ * C++ API
+
+*Apache Drill does not have an open source ODBC driver. However, MapR provides an ODBC driver that you can use to connect to Apache Drill from BI tools. For more information, refer to the following documents:
+
+ * [Using JDBC to Access Apache Drill from SQuirreL](/confluence/display/DRILL/Using+JDBC+to+Access+Apache+Drill+from+SQuirreL)
+ * [Using ODBC to Access Apache Drill from BI Tools](/confluence/display/DRILL/Using+ODBC+to+Access+Apache+Drill+from+BI+Tools)
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/query/interfaces/001-jdbc.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/query/interfaces/001-jdbc.md b/_docs/drill-docs/query/interfaces/001-jdbc.md
new file mode 100644
index 0000000..d2c5dd3
--- /dev/null
+++ b/_docs/drill-docs/query/interfaces/001-jdbc.md
@@ -0,0 +1,138 @@
+---
+title: "Using JDBC to Access Apache Drill from SQuirreL"
+parent: "Drill Interfaces"
+---
+You can connect to Drill through a JDBC client tool, such as SQuirreL, on
+Windows, Linux, and Mac OS X systems, to access all of your data sources
+registered with Drill. An embedded JDBC driver is included with Drill.
+Configure the JDBC driver in the SQuirreL client to connect to Drill from
+SQuirreL. This document provides instruction for connecting to Drill from
+SQuirreL on Windows.
+
+To use the Drill JDBC driver with SQuirreL on Windows, complete the following
+steps:
+
+ * Step 1: Getting the Drill JDBC Driver
+ * Step 2: Installing and Starting SQuirreL
+ * Step 3: Adding the Drill JDBC Driver to SQuirreL
+ * Step 4: Running a Drill Query from SQuirreL
+
+For information about how to use SQuirreL, refer to the [SQuirreL Quick
+Start](http://squirrel-sql.sourceforge.net/user-manual/quick_start.html)
+guide.
+
+### Prerequisites
+
+ * SQuirreL requires JRE 7
+ * Drill installed in distributed mode on one or multiple nodes in a cluster. Refer to the [Install Drill](https://cwiki.apache.org/confluence/display/DRILL/Install+Drill) documentation for more information.
+ * The client must be able to resolve the actual hostname of the Drill node(s) with the IP(s). Verify that a DNS entry was created on the client machine for the Drill node(s).
+If a DNS entry does not exist, create the entry for the Drill node(s).
+
+ * For Windows, create the entry in the %WINDIR%\system32\drivers\etc\hosts file.
+
+ * For Linux and Mac, create the entry in /etc/hosts.
+<drill-machine-IP> <drill-machine-hostname>
+Example: `127.0.1.1 maprdemo`
+
+## Step 1: Getting the Drill JDBC Driver
+
+The Drill JDBC Driver `JAR` file must exist in a directory on your Windows
+machine in order to configure the driver in the SQuirreL client.
+
+You can copy the Drill JDBC `JAR` file from the following Drill installation
+directory on the node with Drill installed, to a directory on your Windows
+machine:
+
+ <drill_installation_directory>/jars/jdbc-driver/drill-jdbc-all-0.7.0-SNAPSHOT.jar
+
+Or, you can download the [apache-
+drill-0.7.0.tar.gz](http://www.apache.org/dyn/closer.cgi/drill/drill-0.7.0
+/apache-drill-0.7.0.tar.gz) file to a location on your Windows machine, and
+extract the contents of the file. You may need to use a decompression utility,
+such as [7-zip](http://www.7-zip.org/) to extract the archive. Once extracted,
+you can locate the driver in the following directory:
+
+ <windows_directory>\apache-drill-<version>\jars\jdbc-driver\drill-jdbc-all-0.7.0-SNAPSHOT.jar
+
+## Step 2: Installing and Starting SQuirreL
+
+To install and start SQuirreL, complete the following steps:
+
+ 1. Download the SQuirreL JAR file for Windows from the following location:
+<http://www.squirrelsql.org/#installation>
+
+ 2. Double-click the SQuirreL `JAR` file. The SQuirreL installation wizard walks you through the installation process.
+ 3. When installation completes, navigate to the SQuirreL installation folder and then double-click `squirrel-sql.bat` to start SQuirreL.
+
+## Step 3: Adding the Drill JDBC Driver to SQuirreL
+
+To add the Drill JDBC Driver to SQuirreL, define the driver and create a
+database alias. The alias is a specific instance of the driver configuration.
+SQuirreL uses the driver definition and alias to connect to Drill so you can
+access data sources that you have registered with Drill.
+
+### A. Define the Driver
+
+To define the Drill JDBC Driver, complete the following steps:
+
+ 1. In the SQuirreL toolbar, select **Drivers > New Driver**. The Add Driver dialog box appears.
+
+ 
+
+ 2. Enter the following information:
+
+ <table class="confluenceTable"><tbody><tr><td valign="top"><p><strong>Option</strong></p></td><td valign="top"><p><strong>Description</strong></p></td></tr><tr><td valign="top"><p>Name</p></td><td valign="top"><p>Name for the Drill JDBC Driver</p></td></tr><tr><td valign="top"><p>Example URL</p></td><td valign="top"><p><code>jdbc:drill:zk=<<em>zookeeper_quorum</em>>[;schema=<<em>schema_to_use_as_default</em>>]</code></p><p><strong>Example:</strong><code> jdbc:drill:zk=maprdemo:5181</code></p><p><strong>Note:</strong> The default ZooKeeper port is 2181. In a MapR cluster, the ZooKeeper port is 5181.</p></td></tr><tr><td valign="top"><p>Website URL</p></td><td valign="top"><p><code>jdbc:drill:zk=<<em>zookeeper_quorum</em>>[;schema=<<em>schema_to_use_as_default</em>>]</code></p><p><strong>Example:</strong><code><code> jdbc:drill:zk=maprdemo:5181</code></code></p><p><strong>Note:</strong><span> The default ZooKeeper port is 2181. In a MapR cluster, the ZooKe
eper port is 5181.</span></p></td></tr><tr><td valign="top"><p>Extra Class Path</p></td><td valign="top"><p>Click <strong>Add</strong> and navigate to the JDBC <code>JAR</code> file location in the Windows directory:<br /><code><windows_directory>/jars/jdbc-driver/<span style="color: rgb(34,34,34);">drill-jdbc-all-0.6.0-</span><span style="color: rgb(34,34,34);">incubating.jar</span></code></p><p>Select the <code>JAR</code> file, click <strong>Open</strong>, and then click <strong>List Drivers</strong>.</p></td></tr><tr><td valign="top"><p>Class Name</p></td><td valign="top"><p>Select <code>org.apache.drill.jdbc.Driver</code> from the drop-down menu.</p></td></tr></tbody></table>
+
+ 3. Click **OK**. The SQuirreL client displays a message stating that the driver registration is successful, and you can see the driver in the Drivers panel.
+
+ 
+
+### B. Create an Alias
+
+To create an alias, complete the following steps:
+
+ 1. Select the **Aliases** tab.
+ 2. In the SQuirreL toolbar, select **Aliases >****New Alias**. The Add Alias dialog box appears.
+
+ 
+
+ 3. Enter the following information:
+
+ <table class="confluenceTable"><tbody><tr><td valign="top"><p><strong>Option</strong></p></td><td valign="top"><p><strong>Description</strong></p></td></tr><tr><td valign="top"><p>Alias Name</p></td><td valign="top"><p>A unique name for the Drill JDBC Driver alias.</p></td></tr><tr><td valign="top"><p>Driver</p></td><td valign="top"><p>Select the Drill JDBC Driver.</p></td></tr><tr><td valign="top"><p>URL</p></td><td valign="top"><p>Enter the connection URL with <span>the name of the Drill directory stored in ZooKeeper and the cluster ID:</span></p><p><code>jdbc:drill:zk=<<em>zookeeper_quorum</em>>/<drill_directory_in_zookeeper>/<cluster_ID>;schema=<<em>schema_to_use_as_default</em>></code></p><p><strong>The following examples show URLs for Drill installed on a single node:</strong><br /><span style="font-family: monospace;font-size: 14.0px;line-height: 1.4285715;background-color: transparent;">jdbc:drill:zk=10.10.100.56:5181/drill/demo_mapr_com-drillbit
s;schema=hive<br /></span><span style="font-family: monospace;font-size: 14.0px;line-height: 1.4285715;background-color: transparent;">jdbc:drill:zk=10.10.100.24:2181/drill/drillbits1;schema=hive<br /> </span></p><div><strong>The following example shows a URL for Drill installed in distributed mode with a connection to a ZooKeeper quorum:</strong></div><div><span style="font-family: monospace;font-size: 14.0px;line-height: 1.4285715;background-color: transparent;">jdbc:drill:zk=10.10.100.30:5181,10.10.100.31:5181,10.10.100.32:5181/drill/drillbits1;schema=hive</span></div> <div class="aui-message warning shadowed information-macro">
+ <span class="aui-icon icon-warning"></span>
+ <div class="message-content">
+ <ul><li style="list-style-type: none;background-image: none;"><ul><li>Including a default schema is optional.</li><li>The ZooKeeper port is 2181. In a MapR cluster, the ZooKeeper port is 5181.</li><li>The Drill directory stored in ZooKeeper is <code>/drill</code>. </li><li>The Drill default cluster ID is<code> drillbits1</code>.</li></ul></li></ul>
+ </div>
+ </div>
+</td></tr><tr><td valign="top"><p>User Name</p></td><td valign="top"><p>admin</p></td></tr><tr><td valign="top"><p>Password</p></td><td valign="top"><p>admin</p></td></tr></tbody></table>
+
+
+ 4. Click **Ok. **The Connect to: dialog box appears.
+
+ 
+
+ 5. Click **Connect.** SQuirreL displays a message stating that the connection is successful.
+
+
+ 6. Click **OK**. SQuirreL displays a series of tabs.
+
+## Step 4: Running a Drill Query from SQuirreL
+
+Once you have SQuirreL successfully connected to your cluster through the
+Drill JDBC Driver, you can issue queries from the SQuirreL client. You can run
+a test query on some sample data included in the Drill installation to try out
+SQuirreL with Drill.
+
+To query sample data with Squirrel, complete the following steps:
+
+ 1. Click the  tab.
+ 2. Enter the following query in the query box:
+``SELECT * FROM cp.`employee.json`;``
+Example:
+ 
+
+ 3. Press **Ctrl+Enter** to run the query. The following query results display:
+ 
+
+You have successfully run a Drill query from the SQuirreL client.
+
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/query/interfaces/002-odbc.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/query/interfaces/002-odbc.md b/_docs/drill-docs/query/interfaces/002-odbc.md
new file mode 100644
index 0000000..1bb82bb
--- /dev/null
+++ b/_docs/drill-docs/query/interfaces/002-odbc.md
@@ -0,0 +1,23 @@
+---
+title: "Using ODBC to Access Apache Drill from BI Tools"
+parent: "Drill Interfaces"
+---
+MapR provides ODBC drivers for Windows, Mac OS X, and Linux. It is recommended
+that you install the latest version of Apache Drill with the latest version of
+the Drill ODBC driver.
+
+For example, if you have Apache Drill 0.5 and a Drill ODBC driver installed on
+your machine, and then you upgrade to Apache Drill 0.6, do not assume that the
+Drill ODBC driver installed on your machine will work with the new version of
+Apache Drill. Install the latest available Drill ODBC driver to ensure that
+the two components work together.
+
+You can access the latest Drill ODBC drivers in the following location:
+
+`<http://package.mapr.com/tools/MapR-ODBC/MapR_Drill/MapRDrill_odbc/>`
+
+Refer to the following documents for driver installation and configuration
+information, as well as examples for connecting to BI tools:
+
+ * [Using the MapR ODBC Driver on Windows](/confluence/display/DRILL/Using+the+MapR+ODBC+Driver+on+Windows)
+ * [Using the MapR Drill ODBC Driver on Linux and Mac OS X](/confluence/display/DRILL/Using+the+MapR+Drill+ODBC+Driver+on+Linux+and+Mac+OS+X)
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/query/query-complex/001-sample-donuts.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/query/query-complex/001-sample-donuts.md b/_docs/drill-docs/query/query-complex/001-sample-donuts.md
new file mode 100644
index 0000000..37010ec
--- /dev/null
+++ b/_docs/drill-docs/query/query-complex/001-sample-donuts.md
@@ -0,0 +1,40 @@
+---
+title: "Sample Data: Donuts"
+parent: "Query Complex Data"
+---
+The complex data queries use sample `donuts.json` and `moredonuts.json` files.
+Here is the single complete "record" (`0001`) from the `donuts.json `file. In
+terms of Drill query processing, this record is equivalent to a single record
+in a table.
+
+ {
+ "id": "0001",
+ "type": "donut",
+ "name": "Cake",
+ "ppu": 0.55,
+ "batters":
+ {
+ "batter":
+ [
+ { "id": "1001", "type": "Regular" },
+ { "id": "1002", "type": "Chocolate" },
+ { "id": "1003", "type": "Blueberry" },
+ { "id": "1004", "type": "Devil's Food" }
+ ]
+ },
+ "topping":
+ [
+ { "id": "5001", "type": "None" },
+ { "id": "5002", "type": "Glazed" },
+ { "id": "5005", "type": "Sugar" },
+ { "id": "5007", "type": "Powdered Sugar" },
+ { "id": "5006", "type": "Chocolate with Sprinkles" },
+ { "id": "5003", "type": "Chocolate" },
+ { "id": "5004", "type": "Maple" }
+ ]
+ }
+
+The data is made up of maps, arrays, and nested arrays. Name-value pairs and
+embedded name-value pairs define the contents of each record. For example,
+`type: donut` is a map. Under `topping`, the pairs of `id` and `type` values
+belong to an array (inside the square brackets).
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/query/query-complex/002-query1-select.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/query/query-complex/002-query1-select.md b/_docs/drill-docs/query/query-complex/002-query1-select.md
new file mode 100644
index 0000000..a5fe5ea
--- /dev/null
+++ b/_docs/drill-docs/query/query-complex/002-query1-select.md
@@ -0,0 +1,19 @@
+---
+title: "Query 1: Selecting Flat Data"
+parent: "Query Complex Data"
+---
+A very simple query against the `donuts.json` file returns the values for the
+four "flat" columns (the columns that contain data at the top level only: no
+nested data):
+
+ 0: jdbc:drill:zk=local> select id, type, name, ppu
+ from dfs.`/Users/brumsby/drill/donuts.json`;
+ +------------+------------+------------+------------+
+ | id | type | name | ppu |
+ +------------+------------+------------+------------+
+ | 0001 | donut | Cake | 0.55 |
+ +------------+------------+------------+------------+
+ 1 row selected (0.248 seconds)
+
+Note that `dfs` is the schema name, the path to the file is enclosed by
+backticks, and the query must end with a semicolon.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/query/query-complex/003-query2-use-sql.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/query/query-complex/003-query2-use-sql.md b/_docs/drill-docs/query/query-complex/003-query2-use-sql.md
new file mode 100644
index 0000000..cf614ad
--- /dev/null
+++ b/_docs/drill-docs/query/query-complex/003-query2-use-sql.md
@@ -0,0 +1,74 @@
+---
+title: "Query 2: Using Standard SQL Functions, Clauses, and Joins"
+parent: "Query Complex Data"
+---
+You can use standard SQL clauses, such as WHERE and ORDER BY, to elaborate on
+this kind of simple query:
+
+ 0: jdbc:drill:zk=local> select id, type from dfs.`/Users/brumsby/drill/donuts.json`
+ where id>0
+ order by id limit 1;
+
+ +------------+------------+
+
+ | id | type |
+
+ +------------+------------+
+
+ | 0001 | donut |
+
+ +------------+------------+
+
+ 1 row selected (0.318 seconds)
+
+You can also join files (or tables, or files and tables) by using standard
+syntax:
+
+ 0: jdbc:drill:zk=local> select tbl1.id, tbl1.type from dfs.`/Users/brumsby/drill/donuts.json` as tbl1
+ join
+ dfs.`/Users/brumsby/drill/moredonuts.json` as tbl2
+ on tbl1.id=tbl2.id;
+
+ +------------+------------+
+
+ | id | type |
+
+ +------------+------------+
+
+ | 0001 | donut |
+
+ +------------+------------+
+
+ 1 row selected (0.395 seconds)
+
+Equivalent USING syntax and joins in the WHERE clause are also supported.
+
+Standard aggregate functions work against JSON data. For example:
+
+ 0: jdbc:drill:zk=local> select type, avg(ppu) as ppu_sum from dfs.`/Users/brumsby/drill/donuts.json` group by type;
+
+ +------------+------------+
+
+ | type | ppu_sum |
+
+ +------------+------------+
+
+ | donut | 0.55 |
+
+ +------------+------------+
+
+ 1 row selected (0.216 seconds)
+
+ 0: jdbc:drill:zk=local> select type, sum(sales) as sum_by_type from dfs.`/Users/brumsby/drill/moredonuts.json` group by type;
+
+ +------------+-------------+
+
+ | type | sum_by_type |
+
+ +------------+-------------+
+
+ | donut | 1194 |
+
+ +------------+-------------+
+
+ 1 row selected (0.389 seconds)
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/query/query-complex/004-query3-sel-nest.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/query/query-complex/004-query3-sel-nest.md b/_docs/drill-docs/query/query-complex/004-query3-sel-nest.md
new file mode 100644
index 0000000..2d279d1
--- /dev/null
+++ b/_docs/drill-docs/query/query-complex/004-query3-sel-nest.md
@@ -0,0 +1,50 @@
+---
+title: "Query 3: Selecting Nested Data for a Column"
+parent: "Query Complex Data"
+---
+The following queries show how to access the nested data inside the parts of
+the record that are not flat (such as `topping`). To isolate and return nested
+data, use the `[n]` notation, where `n` is a number that points to a specific
+position in an array. Arrays use a 0-based index, so `topping[3]` points to
+the _fourth_ element in the array under `topping`, not the third.
+
+ 0: jdbc:drill:zk=local> select topping[3] as top from dfs.`/Users/brumsby/drill/donuts.json`;
+
+ +------------+
+
+ | top |
+
+ +------------+
+
+ | {"id":"5007","type":"Powdered Sugar"} |
+
+ +------------+
+
+ 1 row selected (0.137 seconds)
+
+Note that this query produces _one column for all of the data_ that is nested
+inside the `topping` segment of the file. The query as written does not unpack
+the `id` and `type` name/value pairs. Also note the use of an alias for the
+column name. (Without the alias, the default column name would be `EXPR$0`.)
+
+Some JSON files store arrays within arrays. If your data has this
+characteristic, you can probe into the inner array by using the following
+notation: `[n][n]`
+
+For example, assume that a segment of the JSON file looks like this:
+
+ ...
+ group:
+ [
+ [1,2,3],
+
+ [4,5,6],
+
+ [7,8,9]
+ ]
+ ...
+
+The following query would return `6` (the _third_ value of the _second_ inner
+array).
+
+`select group[1][2]`
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/query/query-complex/005-query4-sel-multiple.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/query/query-complex/005-query4-sel-multiple.md b/_docs/drill-docs/query/query-complex/005-query4-sel-multiple.md
new file mode 100644
index 0000000..832094e
--- /dev/null
+++ b/_docs/drill-docs/query/query-complex/005-query4-sel-multiple.md
@@ -0,0 +1,24 @@
+---
+title: "Query 4: Selecting Multiple Columns Within Nested Data"
+parent: "Query Complex Data"
+---
+The following query goes one step further to extract the JSON data, selecting
+specific `id` and `type` data values _as individual columns_ from inside the
+`topping` array. This query is similar to the previous query, but it returns
+the `id` and `type` values as separate columns.
+
+ 0: jdbc:drill:zk=local> select tbl.topping[3].id as record, tbl.topping[3].type as first_topping
+ from dfs.`/Users/brumsby/drill/donuts.json` as tbl;
+ +------------+---------------+
+ | record | first_topping |
+ +------------+---------------+
+ | 5007 | Powdered Sugar |
+ +------------+---------------+
+ 1 row selected (0.133 seconds)
+
+This query also introduces a typical requirement for queries against nested
+data: the use of a table alias (named tbl in this example). Without the table
+alias, the query would return an error because the parser would assume that id
+is a column inside a table named topping. As in all standard SQL queries,
+select tbl.col means that tbl is the name of an existing table (at least for
+the duration of the query) and col is a column that exists in that table.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/query/query-fs/001-query-json.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/query/query-fs/001-query-json.md b/_docs/drill-docs/query/query-fs/001-query-json.md
new file mode 100644
index 0000000..048903b
--- /dev/null
+++ b/_docs/drill-docs/query/query-fs/001-query-json.md
@@ -0,0 +1,41 @@
+---
+title: "Querying JSON Files"
+parent: "Querying a File System"
+---
+Your Drill installation includes a sample JSON file located in Drill's
+classpath. The sample JSON file, `employee.json`, contains fictitious employee
+data. Use SQL syntax to query the sample `JSON` file.
+
+To view the data in the `employee.json` file, submit the following SQL query
+to Drill:
+
+``0: jdbc:drill:zk=local> SELECT * FROM cp.`employee.json`;``
+
+The query returns the following results:
+
+**Example of partial output**
+
+ +-------------+------------+------------+------------+-------------+-----------+
+ | employee_id | full_name | first_name | last_name | position_id | position_ |
+ +-------------+------------+------------+------------+-------------+-----------+
+ | 1101 | Steve Eurich | Steve | Eurich | 16 | Store T |
+ | 1102 | Mary Pierson | Mary | Pierson | 16 | Store T |
+ | 1103 | Leo Jones | Leo | Jones | 16 | Store Tem |
+ | 1104 | Nancy Beatty | Nancy | Beatty | 16 | Store T |
+ | 1105 | Clara McNight | Clara | McNight | 16 | Store |
+ | 1106 | Marcella Isaacs | Marcella | Isaacs | 17 | Stor |
+ | 1107 | Charlotte Yonce | Charlotte | Yonce | 17 | Stor |
+ | 1108 | Benjamin Foster | Benjamin | Foster | 17 | Stor |
+ | 1109 | John Reed | John | Reed | 17 | Store Per |
+ | 1110 | Lynn Kwiatkowski | Lynn | Kwiatkowski | 17 | St |
+ | 1111 | Donald Vann | Donald | Vann | 17 | Store Pe |
+ | 1112 | William Smith | William | Smith | 17 | Store |
+ | 1113 | Amy Hensley | Amy | Hensley | 17 | Store Pe |
+ | 1114 | Judy Owens | Judy | Owens | 17 | Store Per |
+ | 1115 | Frederick Castillo | Frederick | Castillo | 17 | S |
+ | 1116 | Phil Munoz | Phil | Munoz | 17 | Store Per |
+ | 1117 | Lori Lightfoot | Lori | Lightfoot | 17 | Store |
+ ...
+ +-------------+------------+------------+------------+-------------+-----------+
+ 1,155 rows selected (0.762 seconds)
+ 0: jdbc:drill:zk=local>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/query/query-fs/002-query-parquet.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/query/query-fs/002-query-parquet.md b/_docs/drill-docs/query/query-fs/002-query-parquet.md
new file mode 100644
index 0000000..9b4e874
--- /dev/null
+++ b/_docs/drill-docs/query/query-fs/002-query-parquet.md
@@ -0,0 +1,99 @@
+---
+title: "Querying Parquet Files"
+parent: "Querying a File System"
+---
+Your Drill installation includes a `sample-date` directory with Parquet files
+that you can query. Use SQL syntax to query the `region.parquet` and
+`nation.parquet` files in the `sample-data` directory.
+
+**Note:** Your Drill installation location may differ from the examples used here. The examples assume that Drill was installed in embedded mode on your machine following the [Apache Drill in 10 Minutes ](https://cwiki.apache.org/confluence/display/DRILL/Apache+Drill+in+10+Minutes)tutorial. If you installed Drill in distributed mode, or your `sample-data` directory differs from the location used in the examples, make sure to change the `sample-data` directory to the correct location before you run the queries.
+
+#### Region File
+
+If you followed the Apache Drill in 10 Minutes instructions to install Drill
+in embedded mode, the path to the parquet file varies between operating
+systems.
+
+To view the data in the `region.parquet` file, issue the query appropriate for
+your operating system:
+
+ * Linux
+``SELECT * FROM dfs.`/opt/drill/apache-drill-0.4.0-incubating/sample-
+data/region.parquet`; ``
+
+ * Mac OS X
+``SELECT * FROM dfs.`/Users/max/drill/apache-drill-0.4.0-incubating/sample-
+data/region.parquet`;``
+
+ * Windows
+``SELECT * FROM dfs.`C:\drill\apache-drill-0.4.0-incubating\sample-
+data\region.parquet`;``
+
+The query returns the following results:
+
+ +------------+------------+
+ | EXPR$0 | EXPR$1 |
+ +------------+------------+
+ | AFRICA | lar deposits. blithely final packages cajole. regular waters ar |
+ | AMERICA | hs use ironic, even requests. s |
+ | ASIA | ges. thinly even pinto beans ca |
+ | EUROPE | ly final courts cajole furiously final excuse |
+ | MIDDLE EAST | uickly special accounts cajole carefully blithely close reques |
+ +------------+------------+
+ 5 rows selected (0.165 seconds)
+ 0: jdbc:drill:zk=local>
+
+#### Nation File
+
+If you followed the Apache Drill in 10 Minutes instructions to install Drill
+in embedded mode, the path to the parquet file varies between operating
+systems.
+
+To view the data in the `nation.parquet` file, issue the query appropriate for
+your operating system:
+
+ * Linux
+``SELECT * FROM dfs.`/opt/drill/apache-drill-0.4.0-incubating/sample-
+data/nation.parquet`; ``
+
+ * Mac OS X
+``SELECT * FROM dfs.`/Users/max/drill/apache-drill-0.4.0-incubating/sample-
+data/nation.parquet`;``
+
+ * Windows
+``SELECT * FROM dfs.`C:\drill\apache-drill-0.4.0-incubating\sample-
+data\nation.parquet`;``
+
+The query returns the following results:
+
+ +------------+------------+------------+------------+
+ | EXPR$0 | EXPR$1 | EXPR$2 | EXPR$3 |
+ +------------+------------+------------+------------+
+ | 0 | 0 | ALGERIA | haggle. carefully final deposits det |
+ | 1 | 1 | ARGENTINA | al foxes promise slyly according to t |
+ | 2 | 1 | BRAZIL | y alongside of the pending deposits. |
+ | 3 | 1 | CANADA | eas hang ironic, silent packages. sly |
+ | 4 | 4 | EGYPT | y above the carefully unusual theodol |
+ | 5 | 0 | ETHIOPIA | ven packages wake quickly. regu |
+ | 6 | 3 | FRANCE | refully final requests. regular, iron |
+ | 7 | 3 | GERMANY | l platelets. regular accounts x-ray: |
+ | 8 | 2 | INDIA | ss excuses cajole slyly across the pa |
+ | 9 | 2 | INDONESIA | slyly express asymptotes. regular de |
+ | 10 | 4 | IRAN | efully alongside of the slyly final d |
+ | 11 | 4 | IRAQ | nic deposits boost atop the quickly f |
+ | 12 | 2 | JAPAN | ously. final, express gifts cajole a |
+ | 13 | 4 | JORDAN | ic deposits are blithely about the ca |
+ | 14 | 0 | KENYA | pending excuses haggle furiously dep |
+ | 15 | 0 | MOROCCO | rns. blithely bold courts among the c |
+ | 16 | 0 | MOZAMBIQUE | s. ironic, unusual asymptotes wake bl |
+ | 17 | 1 | PERU | platelets. blithely pending dependenc |
+ | 18 | 2 | CHINA | c dependencies. furiously express not |
+ | 19 | 3 | ROMANIA | ular asymptotes are about the furious |
+ | 20 | 4 | SAUDI ARABIA | ts. silent requests haggle. closely |
+ | 21 | 2 | VIETNAM | hely enticingly express accounts. eve |
+ | 22 | 3 | RUSSIA | requests against the platelets use n |
+ | 23 | 3 | UNITED KINGDOM | eans boost carefully special requ |
+ | 24 | 1 | UNITED STATES | y final packages. slow foxes cajol |
+ +------------+------------+------------+------------+
+ 25 rows selected (2.401 seconds)
+ 0: jdbc:drill:zk=local>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/84b7b36d/_docs/drill-docs/query/query-fs/003-query-text.md
----------------------------------------------------------------------
diff --git a/_docs/drill-docs/query/query-fs/003-query-text.md b/_docs/drill-docs/query/query-fs/003-query-text.md
new file mode 100644
index 0000000..d7ffad5
--- /dev/null
+++ b/_docs/drill-docs/query/query-fs/003-query-text.md
@@ -0,0 +1,120 @@
+---
+title: "Querying Plain Text Files"
+parent: "Querying a File System"
+---
+You can use Drill to access both structured file types and plain text files
+(flat files). This section shows a few simple examples that work on flat
+files:
+
+ * CSV files (comma-separated values)
+ * TSV files (tab-separated values)
+ * PSV files (pipe-separated values)
+
+The examples here show CSV files, but queries against TSV and PSV files return
+equivalent results. However, make sure that your registered storage plugins
+recognize the appropriate file types and extensions. For example, the
+following configuration expects PSV files (files with a pipe delimiter) to
+have a `tbl` extension, not a `psv` extension. Drill returns a "file not
+found" error if references to files in queries do not match these conditions.
+
+ "formats": {
+ "psv": {
+ "type": "text",
+ "extensions": [
+ "tbl"
+ ],
+ "delimiter": "|"
+ }
+
+## SELECT * FROM a CSV File
+
+The first query selects rows from a `.csv` text file. The file contains seven
+records:
+
+ $ more plays.csv
+
+ 1599,As You Like It
+ 1601,Twelfth Night
+ 1594,Comedy of Errors
+ 1595,Romeo and Juliet
+ 1596,The Merchant of Venice
+ 1610,The Tempest
+ 1599,Hamlet
+
+Drill recognizes each row as an array of values and returns one column for
+each row.
+
+0: jdbc:drill:zk=local> select * from dfs.`/Users/brumsby/drill/plays.csv`;
+
+ +------------+
+ | columns |
+ +------------+
+ | ["1599","As You Like It"] |
+ | ["1601","Twelfth Night"] |
+ | ["1594","Comedy of Errors"] |
+ | ["1595","Romeo and Juliet"] |
+ | ["1596","The Merchant of Venice"] |
+ | ["1610","The Tempest"] |
+ | ["1599","Hamlet"] |
+ +------------+
+ 7 rows selected (0.089 seconds)
+
+## Columns[n] Syntax
+
+You can use the `COLUMNS[n]` syntax in the SELECT list to return these CSV
+rows in a more readable, column by column, format. (This syntax uses a zero-
+based index, so the first column is column `0`.)
+
+0: jdbc:drill:zk=local> select columns[0], columns[1]
+from dfs.`/Users/brumsby/drill/plays.csv`;
+
+ +------------+------------+
+ | EXPR$0 | EXPR$1 |
+ +------------+------------+
+ | 1599 | As You Like It |
+ | 1601 | Twelfth Night |
+ | 1594 | Comedy of Errors |
+ | 1595 | Romeo and Juliet |
+ | 1596 | The Merchant of Venice |
+ | 1610 | The Tempest |
+ | 1599 | Hamlet |
+ +------------+------------+
+ 7 rows selected (0.137 seconds)
+
+You can use aliases to return meaningful column names. Note that `YEAR` is a
+reserved word, so the `Year` alias must be enclosed by back ticks.
+
+ 0: jdbc:drill:zk=local> select columns[0] as `Year`, columns[1] as Play
+ from dfs.`/Users/brumsby/drill/plays.csv`;
+
+ +------------+------------+
+ | Year | Play |
+ +------------+------------+
+ | 1599 | As You Like It |
+ | 1601 | Twelfth Night |
+ | 1594 | Comedy of Errors |
+ | 1595 | Romeo and Juliet |
+ | 1596 | The Merchant of Venice |
+ | 1610 | The Tempest |
+ | 1599 | Hamlet |
+ +------------+------------+
+ 7 rows selected (0.113 seconds)
+
+You cannot refer to the aliases in subsequent clauses of the query. Use the
+original `columns[n]` syntax, as shown in the WHERE clause for the following
+example:
+
+0: jdbc:drill:zk=local> select columns[0] as `Year`, columns[1] as Play
+from dfs.`/Users/brumsby/drill/plays.csv` where columns[0]>1599;
+
+ +------------+------------+
+ | Year | Play |
+ +------------+------------+
+ | 1601 | Twelfth Night |
+ | 1610 | The Tempest |
+ +------------+------------+
+ 2 rows selected (0.201 seconds)
+
+Note that the restriction with the use of aliases applies to queries against
+all data sources.
+