You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by li...@apache.org on 2016/12/30 22:46:34 UTC
spark git commit: [SPARK-19016][SQL][DOC] Document scalable partition
handling
Repository: spark
Updated Branches:
refs/heads/master b85e29437 -> 871f6114a
[SPARK-19016][SQL][DOC] Document scalable partition handling
## What changes were proposed in this pull request?
This PR documents the scalable partition handling feature in the body of the programming guide.
Before this PR, we only mention it in the migration guide. It's not super clear that external datasource tables require an extra `MSCK REPAIR TABLE` command is to have per-partition information persisted since 2.1.
## How was this patch tested?
N/A.
Author: Cheng Lian <li...@databricks.com>
Closes #16424 from liancheng/scalable-partition-handling-doc.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/871f6114
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/871f6114
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/871f6114
Branch: refs/heads/master
Commit: 871f6114ac0075a1b45eda8701113fa20d647de9
Parents: b85e294
Author: Cheng Lian <li...@databricks.com>
Authored: Fri Dec 30 14:46:30 2016 -0800
Committer: Cheng Lian <li...@databricks.com>
Committed: Fri Dec 30 14:46:30 2016 -0800
----------------------------------------------------------------------
docs/sql-programming-guide.md | 15 +++++++++++----
1 file changed, 11 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/871f6114/docs/sql-programming-guide.md
----------------------------------------------------------------------
diff --git a/docs/sql-programming-guide.md b/docs/sql-programming-guide.md
index 6287e2b..4cd21ae 100644
--- a/docs/sql-programming-guide.md
+++ b/docs/sql-programming-guide.md
@@ -515,7 +515,7 @@ new data.
### Saving to Persistent Tables
`DataFrames` can also be saved as persistent tables into Hive metastore using the `saveAsTable`
-command. Notice existing Hive deployment is not necessary to use this feature. Spark will create a
+command. Notice that an existing Hive deployment is not necessary to use this feature. Spark will create a
default local Hive metastore (using Derby) for you. Unlike the `createOrReplaceTempView` command,
`saveAsTable` will materialize the contents of the DataFrame and create a pointer to the data in the
Hive metastore. Persistent tables will still exist even after your Spark program has restarted, as
@@ -526,11 +526,18 @@ By default `saveAsTable` will create a "managed table", meaning that the locatio
be controlled by the metastore. Managed tables will also have their data deleted automatically
when a table is dropped.
-Currently, `saveAsTable` does not expose an API supporting the creation of an "External table" from a `DataFrame`,
-however, this functionality can be achieved by providing a `path` option to the `DataFrameWriter` with `path` as the key
-and location of the external table as its value (String) when saving the table with `saveAsTable`. When an External table
+Currently, `saveAsTable` does not expose an API supporting the creation of an "external table" from a `DataFrame`.
+However, this functionality can be achieved by providing a `path` option to the `DataFrameWriter` with `path` as the key
+and location of the external table as its value (a string) when saving the table with `saveAsTable`. When an External table
is dropped only its metadata is removed.
+Starting from Spark 2.1, persistent datasource tables have per-partition metadata stored in the Hive metastore. This brings several benefits:
+
+- Since the metastore can return only necessary partitions for a query, discovering all the partitions on the first query to the table is no longer needed.
+- Hive DDLs such as `ALTER TABLE PARTITION ... SET LOCATION` are now available for tables created with the Datasource API.
+
+Note that partition information is not gathered by default when creating external datasource tables (those with a `path` option). To sync the partition information in the metastore, you can invoke `MSCK REPAIR TABLE`.
+
## Parquet Files
[Parquet](http://parquet.io) is a columnar format that is supported by many other data processing systems.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org