You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iceberg.apache.org by bl...@apache.org on 2021/01/21 03:55:42 UTC
[iceberg] branch master updated: Spark: Add Spark extensions to
iceberg-spark3-runtime (#2127)
This is an automated email from the ASF dual-hosted git repository.
blue pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/iceberg.git
The following commit(s) were added to refs/heads/master by this push:
new b7dde31 Spark: Add Spark extensions to iceberg-spark3-runtime (#2127)
b7dde31 is described below
commit b7dde314edab97d5118063f8c17a65037a828006
Author: Jack Ye <yz...@amazon.com>
AuthorDate: Wed Jan 20 19:55:27 2021 -0800
Spark: Add Spark extensions to iceberg-spark3-runtime (#2127)
---
build.gradle | 1 +
site/docs/spark.md | 2 ++
spark3-runtime/LICENSE | 22 ++++++++++++++++++++++
3 files changed, 25 insertions(+)
diff --git a/build.gradle b/build.gradle
index 58337a1..82ed2fc 100644
--- a/build.gradle
+++ b/build.gradle
@@ -1015,6 +1015,7 @@ project(':iceberg-spark3-runtime') {
dependencies {
compile project(':iceberg-spark3')
+ compile project(':iceberg-spark3-extensions')
compile project(':iceberg-aws')
compile 'org.apache.spark:spark-hive_2.11'
compile(project(':iceberg-nessie')) {
diff --git a/site/docs/spark.md b/site/docs/spark.md
index c20338b..56f140e 100644
--- a/site/docs/spark.md
+++ b/site/docs/spark.md
@@ -35,6 +35,8 @@ Iceberg uses Apache Spark's DataSourceV2 API for data source and catalog impleme
| [DataFrame CTAS and RTAS](#creating-tables) | ✔️ | | |
| [Metadata tables](#inspecting-tables) | ✔️ | ✔️ | |
+To enable Iceberg SQL extensions, set Spark configuration `spark.sql.extensions` as `org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions`.
+
## Configuring catalogs
Spark 3.0 adds an API to plug in table catalogs that are used to load, create, and manage Iceberg tables. Spark catalogs are configured by setting [Spark properties](./configuration.md#catalogs) under `spark.sql.catalog`.
diff --git a/spark3-runtime/LICENSE b/spark3-runtime/LICENSE
index aaf5c4f..32f66f2 100644
--- a/spark3-runtime/LICENSE
+++ b/spark3-runtime/LICENSE
@@ -587,3 +587,25 @@ Copyright: 2020 Dremio Corporation.
Home page: https://projectnessie.org/
License: http://www.apache.org/licenses/LICENSE-2.0
+--------------------------------------------------------------------------------
+
+This product includes code from Apache Spark.
+
+* vectorized reading of definition levels in BaseVectorizedParquetValuesReader.java
+* portions of the extensions parser
+* casting logic in AssignmentAlignmentSupport
+* implementation of SetAccumulator.
+
+Copyright: 2011-2018 The Apache Software Foundation
+Home page: https://spark.apache.org/
+License: https://www.apache.org/licenses/LICENSE-2.0
+
+--------------------------------------------------------------------------------
+
+This product includes code from Delta Lake.
+
+* AssignmentAlignmentSupport is an independent development but UpdateExpressionsSupport in Delta was used as a reference.
+
+Copyright: 2020 The Delta Lake Project Authors.
+Home page: https://delta.io/
+License: https://www.apache.org/licenses/LICENSE-2.0