You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2018/01/22 06:19:02 UTC
spark git commit: [MINOR][SQL] Fix wrong comments on
org.apache.spark.sql.parquet.row.attributes
Repository: spark
Updated Branches:
refs/heads/master 11daeb833 -> 8142a3b88
[MINOR][SQL] Fix wrong comments on org.apache.spark.sql.parquet.row.attributes
## What changes were proposed in this pull request?
This PR fixes the wrong comment on `org.apache.spark.sql.parquet.row.attributes`
which is useful for UDTs like Vector/Matrix. Please see [SPARK-22320](https://issues.apache.org/jira/browse/SPARK-22320) for the usage.
Originally, [SPARK-19411](https://github.com/apache/spark/commit/bf493686eb17006727b3ec81849b22f3df68fdef#diff-ee26d4c4be21e92e92a02e9f16dbc285L314) left this behind during removing optional column metadatas. In the same PR, the same comment was removed at line 310-311.
## How was this patch tested?
N/A (This is about comments).
Author: Dongjoon Hyun <do...@apache.org>
Closes #20346 from dongjoon-hyun/minor_comment_parquet.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8142a3b8
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/8142a3b8
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/8142a3b8
Branch: refs/heads/master
Commit: 8142a3b883a5fe6fc620a2c5b25b6bde4fda32e5
Parents: 11daeb8
Author: Dongjoon Hyun <do...@apache.org>
Authored: Mon Jan 22 15:18:57 2018 +0900
Committer: hyukjinkwon <gu...@gmail.com>
Committed: Mon Jan 22 15:18:57 2018 +0900
----------------------------------------------------------------------
.../sql/execution/datasources/parquet/ParquetFileFormat.scala | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/8142a3b8/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
----------------------------------------------------------------------
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
index 45bedf7..f53a97b 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
@@ -108,8 +108,7 @@ class ParquetFileFormat
ParquetOutputFormat.setWriteSupportClass(job, classOf[ParquetWriteSupport])
- // We want to clear this temporary metadata from saving into Parquet file.
- // This metadata is only useful for detecting optional columns when pushdowning filters.
+ // This metadata is useful for keeping UDTs like Vector/Matrix.
ParquetWriteSupport.setSchema(dataSchema, conf)
// Sets flags for `ParquetWriteSupport`, which converts Catalyst schema to Parquet
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org