You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2021/01/16 01:03:41 UTC
[spark] branch branch-3.1 updated: [SPARK-34060][SQL][FOLLOWUP]
Preserve serializability of canonicalized CatalogTable
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 63b8a66 [SPARK-34060][SQL][FOLLOWUP] Preserve serializability of canonicalized CatalogTable
63b8a66 is described below
commit 63b8a6679ee27560b6e7d3c3c8c670f38c58f76b
Author: Max Gekk <ma...@gmail.com>
AuthorDate: Fri Jan 15 17:02:29 2021 -0800
[SPARK-34060][SQL][FOLLOWUP] Preserve serializability of canonicalized CatalogTable
### What changes were proposed in this pull request?
Replace `toMap` by `map(identity).toMap` while getting canonicalized representation of `CatalogTable`. `CatalogTable` became not serializable after https://github.com/apache/spark/pull/31112 due to usage of `filterKeys`. The workaround was taken from https://github.com/scala/bug/issues/7005.
### Why are the changes needed?
This prevents the errors like:
```
[info] org.apache.spark.SparkException: Job aborted due to stage failure: Task not serializable: java.io.NotSerializableException: scala.collection.immutable.MapLike$$anon$1
[info] Cause: java.io.NotSerializableException: scala.collection.immutable.MapLike$$anon$1
```
### Does this PR introduce _any_ user-facing change?
Should not.
### How was this patch tested?
By running the test suite affected by https://github.com/apache/spark/pull/31112:
```
$ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *AlterTableDropPartitionSuite"
```
Closes #31197 from MaxGekk/fix-caching-hive-table-2-followup.
Authored-by: Max Gekk <ma...@gmail.com>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
(cherry picked from commit c3d81fbe79014f693cf93c02e31af401727761d7)
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
.../main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
index 7c519ce..d6a9148 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
@@ -491,7 +491,10 @@ object CatalogTable {
table.copy(
createTime = 0L,
lastAccessTime = 0L,
- properties = table.properties.filterKeys(!nondeterministicProps.contains(_)).toMap,
+ properties = table.properties
+ .filterKeys(!nondeterministicProps.contains(_))
+ .map(identity)
+ .toMap,
stats = None,
ignoredProperties = Map.empty
)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org