You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2019/04/08 00:15:01 UTC
[spark] branch master updated: [SPARK-27253][SQL][FOLLOW-UP] Add a
note about parent-session configuration priority in migration guide
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 18b36ee [SPARK-27253][SQL][FOLLOW-UP] Add a note about parent-session configuration priority in migration guide
18b36ee is described below
commit 18b36ee5ba7b7b7a88d641b5872c9d1aa87460e0
Author: HyukjinKwon <gu...@apache.org>
AuthorDate: Mon Apr 8 09:14:40 2019 +0900
[SPARK-27253][SQL][FOLLOW-UP] Add a note about parent-session configuration priority in migration guide
## What changes were proposed in this pull request?
This is a followup of https://github.com/apache/spark/pull/24189. It adds a note about parent-session configuration priority.
## How was this patch tested?
Manually built the site and checked.
Closes #24279 from HyukjinKwon/SPARK-27253.
Lead-authored-by: HyukjinKwon <gu...@apache.org>
Co-authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: HyukjinKwon <gu...@apache.org>
---
docs/sql-migration-guide-upgrade.md | 2 ++
1 file changed, 2 insertions(+)
diff --git a/docs/sql-migration-guide-upgrade.md b/docs/sql-migration-guide-upgrade.md
index fef8012..c3837f6 100644
--- a/docs/sql-migration-guide-upgrade.md
+++ b/docs/sql-migration-guide-upgrade.md
@@ -124,6 +124,8 @@ license: |
- Since Spark 3.0, `TIMESTAMP` literals are converted to strings using the SQL config `spark.sql.session.timeZone`, and `DATE` literals are formatted using the UTC time zone. In Spark version 2.4 and earlier, both conversions use the default time zone of the Java virtual machine.
+ - In Spark version 2.4, when a spark session is created via `cloneSession()`, the newly created spark session inherits its configuration from its parent `SparkContext` even though the same configuration may exist with a different value in its parent spark session. Since Spark 3.0, the configurations of a parent `SparkSession` have a higher precedence over the parent `SparkContext`.
+
## Upgrading From Spark SQL 2.3 to 2.4
- In Spark version 2.3 and earlier, the second parameter to array_contains function is implicitly promoted to the element type of first array type parameter. This type promotion can be lossy and may cause `array_contains` function to return wrong result. This problem has been addressed in 2.4 by employing a safer type promotion mechanism. This can cause some change in behavior and are illustrated in the table below.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org