You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by we...@apache.org on 2020/06/09 04:56:37 UTC
[spark] branch branch-3.0 updated:
[SPARK-28199][SPARK-28199][SS][FOLLOWUP] Mention the change of into the SS
migration guide
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new f0a27b5 [SPARK-28199][SPARK-28199][SS][FOLLOWUP] Mention the change of into the SS migration guide
f0a27b5 is described below
commit f0a27b5488608bf4a4c100c4f26ffa3f7cd6b452
Author: Jungtaek Lim (HeartSaVioR) <ka...@gmail.com>
AuthorDate: Tue Jun 9 04:52:48 2020 +0000
[SPARK-28199][SPARK-28199][SS][FOLLOWUP] Mention the change of into the SS migration guide
### What changes were proposed in this pull request?
SPARK-28199 (#24996) made the trigger related public API to be exposed only from static methods of Trigger class. This is backward incompatible change, so some users may experience compilation error after upgrading to Spark 3.0.0.
While we plan to mention the change into release note, it's good to mention the change to the migration guide doc as well, since the purpose of the doc is to collect the major changes/incompatibilities between versions and end users would refer the doc.
### Why are the changes needed?
SPARK-28199 is technically backward incompatible change and we should kindly guide the change.
### Does this PR introduce _any_ user-facing change?
Doc change.
### How was this patch tested?
N/A, as it's just a doc change.
Closes #28763 from HeartSaVioR/SPARK-28199-FOLLOWUP-doc.
Authored-by: Jungtaek Lim (HeartSaVioR) <ka...@gmail.com>
Signed-off-by: Wenchen Fan <we...@databricks.com>
(cherry picked from commit 8305b77796ad45090e9d430e2be59e467fc173d6)
Signed-off-by: Wenchen Fan <we...@databricks.com>
---
docs/ss-migration-guide.md | 2 ++
1 file changed, 2 insertions(+)
diff --git a/docs/ss-migration-guide.md b/docs/ss-migration-guide.md
index 963ef07..002058b 100644
--- a/docs/ss-migration-guide.md
+++ b/docs/ss-migration-guide.md
@@ -31,3 +31,5 @@ Please refer [Migration Guide: SQL, Datasets and DataFrame](sql-migration-guide.
- In Spark 3.0, Structured Streaming forces the source schema into nullable when file-based datasources such as text, json, csv, parquet and orc are used via `spark.readStream(...)`. Previously, it respected the nullability in source schema; however, it caused issues tricky to debug with NPE. To restore the previous behavior, set `spark.sql.streaming.fileSource.schema.forceNullable` to `false`.
- Spark 3.0 fixes the correctness issue on Stream-stream outer join, which changes the schema of state. (See [SPARK-26154](https://issues.apache.org/jira/browse/SPARK-26154) for more details). If you start your query from checkpoint constructed from Spark 2.x which uses stream-stream outer join, Spark 3.0 fails the query. To recalculate outputs, discard the checkpoint and replay previous inputs.
+
+- In Spark 3.0, the deprecated class `org.apache.spark.sql.streaming.ProcessingTime` has been removed. Use `org.apache.spark.sql.streaming.Trigger.ProcessingTime` instead. Likewise, `org.apache.spark.sql.execution.streaming.continuous.ContinuousTrigger` has been removed in favor of `Trigger.Continuous`, and `org.apache.spark.sql.execution.streaming.OneTimeTrigger` has been hidden in favor of `Trigger.Once`.
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org