You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ka...@apache.org on 2021/07/16 11:40:10 UTC
[spark] branch master updated (c1b3f86 -> f2bf8b0)
This is an automated email from the ASF dual-hosted git repository.
kabhwan pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from c1b3f86 [SPARK-35710][SQL] Support DPP + AQE when there is no reused broadcast exchange
add f2bf8b0 [SPARK-34893][SS] Support session window natively
No new revisions were added by this update.
Summary of changes:
python/pyspark/sql/functions.py | 35 ++
python/pyspark/sql/functions.pyi | 1 +
.../spark/sql/catalyst/analysis/Analyzer.scala | 86 +++-
.../sql/catalyst/analysis/FunctionRegistry.scala | 1 +
.../sql/catalyst/expressions/SessionWindow.scala | 77 ++++
.../sql/catalyst/expressions/TimeWindow.scala | 4 +-
.../spark/sql/errors/QueryCompilationErrors.scala | 5 +-
.../org/apache/spark/sql/internal/SQLConf.scala | 24 ++
.../spark/sql/execution/SparkStrategies.scala | 33 +-
.../spark/sql/execution/aggregate/AggUtils.scala | 190 ++++++++-
.../aggregate/UpdatingSessionsIterator.scala | 13 +-
.../execution/python/AggregateInPandasExec.scala | 49 ++-
.../execution/streaming/IncrementalExecution.scala | 20 +
.../state/StreamingSessionWindowStateManager.scala | 17 +-
.../execution/streaming/statefulOperators.scala | 289 +++++++++++++
.../scala/org/apache/spark/sql/functions.scala | 30 ++
.../sql-functions/sql-expression-schema.md | 9 +-
.../spark/sql/DataFrameSessionWindowingSuite.scala | 290 +++++++++++++
.../spark/sql/DataFrameTimeWindowingSuite.scala | 2 +-
.../scala/org/apache/spark/sql/SQLQuerySuite.scala | 3 +-
.../streaming/UpdatingSessionsIteratorSuite.scala | 8 +-
.../sql/expressions/ExpressionInfoSuite.scala | 1 +
.../streaming/StreamingSessionWindowSuite.scala | 460 +++++++++++++++++++++
23 files changed, 1608 insertions(+), 39 deletions(-)
create mode 100644 sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SessionWindow.scala
create mode 100644 sql/core/src/test/scala/org/apache/spark/sql/DataFrameSessionWindowingSuite.scala
create mode 100644 sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingSessionWindowSuite.scala
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org