You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by vi...@apache.org on 2021/07/12 16:18:23 UTC
[spark] branch master updated (5edbbd1 -> 094300f)
This is an automated email from the ASF dual-hosted git repository.
viirya pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 5edbbd1 [SPARK-36056][SQL] Combine readBatch and readIntegers in VectorizedRleValuesReader
add 094300f [SPARK-35861][SS] Introduce "prefix match scan" feature on state store
No new revisions were added by this update.
Summary of changes:
.../streaming/FlatMapGroupsWithStateExec.scala | 5 +-
.../state/FlatMapGroupsWithStateExecHelper.scala | 2 +-
.../streaming/state/HDFSBackedStateStoreMap.scala | 170 +++++++++++
.../state/HDFSBackedStateStoreProvider.scala | 94 +++---
.../sql/execution/streaming/state/RocksDB.scala | 35 +++
.../streaming/state/RocksDBStateEncoder.scala | 252 +++++++++++++++
.../state/RocksDBStateStoreProvider.scala | 117 ++-----
.../sql/execution/streaming/state/StateStore.scala | 54 ++--
.../execution/streaming/state/StateStoreRDD.scala | 8 +-
.../state/StreamingAggregationStateManager.scala | 2 +-
.../state/SymmetricHashJoinStateManager.scala | 6 +-
.../sql/execution/streaming/state/package.scala | 12 +-
.../execution/streaming/statefulOperators.scala | 8 +-
.../sql/execution/streaming/streamingLimits.scala | 2 +-
.../streaming/state/MemoryStateStore.scala | 4 +
.../streaming/state/RocksDBStateStoreSuite.scala | 35 ++-
.../streaming/state/StateStoreRDDSuite.scala | 128 ++++----
.../streaming/state/StateStoreSuite.scala | 338 ++++++++++++---------
.../apache/spark/sql/streaming/StreamSuite.scala | 2 +-
19 files changed, 861 insertions(+), 413 deletions(-)
create mode 100644 sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
create mode 100644 sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateEncoder.scala
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org