You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ka...@apache.org on 2025/01/17 01:03:06 UTC
(spark) branch master updated (11d758f48141 -> 41aa0b5ebf44)
This is an automated email from the ASF dual-hosted git repository.
kabhwan pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
from 11d758f48141 [SPARK-50852][INFRA] Add Java 21 Daily GitHub Action job for `branch-4.0`
add 41aa0b5ebf44 [SPARK-50714][SQL][SS] Enable schema evolution for TransformWithState when Avro encoding is used
No new revisions were added by this update.
Summary of changes:
.../src/main/resources/error/error-conditions.json | 23 +
.../pandas/test_pandas_transform_with_state.py | 353 +++++-
.../org/apache/spark/sql/internal/SQLConf.scala | 22 +
.../apache/spark/sql/avro/SchemaConverters.scala | 111 ++
.../datasources/v2/state/StateDataSource.scala | 32 +-
.../v2/state/StatePartitionReader.scala | 27 +-
.../datasources/v2/state/StateScanBuilder.scala | 22 +-
.../datasources/v2/state/StateTable.scala | 14 +-
.../v2/state/metadata/StateMetadataSource.scala | 6 +-
.../python/TransformWithStateInPandasExec.scala | 17 +-
.../streaming/FlatMapGroupsWithStateExec.scala | 5 +-
.../execution/streaming/IncrementalExecution.scala | 74 +-
.../execution/streaming/ListStateMetricsImpl.scala | 14 +-
.../execution/streaming/MicroBatchExecution.scala | 9 +-
.../StateStoreColumnFamilySchemaUtils.scala | 122 ++-
.../streaming/StatefulProcessorHandleImpl.scala | 159 ++-
.../streaming/StreamingSymmetricHashJoinExec.scala | 8 +-
.../sql/execution/streaming/TimerStateImpl.scala | 17 +-
.../streaming/TransformWithStateExec.scala | 25 +-
.../TransformWithStateVariableUtils.scala | 27 +-
.../state/HDFSBackedStateStoreProvider.scala | 3 +-
.../streaming/state/OperatorStateMetadata.scala | 20 +-
.../streaming/state/RocksDBStateEncoder.scala | 297 ++++-
.../state/RocksDBStateStoreProvider.scala | 72 +-
.../execution/streaming/state/SchemaHelper.scala | 10 +
.../state/StateSchemaCompatibilityChecker.scala | 222 ++--
.../sql/execution/streaming/state/StateStore.scala | 24 +-
.../streaming/state/StateStoreErrors.scala | 56 +
.../execution/streaming/state/StateStoreRDD.scala | 4 +
.../state/SymmetricHashJoinStateManager.scala | 4 +-
.../sql/execution/streaming/state/package.scala | 2 +
.../execution/streaming/statefulOperators.scala | 54 +-
.../sql/execution/streaming/streamingLimits.scala | 2 +-
.../StateDataSourceTransformWithStateSuite.scala | 2 +-
...ngSortWithSessionWindowStateIteratorSuite.scala | 2 +-
.../state/OperatorStateMetadataSuite.scala | 5 +-
.../RocksDBStateStoreCheckpointFormatV2Suite.scala | 3 +-
.../streaming/state/RocksDBStateStoreSuite.scala | 575 +++++++++-
.../execution/streaming/state/RocksDBSuite.scala | 81 +-
.../StateSchemaCompatibilityCheckerSuite.scala | 32 +-
.../streaming/state/StateStoreSuite.scala | 40 +-
.../StreamingSessionWindowStateManagerSuite.scala | 2 +-
.../streaming/state/ValueStateSuite.scala | 3 +-
.../apache/spark/sql/streaming/StreamSuite.scala | 5 +-
.../streaming/TransformWithListStateSuite.scala | 140 +++
.../sql/streaming/TransformWithMapStateSuite.scala | 103 +-
.../TransformWithStateChainingSuite.scala | 4 +-
.../TransformWithStateInitialStateSuite.scala | 4 +-
.../sql/streaming/TransformWithStateSuite.scala | 1140 ++++++++++++++++++--
.../sql/streaming/TransformWithStateTTLTest.scala | 4 +-
.../TransformWithValueStateTTLSuite.scala | 246 ++++-
51 files changed, 3810 insertions(+), 438 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org