You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ka...@apache.org on 2025/01/17 01:03:06 UTC

(spark) branch master updated (11d758f48141 -> 41aa0b5ebf44)

This is an automated email from the ASF dual-hosted git repository.

kabhwan pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


    from 11d758f48141 [SPARK-50852][INFRA] Add Java 21 Daily GitHub Action job for `branch-4.0`
     add 41aa0b5ebf44 [SPARK-50714][SQL][SS] Enable schema evolution for TransformWithState when Avro encoding is used

No new revisions were added by this update.

Summary of changes:
 .../src/main/resources/error/error-conditions.json |   23 +
 .../pandas/test_pandas_transform_with_state.py     |  353 +++++-
 .../org/apache/spark/sql/internal/SQLConf.scala    |   22 +
 .../apache/spark/sql/avro/SchemaConverters.scala   |  111 ++
 .../datasources/v2/state/StateDataSource.scala     |   32 +-
 .../v2/state/StatePartitionReader.scala            |   27 +-
 .../datasources/v2/state/StateScanBuilder.scala    |   22 +-
 .../datasources/v2/state/StateTable.scala          |   14 +-
 .../v2/state/metadata/StateMetadataSource.scala    |    6 +-
 .../python/TransformWithStateInPandasExec.scala    |   17 +-
 .../streaming/FlatMapGroupsWithStateExec.scala     |    5 +-
 .../execution/streaming/IncrementalExecution.scala |   74 +-
 .../execution/streaming/ListStateMetricsImpl.scala |   14 +-
 .../execution/streaming/MicroBatchExecution.scala  |    9 +-
 .../StateStoreColumnFamilySchemaUtils.scala        |  122 ++-
 .../streaming/StatefulProcessorHandleImpl.scala    |  159 ++-
 .../streaming/StreamingSymmetricHashJoinExec.scala |    8 +-
 .../sql/execution/streaming/TimerStateImpl.scala   |   17 +-
 .../streaming/TransformWithStateExec.scala         |   25 +-
 .../TransformWithStateVariableUtils.scala          |   27 +-
 .../state/HDFSBackedStateStoreProvider.scala       |    3 +-
 .../streaming/state/OperatorStateMetadata.scala    |   20 +-
 .../streaming/state/RocksDBStateEncoder.scala      |  297 ++++-
 .../state/RocksDBStateStoreProvider.scala          |   72 +-
 .../execution/streaming/state/SchemaHelper.scala   |   10 +
 .../state/StateSchemaCompatibilityChecker.scala    |  222 ++--
 .../sql/execution/streaming/state/StateStore.scala |   24 +-
 .../streaming/state/StateStoreErrors.scala         |   56 +
 .../execution/streaming/state/StateStoreRDD.scala  |    4 +
 .../state/SymmetricHashJoinStateManager.scala      |    4 +-
 .../sql/execution/streaming/state/package.scala    |    2 +
 .../execution/streaming/statefulOperators.scala    |   54 +-
 .../sql/execution/streaming/streamingLimits.scala  |    2 +-
 .../StateDataSourceTransformWithStateSuite.scala   |    2 +-
 ...ngSortWithSessionWindowStateIteratorSuite.scala |    2 +-
 .../state/OperatorStateMetadataSuite.scala         |    5 +-
 .../RocksDBStateStoreCheckpointFormatV2Suite.scala |    3 +-
 .../streaming/state/RocksDBStateStoreSuite.scala   |  575 +++++++++-
 .../execution/streaming/state/RocksDBSuite.scala   |   81 +-
 .../StateSchemaCompatibilityCheckerSuite.scala     |   32 +-
 .../streaming/state/StateStoreSuite.scala          |   40 +-
 .../StreamingSessionWindowStateManagerSuite.scala  |    2 +-
 .../streaming/state/ValueStateSuite.scala          |    3 +-
 .../apache/spark/sql/streaming/StreamSuite.scala   |    5 +-
 .../streaming/TransformWithListStateSuite.scala    |  140 +++
 .../sql/streaming/TransformWithMapStateSuite.scala |  103 +-
 .../TransformWithStateChainingSuite.scala          |    4 +-
 .../TransformWithStateInitialStateSuite.scala      |    4 +-
 .../sql/streaming/TransformWithStateSuite.scala    | 1140 ++++++++++++++++++--
 .../sql/streaming/TransformWithStateTTLTest.scala  |    4 +-
 .../TransformWithValueStateTTLSuite.scala          |  246 ++++-
 51 files changed, 3810 insertions(+), 438 deletions(-)


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org