You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ya...@apache.org on 2026/02/04 12:48:29 UTC

(spark) branch feature/crossjoin-array-contains-benchmark updated (9e3d4516673e -> bd84f87b7e3a)

This is an automated email from the ASF dual-hosted git repository.

yao pushed a change to branch feature/crossjoin-array-contains-benchmark
in repository https://gitbox.apache.org/repos/asf/spark.git


 discard 9e3d4516673e [SPARK-XXXX][SQL] Add CrossJoinArrayContainsToInnerJoin optimizer rule
    omit 3fa07bb841fd [SPARK-55305][SQL][TESTS] Use `ParquetFooterReader.readFooter` uniformly in test code to read the footer
    omit d3dc60299eab [SPARK-55307][K8S][INFRA] Update `setup-minikube` to v0.0.21
    omit 36fbc71d6e8b [SPARK-55297][PYTHON][PS] Restore timedelta dtype based on the original dtype
    omit f74d7be0a735 [SPARK-55093][CORE] Handle TaskRunner construction failures in launchTask
    omit b7c00eaa903c [SPARK-55293][PS][TESTS] Avoid using old offset aliases
    omit 0aeb6f9f4937 [SPARK-55286][INFRA] Add test summary to GitHub Actions for better failure visibility
    omit 2d41e46338ec [SPARK-55283][PYTHON][PS][TESTS] Add a new argument ignore_null to assert_eq
    omit fbc3471e8748 [SPARK-55285][SQL][PYTHON] Fix the initialization of `PythonArrowInput`
    omit 9b6393d8c030 [SPARK-55284][PYTHON][TEST] Move mypy-data related configs to the script
    omit a3e3da960cc2 [SPARK-55176][PYTHON] Extract `arrow_to_pandas` converter into ArrowArrayToPandasConversion
    omit 545d9e73a4a9 [SPARK-55287][INFRA] Consolidate steps in `lint`
    omit 1ce11023d758 [SPARK-55105][SS] Add Integration Test for Join Operator
    omit c5cb2435400e [SPARK-55123][SS] Add SequentialUnionOffset for tracking sequential source processing
    omit 2d940919c0a1 [SPARK-55193][CORE][BUILD] Use `CompressionHandler` as a replacement for the deprecated `GzipHandler` in `JettyUtils`
    omit 866a6e8af0a0 [SPARK-55290][NETWORK][TESTS] Fix testReloadMissingTrustStore cross-device link error with JDK 21
    omit 65a6a55630f7 [SPARK-55246][SS] Add Test for Pyspark TWS and TWSInPandas and Fix StatePartitionAllColumnFamiliesWriter Bug
    omit fbb4019d70d1 [SPARK-55279][SQL] Add `sketch_funcs` group for DataSketches SQL functions
    omit 76f6c784a0c4 [SPARK-55239][CONNECT][YARN] Allow to launch SparkConnectServer in YARN cluster mode
    omit 04b821c69e85 [SPARK-55256][SQL] Support IGNORE NULLS / RESPECT NULLS for array_agg and collect_list
    omit 1da0e5357ad9 [SPARK-55133][CONNECT] Fix race condition in IsolatedSessionState lifecycle management
    omit 44db44c5ea8f [SPARK-49110][SQL] Simplify SubqueryAlias.metadataOutput to always propagate metadata columns
    omit 4e0537283336 [SPARK-55277][SQL] Add `protobuf_funcs` group for Protobuf SQL functions
    omit 4a58b849484e Revert "[SPARK-55277][SQL] Add `protobuf_funcs` group for Protobuf SQL functions"
    omit efdb49239d62 [SPARK-55277][SQL] Add `protobuf_funcs` group for Protobuf SQL functions
    omit 8e657692986b [SPARK-55236][CORE] Address unexpected exception in some CoarseGrainedExecutorBackendSuite test cases
    omit 4729b9917107 [SPARK-55031][SQL] Add vector avg/sum aggregation function expressions
    omit 75cd9be439b4 [SPARK-55237][SQL] Suppress annoying messages when looking up nonexistent DBs
    omit 4344f3fce786 [SPARK-55273][SQL] Replace `ParquetFileReader.open().getFooter()` with `readFooter()` to avoid unnecessary operations in `ParquetFooterReader`
    omit b3cbff35877c [SPARK-55272][BUILD] Upgrade SBT to 1.12.1
    omit 86f8b3fe6375 [SPARK-55276][BUILD] Upgrade `scala-maven-plugin` to 4.9.9
    omit ea26cac89a07 [SPARK-55266][INFRA] Add pre-commit hooks for format/lint
    omit 66255917ca59 [SPARK-55281][PYTHON] Add ipykernel and IPython to mypy optional package list
    omit 9254e8912aec [SPARK-55263][PYTHON][INFRA] Upgrade Python linter from 3.11 to 3.12 in CI
    omit 23afba2e8e07 [SPARK-55282][PYTHON][CONNECT] Avoid using worker_util in the Driver-side
    omit a1577253d88d [SPARK-55011][DOCS] CURSORs docs
    omit 1aadbc439c05 [SPARK-54887] Add previously removed legacy error class back in
    omit fe9e5c092b7c [SPARK-55262][GEO][SQL] Block Geo types in all file based data sources except Parquet
    omit c32aee117b60 [SPARK-55243][CONNECT] Allow setting binary headers via the -bin suffix in the Scala Connect client
    omit 5c320f4854d9 [SPARK-55259][GEO][SQL] Implement Parquet schema conversion for Geo types
    omit 6ffc45a5bd38 [SPARK-55114][PYTHON][TESTS][FOLLOW-UP] Update the result format to be more friendly to markdown
    omit 7e4a040e07c9 [SPARK-55064][SQL][CORE] Support query level indeterminate shuffle retry
    omit dd2c62f790a8 [MINOR][PYTHON][TESTS] Consolidate DataStreamReader.name() tests into test_streaming.py
    omit 3f6d2317b69b [SPARK-47996][PS] support cross merge in pandas API
    omit 1268c15a4339 [SPARK-55198][SQL] spark-sql should skip comment line with leading whitespaces
    omit 3707f94f41ac [SPARK-54943][PYTHON][TESTS][FOLLOW-UP] Disable `test_pyarrow_array_cast` for now
    omit 3e07518ea2ad Revert "[SPARK-54943][PYTHON][TESTS][FOLLOW-UP] Mistake Commit"
    omit fe73cecbf4a9 [SPARK-54812][SQL] Make executable commands not execute on resultDf.cache()
    omit 44f61d5117fc [SPARK-54943][PYTHON][TESTS][FOLLOW-UP] Disable `test_pyarrow_array_cast`
    omit 38e51eb5e8d4 [SPARK-41398][SQL][FOLLOWUP] Update runtime filtering javadoc to reflect relaxed partition constraints
    omit 3d1188a06bf9 [SPARK-54521][SQL] Add to and from WKB for Geometry type
    omit 75479fdecf82 [SPARK-55150][CONNECT][SQL] Improve observation error handling
    omit fb2a308aca1e [SPARK-54830][SPARK-48037][TESTS][FOLLOWUP] Disable shuffle checksum for the test case of to avoid memory issues
    omit 776e2b93cb15 [SPARK-55244][PYTHON][PS] Use np.nan as default value for pandas string types
    omit dc7b8c1968f8 [SPARK-55225][PYTHON][PS] Restore to the original dtype for Datetime
    omit 0cba1c0a81a8 [SPARK-55233][BUILD] Upgrade ASM to 9.9.1
    omit 3de3d81d01f6 [SPARK-55155][SQL] Support foldable expressions in `SET CATALOG` statement
    omit 7783f46f58ff [SPARK-54759][SQL] SQL scripting Cursor support
    omit 3092a7762658 [SPARK-55265][INFRA] Increase timeout to 150 mins in GitHub Actions
    omit 9931781c4c8e [SPARK-55249][PYTHON] Make DataFrame.toJSON able to return dataframe
    omit 0dad3cb7bbde [SPARK-55030][SQL] Add vector norm/normalize function expressions support
    omit 8719c6c8dfd3 [SPARK-55240][CORE] Refactor LazyTry stacktrace handling to use wrapper instead of suppressed exception
    omit 778661a7e109 [SPARK-55255][BUILD] Upgrade `objenesis` to 3.5
    omit cf5898b4aa05 [SPARK-55250][SQL] Reduce Hive client calls on CREATE NAMESPACE
    omit cecf758b4323 [SPARK-55119][SQL] Fix Continue Handler: prevent INTERNAL_ERROR and incorrect conditional statements interruption
    omit cbc0ecd41007 [SPARK-55253][GEO][SQL][TESTS] Remove nonansi golden files for ST functions
    omit 67bfa7b317d5 [SPARK-55254][BUILD] Upgrade `analyticsaccelerator-s3` to 1.3.1
    omit 135cc503aaeb [SPARK-55252][CORE] Improve `HttpSecurityFilter` to add `Content-Security-Policy` header
    omit 589fedc9b231 [SPARK-55251][INFRA] Make `Python Coverage` test with Python 3.12
    omit 71335da93af5 [SPARK-55248][SS] Clean up Jackson deprecated API usage in `streaming.checkpointing.Checksum`
    omit 78b6cda9de51 [SPARK-55247][CONNECT] Clean up deprecated API usage related to `o.a.c.io.input.BoundedInputStream`
    omit 9ce067c7d532 [SPARK-55235][PYTHON][TESTS] Refactor tests for pandas udf input type coercion
    omit fa73e09d68e6 [SPARK-55238][GEO][SQL] Move Geo SRS mapping logic from `main/scala` to `main/java`
    omit 43529889f240 [SPARK-55179][PYTHON][CONNECT] Skip eager column name validation in `df.col_name`
    omit ff5676727722 [SPARK-54943][PYTHON][TESTS] Add test coverage for `pa.Array.cast` with `safe=False`
    omit 7f819398cb4b [SPARK-55245][PYTHON][PS][TESTS] Fix all timestamp freq usage from M to ME
    omit d069f733e95d [SPARK-55104][CONNECT][PYTHON][SS] Add Spark Connect support for DataStreamReader.name()
    omit 8c9167060f9b [SPARK-54906][PYTHON][TESTS][FOLLOWUP] Try to use a real module name instead of `__main__`
    omit f7a007bf00f4 [SPARK-55186][PYTHON] Make ArrowArrayToPandasConversion.convert_legacy able to return pd.DataFrame
    omit 141074665b06 [SPARK-55118][CORE] Replace ASM Opcodes wildcard imports
    omit e17d2735b44c [SPARK-55020][PYTHON] Disable gc when executing gRPC command
    omit f8a67849e398 [SPARK-55194][PYTHON] Remove GroupArrowUDFSerializer by moving flatten logic to mapper
    omit 3b2ab279b6ef [SPARK-55234][PYTHON][INFRA] Remove unused packages from remaining pyspark test images
    omit 63ea26030f96 [SPARK-55055][PYTHON] Support SparkSession.Builder.create for PySpark Classic
    omit dbb1437d8734 [SPARK-55202] Fix UNEXPECTED_USE_OF_PARAMETER_MARKER when using param…
    omit e0aeec048dab [SPARK-55197][PYTHON] Extract `_write_stream_start` helper to deduplicate START_ARROW_STREAM signal logic
    omit 55061e9c2a58 [SPARK-55223][DOCS] Document sinks in declarative pipelines programming guide
    omit 61d4581b7b2a [SPARK-48455][DOCS][MINOR] Remove unused DECLARE statement from IF statement example in Docs
    omit 8acdc7a83803 [SPARK-54887][CONNECT] Always set a sql state in spark connect client
    omit 0caa27ba67ce [SPARK-55169][PYTHON] Use ArrowBatchTransformer.flatten_struct in ArrowStreamArrowUDTFSerializer
    omit 541a23951b82 [SPARK-55174][SQL][TESTS] Move `Geo*ClientSuite` from `sql/core` to `sql/api` module
    omit c2521834b68d [SPARK-55232][BUILD][TESTS] Upgrade `bytebuddy` to 1.18.4
    omit 50077cd9943b [SPARK-54942][PYTHON][TESTS][FOLLOWUP] Fix struct cast tests for PyArrow < 21 compatibility
    omit b6505bcce837 [SPARK-55226][PYTHON][PS][NS] Recognize datetime and timedelta with units other than
    omit 27dfa09589d9 [SPARK-55158][PYTHON][TESTS] Make the test acknowledge of new string data type
    omit 22738c9042a6 [SPARK-55205][PYTHON][PS] Fix tests where it assumes string type will convert to object
    omit 7011706a0a8d [SPARK-55188][PYTHON][INFRA] Remove unused packages from python 3.14 test images
    omit d907da6d27d0 [SPARK-55187][PYTHON][INFRA] Remove unused packages from python 3.13 test images
    omit b7513740d1b4 [SPARK-55151] Fix RocksDBSuite testWithStateStoreCheckpointIds
    omit dec114ab628d [SPARK-55181][PYTHON][INFRA] Remove unused packages from python 3.10 test images
    omit 9538f4f0318e [SPARK-55182][PYTHON][INFRA] Remove unused packages from python 3.11 test images
    omit 479634c431af [SPARK-55164][PYTHON][TESTS][FOLLOW-UP] Fix golden file prefix
    omit b2056f36ea5f [SPARK-54179][SQL] Add Native Support for Apache Tuple Sketches
    omit d296bc029e6a [SPARK-54365][SS] Add Repartition Integration Test for Aggregate, Dedup, FMGWS and SessionWindow Operators
    omit f1f3af79e975 [SPARK-55058][SS] Throw error on inconsistent checkpoint metadata
    omit 7198633fbe9b [SPARK-55146][SS] State Repartition API for PySpark
    omit 34d9413ca161 [SPARK-55200][K8S][TESTS] Add `RancherDesktopBackend` for K8s integration tests
    omit 6062e2bd8025 [SPARK-55196][K8S][TESTS] Add `commandTestTag` to K8s Integration Test
    omit e1f08286ef6e [SPARK-55189][BUILD] Upgrade lz4-java to 1.10.3
    omit c4f62d459f24 [SPARK-55191][INFRA][DOCS] Adjust the Python version used in the Python-only daily test
    omit baaa62b086d1 [SPARK-55168][PYTHON] Use ArrowBatchTransformer.flatten_struct in GroupArrowUDFSerializer
    omit 718cbf54b397 [SPARK-55184][INFRA] Upgrade Python to 3.12 for Maven's daily testing
    omit 823fd431a125 [SPARK-55180][PYTHON][INFRA] Remove scheduled job python 3.13 with no gil
    omit 2bd4cf39781c [SPARK-53807][CORE][FOLLOWUP] Fix a race condition issue between `unlock` and `releaseAllLocksForTask` for write lock in `BlockInfoManager`
    omit 08823172d7c3 [SPARK-55166][BUILD] Upgrade `kubernetes-client` to 7.5.2
    omit d84345c20862 [SPARK-55021][CORE][TESTS][FOLLOWUP] Move timezone/locale initialization to constructor in SparkTestSuite
    omit 981af6029fe3 [SPARK-55160][PYTHON] Directly pass input schema to serializers
    omit e71eef6cc8a3 [SPARK-55154][PS] Remove fastpath to pd.Series for pandas 3
    omit 6a773e9476c8 [SPARK-54942][PYTHON][TESTS][FOLLOWUP] Fix PyArrow < 21 compatibility for float16 array creation in tests
    omit 1a04188d0c75 [SPARK-55170][PYTHON] Extract grouped stream reading pattern from serializers
    omit 3d7b5144948e [SPARK-55162][PYTHON] Extract transformers from ArrowStreamUDFSerializer
    omit 0f295cb0438b [SPARK-55172][PYTHON][INFRA] Remove unused packages from python 3.12 test image
    omit 78ec6f783b54 [SPARK-55173][K8S][TESTS] Improve K8s IT to use `TestConstants` consistently and `match` syntax for extensibility
    omit 03dedbbaeb1b [SPARK-55164][PYTHON][TESTS] Refactor tests for python udf input type coercion
    omit f7ce07aadd64 [SPARK-55171][PYTHON] Fix memory profiler on iter UDF
    omit ab3ec9e34441 [SPARK-55134] Fix `BasicExecutorFeatureStep` to throw `IllegalArgumentException` for executor cpu misconfigs
    omit 357c780d3942 [SPARK-55149][PYTHON][INFRA] Upgrade python to 3.12 in SQL tests
    omit 9c91dace9b20 [SPARK-55111][SS] Unfinished repartitioning detection on query restart
    omit 9d414bd53e30 [SPARK-54942][PYTHON][TESTS][FOLLOWUP] Add tests for pa.Array.cast of nested/container types
    omit faa465362b07 [SPARK-54996][SS] Report ArrivalTime for records from LowLatencyMemoryStream
    omit aa10c87dd319 [SPARK-55046][PYTHON] PySpark add udf processing time metric
    omit 8e1e126c68aa [SPARK-55121][PYTHON][SS] Add DataStreamReader.name() to Classic PySpark
    omit 137aa8ed6b1e [SPARK-55140][PYTHON][PS] Do not map builtin functions to numpy version for pandas 3
    omit 94cfa3d0e431 [SPARK-55132][INFRA] Upgrade numpy version on lint image
    omit c583ab2727ef [SPARK-55138][SQL] Fix `convertToMapData` throwing a `NullPointerException`
    omit 3f54007bfe4e [SPARK-54887][CONNECT] Remove deprecated annotation
    omit 310d687fcbd6 [SPARK-55137][SQL] Refactor `GroupingAnalyticsTransformer` and `Analyzer` code
    omit 655787cae15b [SPARK-55115][INFRA] Use master branch's base Dockerfile for release builds
    omit 9bc5c2a59414 [SPARK-55115][INFRA] Use composable Dockerfile for release builds
    omit 3f1c9a37a53c [SPARK-55128][INFRA][FOLLOWUP] Restore SQL tests by pin 'pandas==2.3.3' for maven daily test
    omit b06e9c5466a7 [SPARK-55142][PYTHON][INFRA] Apply Python 3.12 for PySpark Tests in PR build
    omit 8b26f49a226f [SPARK-55141][PYTHON][INFRA] Set up a scheduled workflow for Pandas 3
    omit 9c4d96200277 [SPARK-55135][PYTHON][TESTS] Refactor tests for pandas udf return type coercion
    omit 967ce4a9251c [SPARK-55125][PYTHON] Remove redundant `__init__` methods in Arrow serializers
    omit a8358c3b39d0 [SPARK-55021][CORE][TESTS] Introduce `SparkTestSuite` as the base class for all scalatest styles
    omit 7b379b77310c [SPARK-55127][SQL] Add avro_funcs group for Avro SQL functions
    omit 99b5571b9850 [SPARK-54965][PYTHON] Factor out legacy pa.Array -> pd.Series converter
    omit 17d361e23b46 [MINOR][PYTHON][TESTS] Delete obsolete `test_connect_plotting`
    omit 01a79c857639 [SPARK-54314][PYTHON][CONNECT] Improve Server-Side debuggability in Spark Connect by capturing client application's file name and line numbers
    omit dd48646ab868 [SPARK-55037][CONNECT][SQL] Re-implement Observation Without Using the QueryExecutionListener
    omit f762af639197 [SPARK-55130][BUILD] Upgrade log4j to 2.25.3
    omit 15d59810de74 [SPARK-54942][PYTHON][TESTS][FOLLOWUP] Add non-numerical type cast tests for PyArrow
    omit 2d0d317d749e [SPARK-55126][PYTHON] Remove unused timezone and assign_cols_by_name from ArrowStreamArrowUDFSerializer
    omit cb69e28758f0 [SPARK-55124][PS][CONNECT][TESTS] Move `test_spark_arrow_c_streamer_arrow_consumer` to PS and enable the connect parity test
    omit e77c38a8d439 [SPARK-55113][SQL] `EnsureRequirements` should copy tags
    omit dafb2cd2447d [SPARK-55128][INFRA] Restore SQL tests by pin 'pandas==2.3.3'
    omit 8fa9a79b1b5a [SPARK-55112][SQL][TESTS] Add micro-benchmark for `o.a.spark.util.sketch.BloomFilter`
    omit 6a7f5def6319 [SPARK-55116][BUILD] Upgrade Jackson to 2.21.0
    omit b160c29e2b74 [SPARK-54942][PYTHON][TESTS] Add tests for pa.Array.cast of numerical types with default arguments
    omit 5fca84e03689 [SPARK-54941][PYTHON][TESTS][FOLLOW-UP] pa.scalar type coercion non-numerical
    omit 76f74e96aa72 [SPARK-55114][PYTHON][TESTS] Refactor tests for python udf return type coercion
    omit bed09429092f [SPARK-55040][PYTHON] Refactor TaskContext and related worker protocol
    omit 5ef18f8bebc9 [SPARK-55074][SQL] Add test for Merge Into ANSI type coercion
    omit e3bc281c58d3 [SPARK-54217][PYTHON] Synchronize PythonRunner's MonitorThread kill decision
    omit 45094dc41d3b [SPARK-55044][SPARK-55088][TESTS][FOLLOW-UP] Add test for passing metadata in arrow batches from JVM to Python
    omit 4e52198513ec [SPARK-54914][SQL] Fixing DROP operator in pipe syntax to support qualified column names
    omit 9b3e203e1d02 [SPARK-55045][CORE][TESTS] Fix a flaky test in ExecutorSuite
    omit 5932f11fb916 [SPARK-54990][PYTHON] Fix how classproperty is implemented in session.py
    omit 06468a85a7ea [SPARK-54925][PYTHON][FOLLOWUP] Ignore missing import for pystack
    omit 9d9fbeea8bdc [SPARK-54939][PYTHON][TESTS][FOLLOWUP] Handle PyArrow version difference in type coercion tests
    omit a2198a1e2542 [SPARK-55108][PYTHON] Use the latest pandas-stubs for type check
    omit 4175544c5747 [SPARK-54941][PYTHON][TESTS] Add tests for pa.scalar type coercion for numerical types
    omit a28133134c4d [SPARK-55103][CORE][TESTS] Fix a flaky test in `RpcIntegrationSuite`
    omit d14bed9ae422 [SPARK-55098][PYTHON] Vectorized UDFs with output batch size control fail with memory leak
    omit 888fb67699ca [SPARK-55071][CONNECT][PYTHON] Make spark.addArtifact work with Windows paths
    omit dfea03626309 [SPARK-55097][SQL] Fix re-adding cached artifacts drops blocks silently issue
    omit edc7c96741c4 [SPARK-54374][SQL][UI] Enlarge the SVG viewBox attribute of SQL plan visualization initialization
    omit 5f925fddfff8 [SPARK-55096][PYTHON] Update pandas minimum version in `connect/setup.py`
    omit 4d3b45db9195 [SPARK-54849][PYTHON][FOLLOW-UP] Update minimum version of pyarrow in `setup.py` and `pyspark_upgrade.rst`
    omit 2af1cfeedaa5 [SPARK-55091][SQL] Reduce Hive RPC calls for DROP TABLE command
    omit 4aea124e58f2 [SPARK-55026][CORE][SQL] Optimize BestEffortLazyVal
    omit a5ed572a6e60 [SPARK-55036][PYTHON] Add `ArrowTimestampConversion` for arrow timezone handling
    omit 9f0b2383a4a6 [SPARK-55053][PYTHON] Refactor data source/udtf analysis workers so they have a unified entry
    omit fe2985172f57 [SPARK-55016][SQL] Make SQLConf a direct property of SparkSession to prevent stack overflow
    omit 880a0cf0910f [SPARK-55089][CONNECT] Correct the output schema of toJSON
    omit daa21cd2bd46 [SPARK-54873][SQL] Simplify V2TableReference resolution as only temp view may contain it
    omit 789acc92d143 [SPARK-55090][PYTHON][CONNECT] Implement `DataFrame.toJSON` in Python Client
    omit 6d0dacba1958 [SPARK-55063][SQL][DOCS] Reorganize built-in functions documentation by category
    omit 6167ef277d0a [SPARK-55065] Avoid making two JDBC API calls
    omit eec21d0c803f [SPARK-55035] Perform shuffle cleanup in child executions
    omit 3ac9d9e32bd2 [SPARK-54830][SQL][BUILD][FOLLOWUP] Add comments to remind that the SQL module should use the same `-Xmx` setting in both Maven and SBT tests
    omit e696c5ebc6a4 [SPARK-54830][SQL][TESTS][FOLLOWUP] Increase `-Xmx` for the sql module to 6g for maven test
    omit 7a0ea39fd6a9 [SPARK-55033][SQL] Fix stringArgs of DSv2 writing commands
    omit 8549e725292e [SPARK-55088][PYTHON] Keep the metadata in to/from_arrow_type/schema
    omit b0089c3887e8 [SPARK-55041][CORE] Cleanup some unused private `funcion/val` from core module
    omit 5b1047838dc1 [SPARK-54866][SQL][FOLLOWUP] Fix DROP FUNCTION IF EXISTS to not fail when namespace doesn't exist
    omit 76bdd24677e4 [SPARK-55070][SQL][CONNECT] Allow hidden column in dataframe column resolution
    omit c6ca25a8028b [SPARK-54956][CORE] Unify the indeterminate shuffle retry solution
    omit f30bf06540af [SPARK-55051][CORE] Byte string accepts KiB, MiB, GiB, TiB, PiB
    omit 0a487a994f51 [MINOR] add .venv into .gitignore
    omit 67f6a3fe5e87 [SPARK-55043][SQL] Fix time travel with subquery containing table references
    omit 70a3ab5daa48 [MINOR][PYTHON][TESTS] Add test for Row with duplicated field names
    omit 236e71d67540 [SPARK-54987][PYTHON][DOCS][FOLLOW-UP] Update the docstring of `to_arrow_type`
    omit f5b620ecbec7 [SPARK-55034][PYTHON][TESTS] Test `test_arrow_udf_output_timestamps_ltz` without timezone conversion to UTC
    omit b67ef6c29da3 [SPARK-55032][PYTHON] Refactor profilers in workers.py
    omit 2e83db4a17d6 [SPARK-55027][PYTHON] Move writeConf to PythonWorkerUtils
    omit d7932c9e29ff [SPARK-55044][SQL] Keep the metadata in toArrowSchema/fromArrowSchema
    omit e4bbafdd3d39 [SPARK-54922][PYTHON][FOLLOWUP] Ignore ununsed-ignore because we support both numpy 1 and 2
    omit 0a354bc975ad [SPARK-55076][PYTHON] Fix the type hint issue in ml/mllib and add scipy requirement
    omit 69f1d5c5e461 [SPARK-55025][PS] Improve performance in pandas by using list comprehension
    omit 84f978ae6b42 [SPARK-46165][PS] Add support for pandas.DataFrame.all axis=1
    omit e6c97ebf3f0f [SPARK-54337][PS][TESTS][FOLLOWUP] Fix test_arrow_c_stream when pandas/arrow does not exist
    omit 5688ee00b474 [SPARK-55057][SS] Add streaming source naming infrastructure and resolution pipeline
    omit 0497b386ec4e [SPARK-54824] Add support for multiGet and deleteRange for Rocksdb State Store
    omit 195c7dc9d2a2 [SPARK-55054][SS][SQL] Add IDENTIFIED BY support for streaming table-valued functions
    omit 57c5238b276a [SPARK-54975][SQL][TESTS] Refactor Merge Into Schema Evolution Tests
    omit 6939fcd4a313 [SPARK-43986][SQL][FOLLOWUP] Catch ArrayIndexOutOfBoundsException in Datasketches HLL functions
    omit 31857a2efad6 [SPARK-55068] Upgrade `kubernetes-client` to 7.5.1
    omit ac13473fff64 [SPARK-55047][CONNECT] Add client-side limit for local relation size
    omit 5b9e78b059d5 [SPARK-54980][SQL] Convert the JDBC isolation level to a string
    omit 6f0900ec0230 [SPARK-55061][PYTHON][TESTS] Make module missing message consistently string
    omit af058d59f300 [SPARK-55015][SS][SQL] Fix decodeRemainingKey numFields calculation in PrefixKeyScanStateEncoder
    omit 813f7576aef2 [SPARK-54590][SS] Support Checkpoint V2 for State Rewriter and Repartitioning
    omit 83e46000f475 [SPARK-55039][SS][SQL] Add IDENTIFIED BY syntax for naming streaming sources
    omit f310f4fcc955 [SPARK-54886] Lazily create Spark Connect reuse session
    omit 0e580b3ef71a [SPARK-55042][WEBUI][BUILD][TESTS] Upgrade Node.js used for UI test to `24.13.0`
    omit a03bedb6c128 [SPARK-54559] Use runnerConf to pass profiler option
    omit 62efc7f364cf [SPARK-54939][PYTHON][TESTS] Add tests for pa.array type coercion
    omit d1996db65611 [SPARK-55018][PYTHON][TESTS][FOLLOWUP] Revert the interval for test_convergence
    omit 3ad4ecaffff2 [SPARK-54822][BUILD] Bump Parquet 1.17.0
    omit 9df8644a59e9 [SPARK-55029][SS] Propagate streaming source identifying name through resolution pipeline
    omit 075b4bf63ee1 [SPARK-55024][SQL] Use REQUIRES_SINGLE_PART_NAMESPACE error for session catalog namespace validation
    omit 22c6f9f084c2 [SPARK-54870][SQL][FOLLOWUP] Remove dead code and fix typo
    omit 6831481fd7a2 [SPARK-55023][CORE][SQL][TESTS] Remove unnecessary `BeforeAndAfterAll` and `BeforeAndAfterEach`
    omit f89fbd49c450 [MINOR][SQL] Extend Uniform to be TimeZoneAwareExpression
    omit 9eb9ba0c09e0 [SPARK-55028][PYTHON] Pass result_id unconditionally to UDF workers
    omit 98fccd93ebee [SPARK-54938][PYTHON][TESTS] Add tests for pa.array type inference
    omit 44a96b4a0932 [MINOR][INFRA] Add GitHub copilot instructions/agents/prompts/skills to .gitignore
    omit e1f5387669c3 [SPARK-54992][PYTHON] Replace cast with a runtime check for make_timestamp
    omit be0a5be5d064 [SPARK-55013][SS][SQL] Add SQL parser support for streaming source naming infrastructure
    omit c3843a0bbc47 [SPARK-54881][SQL] Improve `BooleanSimplification` to handle negation of conjunction and disjunction in one pass
    omit 5bcbc54acc11 [SPARK-55022][SQL][TESTS] Remove unnecessary mixins with `SharedSparkSession`
    omit 81dea41e3404 [SPARK-54961][SQL] Introduce `GroupingAnalyticsTransformer`
    omit b213089c3b5d [SPARK-55019][SQL] Allow DROP TABLE to drop VIEW
    omit 69e5ba3b56d9 [SPARK-54971] Add WITH SCHEMA EVOLUTION syntax for SQL INSERT
    omit 50dd494a7b1a [SPARK-55018][PYTHON][TEST] Fix test_convergence by not overflowing the list
    omit 44e38b56cf97 [SPARK-55020][PYTHON][TEST] Skip flaky test_distributed_lda test temporarily
    omit ffb1624ef6d3 [SPARK-55008][SQL][WEBUI] Display Query ID in SparkUI
    omit 0218b8ad3f8f [SPARK-55009][CORE] Remove unnecessary memory copy from the constructor of `LevelDBTypeInfo/RocksDBTypeInfo.Index`
    omit c5965ed0cb39 [SPARK-54889][INFRA][FOLLOWUP] Enable codecov test report on all stable workflows
    omit ea107b4cdcfb [SPARK-54995][PYTHON] Make a fast path for foreachPartition
    omit c270cc279ba4 [SPARK-47086][BUILD][CORE][FOLLOWUP] Fix `NoClassFoundError` related to Jetty 12 when building with Maven
    omit 236727f5f897 [SPARK-54991][PYTHON] Correct type hint for streaming/listener.py
    omit 99114890ec53 [SPARK-54957][PYTHON][TEST] Remove the unnecessary sleep in streaming tests
    omit 192327ff8acf [SPARK-54870][SQL] Collation support for char/varchar and CTAS/RTAS
    omit fc15f726eabd [SPARK-54785][SQL] Add support for binary sketch aggregations in KLL
    omit 721593ebc6c0 [SPARK-54864][SQL][FOLLOWUP] Fix rCTEs in NormalizePlan
    omit 41b52e3a8200 [SPARK-54866][SQL] Refactor Drop/RefreshFunction to avoid catalog lookup
    omit f1bab5a9e14c [SPARK-54989][PYTHON][TEST] Improve eventually util
    omit 209e935ea2ca [SPARK-54987][PYTHON] Change the default value of prefer_timestamp_ntz to True in `from_arrow_type/from_arrow_schema`
    omit 959a1476bb88 [SPARK-54821][DOCS] Deprecate support for Derby JDBC datasource
    omit 5fed54b074b3 [SPARK-54979][BUILD] Upgrade ORC to 2.2.2
    omit de3430932757 [SPARK-54960][INFRA] Simplify generating contributors of the release process
    omit 70a4a1e7ac05 [SPARK-54988][SQL] Simplify the implementation of ObservationManager.tryComplete
    omit 3ac3b51021b4 [SPARK-54959][CORE] Disable shuffle checksum mismatch fully retry when push based shuffle enabled
    omit 7a40891ac134 [SPARK-55005][SQL] Fix CONTINUE HANDLER to continue loop execution after handling exceptions in loop body
    omit ecf179c3485b [SPARK-54337][PS] Add support for PyCapsule to Pyspark
    omit ad607839421b [SPARK-54925][PYTHON][FOLLOWUP] Fix threaddump for run-tests
    omit b91878ed35f0 [MINOR][DOCS] Add a bit of more description for `ASF_NEXUS_TOKEN`
    omit acc80fd1b205 [SPARK-54984][SS] State Repartition execution and integrate with State Rewriter
    omit cc9e4112f5a6 [SPARK-54907][SS] Introduce NameStreamingSources analyzer rule for streaming source evolution
    omit d2b00ae84341 [SPARK-54443][SS] Integrate PartitionKeyExtractor in Re-partition reader
    omit 7009fb58a98e [SPARK-54660][SS] Add RTM trigger to python
    omit 5e18c9dbe8b6 [SPARK-53882][CONNECT][DOCS] Add documentation comparing behavioral differences between Spark Connect and Spark Classic
    omit 8138db757764 [SPARK-54887][CONNECT] Add optional sqlstate to sparkthrowables
    omit 8e63b6111efe [SPARK-54972][SQL] Improve NOT IN subqueries with non-nullable columns
    omit 6ec751177664 [SPARK-54940][PYTHON] Add tests for pa.scalar type inference
    omit 94662eb63391 [SPARK-47086][SPARK-48022][BUILD][CORE][WEBUI] Upgrade Jetty to 12.1.5, Jersey to 3.1.11 and Servlet to 6.0
    omit d4bd663581e7 [SPARK-54122][SS][FOLLOWUP] Fix Javadoc generation error
    omit 98a17fdd6cb7 [SPARK-54864][SQL] Add rCTE nodes to NormalizePlan
    omit 60bbac0b892d [SPARK-54962][PYTHON] Fix nullable integers handling in Pandas UDF
    omit bf71e593a87c [SPARK-54949][PYTHON] Move pyproject.toml to repo root
    omit c26a3e481e02 [SPARK-54982][INFRA] Use ASF_NEXUS_TOKEN to release and drop staging repository when finalizing the RC
    omit 309407c2c412 [SPARK-54977][PYTHON][TEST] Accelerate test_observation.py
    omit 5910b09c4bfb [SPARK-54122][SS] Implement TwsTester in Scala
    omit 6933078aef09 [SPARK-54976][PYTHON][TEST] Accelerate test_job_cancellation
    omit 78d5229980e1 [SPARK-54958][PYTHON][ML][TEST] Accelerate test_cv_io_pipeline.py
    omit fe9ffd50e58b [SPARK-54963][PYTHON][CONNECT] Make `createDataFrame` respect `prefer_timestamp_ntz` when `infer_pandas_dict_as_map`
    omit 38d3af27535d [SPARK-54954][PYTHON] Fix remote related type hints in util.py
    omit 49afd6b58144 [SPARK-54910][SS] Add sourceIdentifyingName parameter to StreamingRelationV2
    omit 2fff424dba26 [SPARK-54776][SQL] Improved the logs message regarding lambda function with SQL UDF
    omit 9c675099c9bb [SPARK-54634][SQL] Add clear error message for empty IN predicate
    omit 858574213d9a Revert "[SPARK-54922][PYTHON] Remove ignore[arg-type] for np.set_prin…
    omit 3115669444e0 [SPARK-54925][PYTHON] Add the capability to dump threads for pyspark
    omit 3dc641b4a384 [SPARK-54922][PYTHON] Remove ignore[arg-type] for np.set_printoptions
    omit 6b39e94e250e [SPARK-54950][PYTHON][TEST] Move unittest magic to unittestutils
    omit 5a5308695304 [SPARK-54930][PYTHON] Remove redundant _accumulatorRegistry.clear() call in worker.py
    omit 186570cdc569 [SPARK-54937][PYTHON][TESTS] Add tests for timezone conversion w/o PYARROW_IGNORE_TIMEZONE
    omit f27f8d9c7105 [SPARK-54933][PYTHON][CONNECT] Avoid repeatedly fetching config `binary_as_bytes` in `toLocalIterator`
    omit bdafb1eb7b81 [SPARK-54929][PYTHON] Fix taskContext._resources reset in loop causes only last resource to be kept
    omit 934a171124cc [SPARK-54948][PYTHON][TEST] Use the same testsuite name for same tests
    omit 9aea212dfd4f [SPARK-53991][SQL][TEST][FOLLOW-UP] Improve KLL quantile errors by removing internal details
    omit 2871905e9ba8 [SPARK-54924][SS] State Rewriter to read state, transform it and write new state
    omit 1e8048a8810f [SPARK-54865][CONNECT][SQL] Add foreachWithSubqueriesAndPruning method to QueryPlan
    omit 442f9083c0cb [SPARK-54905][SQL] Simplify foreachWithSubqueries implementation in QueryPlan
    omit 6df8d57b30e7 [SPARK-54771][SQL] Remove ResolveUserSpecifiedColumns rule from RuleIdCollection
    omit 57b2a065c418 [SPARK-54852][SQL] `NOT IN` subquery returns incorrect results with a collated table
    omit d1aa7d6c4427 [SPARK-54713][SQL] Add vector similarity/distance function expressions support
    omit ed201397c2ac [SPARK-49872][FOLLOWUP] Remove Jackson JSON string length limit in KVStoreScalaSerializer
    omit ce6b5a62567e [SPARK-54912][BUILD] Upgrade `commons-pool2` to 2.13.1
    omit c27ede3a070d [SPARK-54906][PYTHON][TESTS] Unify test entry for all pyspark tests
    omit f8760286a679 [SPARK-54922][PYTHON] Unify how args are passed to python workers
    omit 400de3cf66c9 [SPARK-54916][K8S][BUILD] Enable `volcano` profile by default
    omit dfb8834f9cad [SPARK-54768][SS] Python Stream Data Source should classify error if data returned doesn't match configured schema
    omit f45fbbd1ab11 [SPARK-54920][SQL] Move grouping analytics extraction logic to a common `GroupingAnalyticsExtractor`
    omit fbdbd016cd71 [SPARK-54872][SQL] Unify column default value handling between v1 and v2 commands
    omit 2829fb9304d0 [SPARK-54682][SQL] Support showing parameters in DESCRIBE PROCEDURE
    omit 92966b31b1e1 [SPARK-54915][K8S][INFRA][DOCS] Upgrade Volcano to 1.13.1
    omit b615f34aebd7 [SPARK-54913][BUILD] Upgrade `gcs-connector` to 2.2.31
    omit 752e8a1a7ee0 [SPARK-53807][SPARK-50771][CORE] Fix race condition issues between `unlock` and `releaseAllLocksForTask` in `BlockInfoManager`
    omit 145be5fc6ff5 [SPARK-54313][CORE] Add --extra-properties-file option for configuration layering
    omit c806dfd90c45 [SPARK-54911][BUILD][TESTS] Upgrade `bouncycastle` to 1.83
    omit 8cbff856735f [SPARK-54902][PYTHON][TESTS] Rename testing helper contextmanager for temp view
    omit 7b62f5fc6360 [SPARK-54903][PYTHON] Make to_arrow_schema/to_arrow_type able to set timezone
    omit 98874f0a376c [SPARK-54904][BUILD] Upgrade SBT to `1.12.0`
    omit 1d86c4d023ea [SPARK-54632][PYTHON][FOLLOWUP] Add ruff to cli help options
    omit 8797f6f6b55d [MINOR] Fix an invalid escape string
    omit e13fb1184b74 [SPARK-54803] Support BY NAME with INSERT INTO ... REPLACE WHERE
    omit c76eeebc4780 [SPARK-54468][SDP] Add missing error classes
    omit 0b5e92b7815d [SPARK-54886] Add base session created in SparkConnectService
    omit 7bc810745c9e [SPARK-54854][SQL] Add a UUIDv7 queryId to SQLExecution Events
    omit 3e73da2d904c [SPARK-54889][INFRA] Report test results to codecov
    omit 5c88e9385e31 [SPARK-54894][PYTHON] Fix argument passing in `to_arrow_type`
    omit 25307abfbfbc [SPARK-54830][CORE] Enable checksum based indeterminate shuffle retry by default
    omit c12bb71a069e [MINOR] Improve test coverage for Dataset API CoGroup
    omit 9d8126027875 [SPARK-54883][MINOR] Clean up error messages for CLI and add new error mode DEBUG
    omit 20de5e7f2bdb [SPARK-54884][CORE][TESTS] Remove private function `totalRunningTasksPerResourceProfile` from `ExecutorAllocationManager`
    omit 2f12bc655875 [SPARK-54610][SDP][DOCS] Improve SDP Programming Guide
    omit ea2617af1ca4 [SPARK-54543][PYTHON][INFRA] Retire flake8
    omit 735eda8a5b6b [SPARK-54874][TESTS][INFRA] Avoid interleave failed test logs with test outputs
    omit 1fc6ff235345 [SPARK-54868][PYTHON][INFRA][FOLLOW-UP] Combine timeout path into TestRunner
    omit 03537905f144 [SPARK-54871][SQL] Trim aliases from grouping and aggregate expressions before handling grouping analytics
    omit 0567dcdf0c3a [SPARK-54885][PYTHON] Remove unreachable code after upgrading pyarrow minimum version to 18.0.0
    omit 5da9d2a47e3a [SPARK-54867][SS] Introduce NamedStreamingRelation wrapper for source identification during analysis
    omit a309a0570279 [SPARK-54390][SS] Fix JSON Deserialize in StreamingQueryListenerBus
    omit 8fe006b20877 [SPARK-54882][PYTHON] Remove legacy PYARROW_IGNORE_TIMEZONE
    omit 33769b8b78d2 Revert "[SPARK-54182][SQL][PYTHON] Optimize non-arrow conversion of df.toPandas`"
    omit a911310f696e [SPARK-54877][UI] Make display stacktrace on UI error page configurable
    omit eaed471c8f4e [MINOR][BUILD] Remove undefined -Pscala-2.13 profile from dev scripts
    omit 6695a6911e78 [SPARK-54701][PYTHON] Improve the runnerConf chain for Python workers
    omit b45f071cd1b6 [SPARK-54868][PYTHON][INFRA][FOLLOW-UP] Also enable `faulthandler` in classic tests
    omit 5b5ff6d732ce [SPARK-54753][SQL] fix memory leak of ArtifactManager
    omit 2ab68d10e806 [SPARK-54859][PYTHON] Arrow by default PySpark UDF API reference doc
    omit 2fb3359088f9 [SPARK-54868][PYTHON][INFRA][FOLLOWUP] Increase test timeout to 450s
    omit faa4d19a1cd9 [SPARK-54849][PYTHON][FOLLOW-UP] Update the version in `require_minimum_pyarrow_version`
    omit f80b919d19ce [SPARK-54849][PYTHON] Upgrade the minimum version of `pyarrow` to 18.0.0
    omit 4b77986b3945 [SPARK-54868][PYTHON][INFRA] Fail hanging tests and log the tracebacks
    omit 1bc5b29161b5 [SPARK-54862][DSTREAM] Remove unused private object `RawTextHelper`
    omit 2c5b4b09868d [SPARK-54861][CORE] Reset task thread name to IDLE_TASK_THREAD_NAME  when task completes
    omit ea8e9412b1ce Revert "[SPARK-54830][CORE] Enable checksum based indeterminate shuffle retry by default"
    omit 8095c39aaa92 [SPARK-54869][PYTHON][TESTS] Apply the standard import of pyarrow compute in `test_arrow_udf_scalar`
    omit 678f314c3345 [SPARK-54860][INFRA] Add JIRA Ticket Validating in GHA
    omit 082de7fcdc40 [SPARK-54850][SQL] Improve `extractShuffleIds` to find `AdaptiveSparkPlanExec` anywhere in plan tree
    omit 2afc713698bc [SPARK-54830][CORE] Enable checksum based indeterminate shuffle retry by default
    omit 8cc2e4677d59 [SPARK-54856][SQL] Refactor scalar function lookup and resolution
    omit df604f04f103 [SPARK-54857][WEBUI][TESTS] Add test ensuring user name and app name in historypage get escaped
    omit 578ee01678e3 [SPARK-54837][DOCS] Document all built-in JDBC datasource providers
    omit 8b68a172d34d [SPARK-54858][INFRA] Fix syntax error in `test_report.yml` and recover GA workflow
    omit 8a6b685bf1b6 [SPARK-54853][SQL] Always check `hive.exec.max.dynamic.partitions` on the spark side
    omit eec092c9f9d1 [SPARK-54851][BUILD] Support generating bloop files via sbt
    omit 174b62d01212 [SPARK-54847][BUILD] unify the proto/antlr output folder between sbt and maven
    omit df720c179584 [SPARK-54843][SQL] Try_to_number expression not working for empty string input
    omit 355c7cbdc149 [SPARK-54840][SQL] OrcList Pre-allocation
    omit f63ccce6647f [SPARK-54818][SQL] TaskMemoryManager allocate failed should log errorstack to help check memory usage
    omit 24207e2e9cd1 [SPARK-54834][SQL] Add new interfaces SimpleProcedure and SimpleFunction
    omit 76c9516417d1 [SPARK-54835][SQL] Avoid unnecessary temp QueryExecution for nested command execution
    omit eaaf3cabb97f [SPARK-54844][BUILD] Upgrade maven.version to 3.9.12
    omit f7b20da56156 [SPARK-54842][PYTHON][TESTS] Fix `test_arrow_udf_chained_iii` in Python-Only MacOS26
    omit 3cc3cc1c76d0 Revert "[SPARK-54839][PYTHON] Upgrade the minimum version of `numpy` to 2.0.0"
    omit 51c6649e2868 [SPARK-54839][PYTHON] Upgrade the minimum version of `numpy` to 2.0.0
    omit a021fecf44d2 [MINOR] Add Python 3.14 no GIL workflow in README
    omit e80b65c7424e [SPARK-54829][PYTHON][INFRA][FOLLOW-UP] Also refresh pypy images
    omit 7d61638634d9 [SPARK-54826][INFRA][FOLLOW-UP] Delete removed path from build image caches
    omit ec12946129cd [SPARK-54829][PYTHON][INFRA] Refresh docker images for numpy 2.4
    omit b14a63ad7c40 [SPARK-54826][INFRA] Delete scheduled job for numpy 2.1.3
    omit 37ead02eb13a [SPARK-54827][SQL] Add helper function `TreeNode.containsTag`
    omit f25b381dfc0c [SPARK-54828][PYTHON][INFRA] Set up a scheduled builder for free-threaded Python 3.14
    omit e316d28be78d [SPARK-54817][SQL] Refactor `Unpivot` resolution logic to `UnpivotTransformer`
    omit 87a3b0655139 [SPARK-54816][SQL] Make SQL configs not lazy
    omit 72238e7e62b1 [SPARK-54777][SQL] Changed dropTable error handling in JDBCTableCatalog.dropTable(...)
    omit 316322cbcb55 [SPARK-54820][PYTHON] Make `pandas_on_spark_type` compatible with numpy 2.4.0
    omit 512099bfb77c [SPARK-54760][SQL] DelegatingCatalogExtension as session catalog supports both V1 and V2 functions
    omit db36f74f8fb0 [SPARK-54815][CONNECT] Do not close the class loader of the session state if session is still in use
    omit d65ee811b0f3 [SPARK-46741][SQL] Cache Table with CTE should work when CTE in plan expression subquery
    omit 1a1ab19617b1 [SPARK-54799][SQL] Refactor `UnpivotCoercion`
    omit a76e4fa54b38 Revert [SPARK-54758][SQL] Fix generator resolution order in Project
    omit 14cc174a4e4c [SPARK-54802][SQL][DOCS] Fix typos in OrderedFilters.scala
    omit 43575015be30 [SPARK-54787][PS] Use list comprehension instead of for loops in pandas
    omit c9cb5a240144 [SPARK-54801][SQL] Mark a few new 4.1 configs as internal
    omit 0a5a8431f740 [SPARK-54137][SQL][CONNECT] Remove redundant observed-metrics responses
    omit 52df31ae71dc [MINOR][PYTHON][TESTS] Move `test_query_execution_unsupported_in_classic` in `DataFrameTestsMixin`
    omit 5e41971f2734 [SPARK-54411][SS] Introduce Repartition Writer that Support Multi-CF
    omit aad5b6e19c6e [SPARK-54800] Changed default implementation for isObjectNotFoundException
    omit dfb3a5105f58 [SPARK-54798][BUILD] Upgrade commons-text to 1.15.0
    omit 62cc5a72881a [SPARK-54797][SS][TESTS] Fix deprecation warning in `GlobalSyncClockClient.getTimeMillis`
    omit e886428cdfd3 [SPARK-54735][SQL] Properly preserve column comments in view with SCHEMA EVOLUTION
    omit e4b75089034b [SPARK-52407][SQL][FOLLOWUP] Expression description fix for ThetaIntersectionAgg
    omit e4b99932d02c [SPARK-53991][SQL][TEST][FOLLOWUP] Make KLL quantile golden file tests deterministic
    omit 703da9ded7fc [SPARK-54597][BUILD][FOLLOW-UP] Explicitly exclude `org.lz4:lz4-java` in SBT build
    omit d0fbb15bbc98 [SPARK-54794][CORE] Suppress verbose `FsHistoryProvider.checkForLogs` scanning logs
    omit d5b51fe31b9e [SPARK-45720][BUILD][DSTREAM][KINESIS] Upgrade KCL to 2.7.2 and remove AWS SDK for Java 1.x dependency
    omit d0cbad56a105 [SPARK-54782][SQL] Correct the config versions
    omit 20af8bdfb907 [SPARK-54787][PS] Use list comprehension in pandas _bool_column_labels
    omit 27c51009dc33 [SPARK-46166][PS] Implementation of pandas.DataFrame.any with axis=None
    omit a8f817d1ad9d [SPARK-54745][PYTHON] Fix PySpark import error caused by missing UnixStreamServer on Windows
    omit 532de6476ff5 [SPARK-54790][PYTHON][TESTS] Add UDTF analyze coverage tests
    omit f8f9f137ddef [SPARK-54572][PYTHON][FOLLOWUP] Cleanup all child processes for vscode script
    omit 588fab4afc2d [SPARK-54561][PYTHON] Support breakpoint() for `run-tests.py`
    omit a3934273c553 [SPARK-54739][INFRA] Cancel workflow if a new commit is pushed
    omit d3633f18f83e [SPARK-51966][PYTHON] Replace select.select() with select.poll() when running on POSIX os
    omit 00163b828b33 [SPARK-54754][SQL] OrcSerializer should not parse the schema every time it is serialized
    omit 4b90a265c623 [SPARK-54761][SQL] Throw unsupportedTableOperation for constraint operation on DSv1/HMS table
    omit 1e6d743c7ae2 [SPARK-54750][SQL] Fix ROUND returning NULL for Decimal values with l…
    omit 0da9e0505b59 [SPARK-54556][CORE] Rollback succeeding shuffle map stages when shuffle checksum mismatch detected
    omit 09a2cadc1fb4 [SPARK-54696][CONNECT] Clean-up Arrow Buffers - follow-up
    omit 9dfd84ee3361 [SPARK-54781][ML][CONNECT] Return model cache information in JSON format
    omit bcbf7d6abf40 [SPARK-54780][K8S][DOCS] Drop K8s v1.32 Support
    omit 59977a84257e [SPARK-54720][SQL] Add SparkSession.emptyDataFrame with a schema
    omit 1aa4b15eec3d [SPARK-54763][TEST] Accelerate test_udf_return_types with multi-threading
    omit 80b4486a6fdc [SPARK-54769][PYTHON] Remove dead code in conversion.py
    omit 2549a58f1abc [SPARK-54767][K8S][INFRA] Update K8s IT CI to use K8s 1.35
    omit 81cb3e88b956 [SPARK-54744][PYTHON] Only invalidate cache when necessary
    omit bc47877693ac [SPARK-54758][SQL] Fix generator resolution order in Project
    omit 1d8c11120fde [SPARK-54652][SQL] Complete conversion of IDENTIFIER()
    omit 02c016fe4911 [SPARK-54762][PYTHON] Fix `_create_converter` and `covert` overload signature
    omit a26c20953bb3 [SPARK-54749][SQL] Fix incorrect numOutputRows metric in OneRowRelationExec
    omit 40ccc0182eee [SPARK-54752][ML][CONNECT][TESTS] Test model offloading of LDA and FPGrowth
    omit de81f4b6e488 [SPARK-54690][PS] Make `Frame.__repr__` insensitive to arrow optimization
    omit 8f696791507d [SPARK-46741][SQL] Cache Table with CTE won't work
    omit 92e5b3644d56 [SPARK-54621][SQL] Merge Into Update Set * preserve nested fields if …coerceNestedTypes is enabled
    omit 19f025ceea42 [SPARK-54553][K8S] Support `spark.kubernetes.scheduler.volcano.podGroupTemplateJson`
    omit 40879c6cd2e6 [SPARK-54746][DOC] Fix wrong link in SQL Scripting page
    omit 025f5a8205e2 [SPARK-54727][BUILD] Upgrade Netty to `4.2.9`
    omit a4e456318eae [SPARK-54738][PYTHON] Add Profiler Support for Pandas Grouped Iter Aggregate UDF
    omit 8af0331bf578 [SPARK-54742][ML][CONNECT][TESTS] Add tests for connect ML model offloading
    omit 81540fa39b9f [SPARK-54722][PYTHON][SQL] Register Pandas Grouped Iter Aggregate UDF for SQL usage
    omit c64f8cb6b0aa [SPARK-54723][PYTHON][TESTS] Add Test for @udf Usage on Arrow Grouped Aggregate Iter UDF
    omit 28938e331d2b [SPARK-54724][INFRA] Hook worker exception handler to dump report before sending exception
    omit 906c646a9473 [SPARK-54741][ML][CONNECT][TESTS] Restore ClusteringParityTests
    omit 0faa10e7d128 [SPARK-54419][SS] Offline Repartition State Reader Supports Multi-Column Families
    omit e308660e8d83 [SPARK-54736][TEST][INFRA] Change the default executable to sys.executable
    omit b634d3cd7e45 [SPARK-52764][ML][CONNECT][TESTS] Restore `test_parity_classification` and `test_parity_regression`
    omit 36f8f1e1003c [SPARK-54730][SQL][CONNECT] Delay failure of dataframe column resolution
    omit 5a7e96c05f4c [SPARK-54711][PYTHON] Add a timeout for daemon created worker connection
    omit 92d2a158472e [SPARK-54541][SQL] Rename _LEGACY_ERROR_TEMP_1007 and add sqlState
    omit e3569bafd8da [SPARK-52819][SQL] Making KryoSerializationCodec serializable to fix java.io.NotSerializableException errors in various use cases
    omit 1413dbb19d02 [SPARK-41916][FOLLOW-UP] Remove unused imports
    omit 7735522de0cd [SPARK-54718][SQL] Preserve attributes names during CTE newInstance()
    omit cebd4abe40bd [SPARK-41916][FOLLOW-UP] Restore python linter
    omit b9ac53d1fc84 [SPARK-41916][ML] Torch distributor: support multiple torchrun processes per task if task.gpu.amount > 1
    omit 10bc8bcba103 [SPARK-54728][PYTHON][DOCS] Remove a wrong note in `dataframe.isEmpty`
    omit eea141813068 [SPARK-54726][SQL] Improve a bit of performance for InsertAdaptiveSparkPlan
    omit b7fda7cb1128 [SPARK-54405][SQL][METRIC VIEW] CREATE command and SELECT query resolution
    omit 31c38315917b [SPARK-54717][TESTS] Fix npm vulnerabilities by `npm audit fix`
    omit 8a96d59fa810 [SPARK-54706][ML][CONNECT] Make DistributedLDAModel work with local file system
    omit bb112e2089f5 [SPARK-52326][SQL] Add partitions related ExternalCatalogEvent and post them in corresponding operations
    omit 04624f62db0e [SPARK-54707] [SQL] Refactor `PIVOT` resolution main logic to the `PivotTransformer`
    omit 31d996d4f3d8 [SPARK-54453][PYTHON][TEST] Add more coverage tests to conversion
    omit 25c57d87382c [SPARK-54589][PYTHON] Consolidate ArrowStreamAggPandasIterUDFSerializer into ArrowStreamAggPandasUDFSerializer
    omit 716f73fa06be [SPARK-54629][CONNECT][TEST] Supplement getString with BINARY type (UTF-8) test
    omit 86fc601a62a5 [SPARK-54702][INFRA] Recognize existing coverage object to install worker hooks
    omit fb8f5f69bc63 [SPARK-54708][CONNECT][ML] Optimize ML cache cleanup with lazy directory creation
    omit bd46e21a2cb4 [SPARK-53991][SQL][FOLLOWUP] Remove wrong comments added in PR #53463
    omit c36b7e58d042 [SPARK-54696][CONNECT] Clean-up ArrowBuffers in Connect
    omit 2a28bb01ae16 [SPARK-54686][SQL] Relax DSv2 table checks in temp views to allow new top-level columns
    omit cc03f8548317 [SPARK-54632][FOLLOW-UP] Add `ruff` in dev/requirement
    omit 1e14c5c9edb0 [SPARK-54693][CORE][TESTS] Add LZ4TPCDSDataBenchmark
    omit a228c7dcba61 [SPARK-54705][BUILD][TESTS] Upgrade js-yaml to `3.14.2`
    omit 392666f167ba [SPARK-53991][SQL][FOLLOWUP] Enforce KLL_SKETCH_AGG_GET_RANK/QUANTILE arguments are foldable
    omit 48c323aa6e26 [SPARK-54692][DOCS] Add python_worker_logs tvf doc to API reference
    omit 4a18179d6abc [SPARK-54683][SQL] Unify geo and time types blocking
    omit 348164964c1d [SPARK-7008][SDP][ML] Make `org.apache.spark.sql.pipelines` internal package and make `EstimatorUtils` private
    omit 45a2604df52a [SPARK-54695][TEST][CORE] StandaloneDynamicAllocationSuite.syncExecutors should ensure executors have fully setup
    omit 276efe344234 [SPARK-54442][SQL][FOLLOWUP] Add codegen for TIME numeric conversion functions
    omit 43f7936d7b3a [SPARK-53446][CORE] Optimize BlockManager remove operations with cached block mappings
    omit 8446b6d3da36 [SPARK-54687][SQL] Add golden file with generators resolution edge cases
    omit f601aa65c0dc [SPARK-53616][PYTHON] Introduce iterator API for pandas grouped agg UDF
    omit 4c58ecb872f9 [SPARK-54632][INFRA][FOLLOW-UP] Reenable ruff on our CI and lint-python
    omit beeea014f824 [SPARK-54668] [SQL] [TEST-ONLY] Add tests for CTEs in multi-children operators
    omit d57136cf3cf5 [SPARK-54673][SQL] Refactor SQL pipe syntax analysis code for sharing and reuse
    omit ac5fce4e8bb4 [SPARK-51920][SS][PYTHON] Fix type handling of namedTuple for transfromWithState
    omit ca175146970a [SPARK-54643][PYTHON][TESTS][FOLLOW-UP] Restore classic only workflows
    omit bed030cd23cc [SPARK-54669][SQL] Remove redundant casting in rCTEs
    omit 18a9435f28e6 [SPARK-54609][SQL] Disable TIME type by default
    omit d80a0f382d40 [SPARK-52837][PYTHON][FOLLOWUP] Add `versionadded` to `TimeType` doc
    omit 22e26598785e [SPARK-54583][SS] Add SQLConf to enable use of OffsetMap
    omit f43d5cea838a [SPARK-53482][SQL][FOLLOWUP] Rename `spark.sql.merge(.nested.type.coercion.enabled -> NestedTypeCoercion.enabled)`
    omit 2eabf4d08b15 [SPARK-54679][SQL] Rename `spark.sql.(xml.legacyXMLParser.enabled -> legacy.useLegacyXMLParser)`
    omit 9daad92710b4 Revert "[SPARK-54632][INFRA][FOLLOW-UP] Enable ruff on our CI and lint-python"
    omit 6cec3c3647e2 [SPARK-54675][SPARK-54655][SS] Add configurable force shutdown timeout for StateStore maintenance thread pool
    omit 19a1da938a61 [SPARK-54639][PYTHON] Avoid unnecessary Table creation in Arrow serializers
    omit 19ec9915a771 [SPARK-54632][INFRA][FOLLOW-UP] Enable ruff on our CI and lint-python
    omit 7e684b378efa [SPARK-54603][PYTHON][DOCS] Fix typo in contributing.rst
    omit 179ade26c62c [SPARK-54604][PYTHON][DOCS] Fix grammar issues in python_data_source.rst
    omit 7d62da3f8afa [SPARK-54532][PYTHON] Add sqlstate for a set of PySparkException error classes
    omit 8065ba705ce7 [SPARK-54653][DOCS] Add cross-session note to cache/persist public APIs
    omit 0ba4a30d5241 [SPARK-54656][SQL] Refactor SupportsPushDownVariants to be a ScanBuilder mix-in
    omit 6653e00a10c6 [SPARK-54420][SS] Introduce Offline Repartitioning StatePartitionWriter for Single Column Family
    omit 4520c17b19f6 [SPARK-54340][PYTHON][FOLLOW-UP] Add link and examples for `run-with-viztracer`
    omit 070185502e1f [SPARK-54619][PYTHON] Add a sanity check for configuration numbers
    omit e934c43b7c58 [SPARK-54585][SS] Fix State Store rollback when thread is in interrupted state
    omit 7df7dad29a45 [SPARK-53998][TESTS] Add addition E2E tests for RTM
    omit a9fcfb5e0f40 [SPARK-54664][CONNECT] Clean up the code related to `listenerCache` from `connect.StreamingQueryManager`
    omit a20ecc9a28fb [MINOR][DOCS] Fix a typo in `spark-standalone.md`
    omit 8ce3dba73c3e [SPARK-54662][PYTHON] Add `viztracer` and `debugpy` in `dev/requirements.txt`
    omit c09bdb984363 [SPARK-54650][PYTHON][FOLLOW-UP] Apply `pandas.api.types.is_integer_dtype` to check the series dtype
    omit dc47def56265 [SPARK-54403][SQL][METRIC VIEW] Add YAML serde infrastructure for metric views
    omit e09f2a3c0950 [SPARK-54632][PYTHON] Add the option to use ruff for lint
    omit c9165e6908e1 Revert "[SPARK-54450][INFRA] Allow run-tests to accept unittest style string for test names"
    omit 87f6ff898d8e Revert "[SPARK-54450][FOLLOWUP] Relax the restriction of run-tests"
    omit 01662e313ecd [SPARK-54644][PYTHON][SS][TESTS] Always stop the query before await it
    omit 0d9d4ade4226 [SPARK-54441][INFRA][FOLLOW-UP] Enforce new forked process to create its own coverage object
    omit 52ec186a7e72 [MINOR][TESTS] Remove some unnecessary imports in tests
    omit ab9cc0870e96 [SPARK-49635][SQL] Remove ANSI config suggestion in CAST error messages
    omit 8e75fc96af99 [SPARK-54637][CONNECT][TESTS] Add SQL API test helpers to SparkConnectServerTest
    omit 0824934103b1 [SPARK-54643][SDP][PYTHON][TESTS] Run missing pyspark pipelines tests in CI
    omit a189a450e990 [SPARK-54650][PYTHON] Move int to decimal conversion into `_create_converter_from_pandas`
    omit 08c078319fa7 [SPARK-54617][PYTHON][SQL] Enable Arrow Grouped Iter Aggregate UDF registration for SQL
    omit 3dffd12ef594 [SPARK-54438][PYTHON] Consolidate ArrowStreamAggArrowIterUDFSerializer into ArrowStreamAggArrowUDFSerializer
    omit 763106517eca [SPARK-54441][INFRA][FOLLOWUP] Save coverage even when an exception is raised
    omit 18402fd11e67 [SPARK-54572][PYTHON][FOLLOWUP] Rule out some process and add a warning for zip file
    omit a48e2e824641 [SPARK-54636][BUILD][YARN] Correctly relocate Netty native libs for YARN ESS
    omit dc9b1d53b241 [SPARK-54443][SS] Partition key extraction for all streaming stateful operators
    omit 412ce481a667 [SPARK-54646][BUILD] Upgrade `gcs-connector` to 2.2.29
    omit 9bbdc0743034 [SPARK-54645][BUILD] Upgrade Scala to 2.13.18
    omit 4c306ff5162c [SPARK-54581][SQL] Making fetchsize option case-insensitive for Postgres connector
    omit d9d7f1adfd9b [SPARK-54642][BUILD] Upgrade AWS SDK v2 to 2.35.4
    omit 70ee0e1d9f95 [SPARK-54595][SQL] Keep existing behavior of MERGE INTO without SCHEMA EVOLUTION clause
    omit d587eec2e25c [SPARK-54641][BUILD] Upgrade `ammonite` to 3.0.5
    omit fc6f31137634 [SPARK-54640][PYTHON] Replace select.select with select.poll on UNIX
    omit ce9dcabc7089 [SPARK-54638][SQL][TEST] Reuse statement instances if possible in `SparkConnectJdbcDataTypeSuite`
    omit 6a606163ace4 [SPARK-54087][CORE][FOLLOWUP] Spark Executor launch task failed return task killed message should not special handling OOM
    omit ccb039559fd2 [SPARK-54635][BUILD] Remove commons-lang3 from network-shuffle
    omit 4e1e99590a40 [SPARK-54068][PYTHON] Fix `to_feather` to support PyArrow 22.0.0
    omit d4de9131ae31 [SPARK-54615][PYTHON] Always pass runner_conf to python worker
    omit 44efa48dd95e [SPARK-54623][PYTHON][TEST] Add coverage test for UDTF null checker
    omit a1e93307f85e [SPARK-54627][PYTHON] Remove redundant initializations in serializers
    omit 7e89a63b2438 [SPARK-54628][PYTHON] Remove all unnecessary explicit super() arguments
    omit c7338f6501a3 [SPARK-53615][FOLLOWUP][PYTHON][TEST] Fix test case for arrow grouped agg UDF partial consumption
    omit 8fd5f1c73c3e [SPARK-54631][PYTHON] Add profiler support for Arrow Grouped Iter Aggregate UDF
    omit 6bfb83315e45 [SPARK-54622][SQL] Promote `RequiresDistributionAndOrdering` and its required interfaces to `Evolving`
    omit 98058da21e8a [SPARK-54580][SQL] Consider Hive 4.1 in HiveVersionSuite and HiveClientImpl
    omit e6c892397596 [SPARK-54620][SQL] Add safety check in ObservationManager to avoid Observation blocking
    omit e2722b82ca68 [SPARK-54625][SQL] UTF8String#reverse should check offset and length on copying
    omit bbbad56aeec5 [SPARK-54626][BUILD] Upgrade gson to `2.13.2`
    omit 9f3c6a36d824 [SPARK-54624][UI] Ensure user name in historypage get escaped
    omit 59fcc0f97bd5 Update DAGScheduler.scala
    omit 6bf8c172f04a Revert "[SPARK-54595][SQL] Keep existing behavior of MERGE INTO without SCHEMA EVOLUTION clause"
    omit 59e7b3c0dd8f [SPARK-54614][BUILD] Upgrade `tink` to 1.19.0
    omit 1f7e35849419 [SPARK-54613][BUILD] Upgrade `ammonite` to 3.0.4
    omit e8cccbd96cba [SPARK-54607][CORE] Remove unused method `toStringHelper` from `AbstractFetchShuffleBlocks.java`
    omit 3d42605073db [SPARK-54612][BUILD] Upgrade `scala-maven-plugin` to 4.9.7
    omit 7d1ea3dbc655 [SPARK-54392][SS] Optimize JVM-Python communication for TWS initial state
    omit 74b6a931250a [SPARK-54595][SQL] Keep existing behavior of MERGE INTO without SCHEMA EVOLUTION clause
    omit 4baf94d83f51 [SPARK-54618][SQL] Promote `LocalScan` to `Stable`
    omit 102d6eb8d38e [SPARK-54611][BUILD] Upgrade `lz4-java` to 1.10.1
    omit f46f387cc2fb [SPARK-54608][PYTHON] Avoid double caching of type converter in UDTF
    omit 1679aaf28455 [SPARK-51165][DOCS][FOLLOWUP] Fix typo in migration guide
    omit ed6c2a8a88f7 [SPARK-54606][DOCS] Fix `spark.eventLog.rolling.enabled` deafult value
    omit dd418e3b3502 [SPARK-52334][CORE][K8S] update all files, jars, and pyFiles to reference the working directory after they are downloaded
    omit bba8bb86f19a [SPARK-54598][PYTHON] Extract logic to read UDFs
    omit 191ce4c806eb [SPARK-54602][BUILD] Update `NOTICE-binary` with `Netty` `4.2.7.Final` license
    omit 7e53ce8c3647 [SPARK-54450][FOLLOWUP] Relax the restriction of run-tests
    omit f552989381fe [SPARK-54587][PYTHON] Consolidate all runner_conf related code
    omit db256f2a0df4 [SPARK-54591][BUILD] Upgrade `commons-lang3` to 3.20.0
    omit aa65bda31b95 [SPARK-54597][BUILD] Upgrade `lz4-java` to 1.10.0
    omit bc7c76d8308f [SPARK-54563][UI][TEST] Add test to ensure Jetty sanitizes newlines in X-Frame-Options header
    omit eaaa7ffe7339 [SPARK-54600][PYTHON][ML] Don't use pickle to save/load models in `pyspark.ml.connect`
    omit 212d0a7aef2f [SPARK-54330][PYTHON][FOLLOW-UP] Fetch `spark.sql.execution.pandas.inferPandasDictAsMap` with other configs
    omit 38d601076c1a [SPARK-54253][GEO][CONNECT][FOLLOWUP] Enforce geospatial support config in Spark Connect
    omit 081e5c74d5cb [SPARK-54573][SQL][TESTS] Add test for CONSTRAINTS in case of source with less nested struct field than target
    omit 8b5d900e0f78 [SPARK-54578][SQL] Perform Code Cleanup on `AssignmentUtils`
    omit 7bea0b785f10 [SPARK-54594][UI] Skip duplicated edges starting from subqueries in plan dot
    omit 1b163fd31b20 [SPARK-54592][ML] Make `estimatedSize` private
    omit 1e50354c4bbd [SPARK-54562] Block eager analysis / execution inside flow function from the server side
    omit b5bf0a7adabf [SPARK-54572][PYTHON][FOLLOWUP] Add a file lock for the VSCode magic file
    omit ee41857903ee Revert "[SPARK-54310][SQL] Add `numSourceRows` metric for `MergeIntoExec`"
    omit 9969548e717d [SPARK-54570][PYTHON] Propagate the error class correctly in Spark Connect
    omit 74ff452c64f1 [SPARK-54555][PYTHON][TESTS] Set `spark.sql.execution.pandas.structHandlingMode` in pyspark pandas doctest
    omit 0101cf5a94aa [SPARK-54572][PYTHON] Support VSCode breakpoints for pyspark
    omit 4d3e4d605ba0 [SPARK-54316][CORE][PYTHON][SQL] Consolidate `GroupPandasIterUDFSerializer` with `GroupPandasUDFSerializer`
    omit 33d2dc274ed0 [SPARK-54577][PYTHON] Optimize Py4J calls in schema inference
    omit c878dfde3c72 [SPARK-54575][PYTHON][TESTS] Reenable test `SparkConnectCreationTests.test_with_none_and_nan`
    omit 24a341e7e33a [SPARK-54576][SQL] Add documentation for new Datasketches-based aggregate functions
    omit df63cb73215a [SPARK-54388][SS] Introduce StatePartitionReader that scan raw bytes for Single ColFamily
    omit 7a7ea6036eec [SPARK-54564][SQL] Make QueryPlanningTracker as HybridAnalyzer field
    omit 415f50511463 [SPARK-54536][CORE] Shuffle FetchWaitTime missing collect create client/wait cost
    omit 0790fffb918e [SPARK-54551][CONNECT][TESTS][FOLLOWUP] Redirect Test Spark connect server output stream to files
    omit 095c2c3a081f [SPARK-54565][CORE] SparkBuildInfo should load `spark-version-info.properties` from its own classloader
    omit 34d47161f9af [SPARK-54568][PYTHON] Avoid unnecessary pandas conversion in create dataframe from ndarray
    omit bbe831575502 [SPARK-54555][PYTHON][TESTS][FOLLOW-UP] Doc build
    omit 1ea8a8529b30 [SPARK-54555][PYTHON][TESTS][FOLLOW-UP] Skip pyspark-pandas doctest
    omit 7db7f3cbc6c3 [SPARK-54574][ML][CONNECT] Reenable FPGrowth on connect
    omit a00709b613bb [SPARK-54555][PYTHON][TESTS][FOLLOW-UP] Disable test_with_none_and_nan for now
    omit 769ccc62679a [SPARK-54533][UI] Set metric ExecutorSource.METRIC_RESULT_SIZE with correct value
    omit 9d1f4d3e7711 [SPARK-54557][SQL] Make CSV/JSON/XmlOptions and CSV/JSON/XmlInferSchema comparable
    omit 8dd478f90a4d [SPARK-54534][SQL] Migrate Hive-related legacy error codes to proper error conditions
    omit ea0a35e065d3 [SPARK-54555][PYTHON] Enable Arrow-optimized Python UDFs and Arrow-based PySpark IPC by default
    omit 7251e95e22cc [SPARK-53615][PYTHON] Introduce iterator API for Arrow grouped aggregation UDF
    omit d4e34f5087c6 [SPARK-54446][SQL][ML][CONNECT] FPGrowth supports local filesystem with Arrow file format
    omit 250ccca3f2b5 [SPARK-54423][SS] Introducing OffsetMap to enable Named Streaming Sources
    omit 4435a3a7f088 [SPARK-54558][SQL] Fix Internal Exception when Exception Handlers with no BEGIN/END are used
    omit b4c84c917aee [SPARK-54544][PYTHON] Enable flake8 F811 check
    omit 4e2b3155b6c0 [SPARK-54501][SQL] Improve error handling for Hive metastore partition filter failures
    omit 54cde812c265 [SPARK-54481][SPARK-54495][SPARK-54483][TESTS][PYTHON][CONNECT] Reenable a few spark connect tests
    omit aecd932c34d0 [SPARK-54478][SPARK-54479][SPARK-54480][SPARK-54484] Re-enable streaming tests for connect compat test CI
    omit 7833e2fccf9f [SPARK-54424][SQL] Failures during recaching must not fail operations
    omit 8f6c6d646b45 [SPARK-52923][CORE] Allow ShuffleManager to control push merge during shuffle registration
    omit 23d925399424 [SPARK-54525][SQL] Disable nested struct coercion in MERGE INTO under a config
    omit 99270f0b7224 [SPARK-54551][CONNECT][TESTS] Fix random hangs in SparkConnectJdbcDataTypeSuite
    omit 9e9358a309d2 [SPARK-44639][SS][YARN] Use Java tmp dir for local RocksDB state storage on Yarn
    omit f5b9ea8103dd [SPARK-54552][CONNECT] Fix `SparkConnectResultSet.getString` to handle BINARY data type with `UTF_8`
    omit bb36695a6b94 [SPARK-54550][CONNECT] Handle `ConnectException` gracefully in `SparkConnectStatement.close()`
    omit c5cc4ee9ced3 [SPARK-54547][CORE] Rename `hostPort` variable to `host` in `TaskSchedulerImpl.(executorLost|logExecutorLoss)` methods
    omit 743ea2a8f3ec [SPARK-54548][INFRA] Install `zstandard` python library in Spark Connect post-merge builds
    omit 87a8b5629d35 [SPARK-54206][CONNECT][FOLLOWUP] Use VARBINARY type and reasonable max length for BinaryType
    omit 3a06297fb9a6 [SPARK-54454][SQL] Enable variant shredding and variant logical type annotation configs by default
    omit a90b6157fc6f [SPARK-54526][SQL] Rename _LEGACY_ERROR_TEMP_1133 and add sqlState
    omit fe1f7e4ea48c [SPARK-54490][SQL] Classify "withConnection" exceptions in JDBCUtils
    omit a75815e7b7e7 [SPARK-54505][UI][SHUFFLE] Correct the arguments order of createMetrics call in makeNegative
    omit 37104b6b6922 [SPARK-54462][SQL] Add `SupportsV1OverwriteWithSaveAsTable` mixin for `TableProvider`
    omit 0a6fff18db6b [SPARK-54492][SQL] Rename _LEGACY_ERROR_TEMP_1201 and add sqlState
    omit 830ac653dd2c [SPARK-54299][SQL] Fix the wrong example query in `WindowGroupLimit`
    omit 26ab773478b9 [SPARK-53614][PYTHON][TESTS][FOLLOW-UP] Fix the maxBytes config in tests
    omit 68653991a4dd [SPARK-54540][CONNECT] Couples of minor fix for Connect JDBC driver
    omit 0a76a2138286 [SPARK-54537][CONNECT] Fix SparkConnectDatabaseMetaData getSchemas/getTables on catalogs missing namespace capbalitity
    omit 5b0a667d9a5c [SPARK-54522][SQL][TEST] Variant test suite fixes for shredding configs
    omit 154a2708e988 [SPARK-54531][PYTHON] Introduce ArrowStreamAggPandasUDFSerializer
    omit 9b2d7a4cd17b [SPARK-54532][PYTHON] Add support for sqlstate for PySparkException
    omit a8482ad2a725 [SPARK-54272][SQL] Add aggTime for SortAggregateExec
    omit e9f390da288a [SPARK-54223][PYTHON] Add task context and data metrics to Python runner logs
    omit 8940dad7e33d [SPARK-53469][SQL] Ability to cleanup shuffle in Thrift server
    omit 161ed3d18dc3 [SPARK-46166][PS] Implementation of pandas.DataFrame.any with axis=1
    omit a3e55e5dc8dd [SPARK-53597][PYTHON][SQL][DOCS] Add `asTable()` and `TableArg` examples
    omit e2649651d78d [SPARK-54528][CONNECT] Close URLClassLoader eagerly to avoid OOM
    omit c11edf9e8d21 [SPARK-54517][PYTHON][TESTS][FOLLOWUP] Fix typehints using ellipsis
    omit 0908ec5272e7 [SPARK-54529][BUILD] Upgrade `commons-codec` to 1.20.0
    omit b9dd02baed3e [SPARK-54524][BUILD] Fix Connect JDBC driver dependencies
    omit d36bd625332c [SPARK-54523][SQL] Fix default resolution during variant pushdown
    omit d9dcb5c4dbb5 [SPARK-54517][PYTHON][TESTS] Added utility decorators for Spark Connect parity tests
    omit 9feb1b2c0220 [SPARK-54496][SQL] Fix Merge Into Schema Evolution for Dataframe API
    omit 91ca9cb89b59 [SPARK-54493][PYTHON] Fix assertSchemaEqual for MapType
    omit c022e0e1655f [SPARK-54519][DOCS] Reformat tutorial comments so it's easier to read on the website
    omit 78e911826d64 [SPARK-54520][INFRA] Ignore Missing Packages in CI free_disk_space Step
    omit 27a484983440 [SPARK-54482][PYTHON][CONNECT][TESTS] Reeanble `test_apply_in_pandas_returning_column_names_*` for Spark Connect 4.0 client <> master server
    omit 64c9d89dddcd [SPARK-54503][PYTHON][DOCS] Include `pyspark.pipelines` in API reference
    omit c61d40c88f7b [SPARK-54508][PYTHON] Fix `spark-pipelines` to resolve `spec` file path more robustly
    omit 754eda2c9c28 [SPARK-54504][SQL] Fix version refresh for DSv2 tables with subqueries
    omit d73b019b1be6 [SPARK-54475][INFRA][FOLLOW-UP] Make master-server <> branch-4.0-client a scheduled workflow
    omit 3095eee55626 [SPARK-54500][PYTHON][TESTS] Restore `pythonUDF.arrow.concurrency.level` in ArrowPythonUDFLegacyTests
    omit 38bc351fcbdd [SPARK-54497][PYTHON] Apply `functools.lru_cache` in converter caching
    omit 08273487ec3e [SPARK-54444][SQL] Relax DSv2 table checks to restore previous behavior
    omit 211dd995b221 [SPARK-54130][SQL][FOLLOWUP] Refactor tests to use withEmptyCatalog for consistency
    omit 7191a14d25e2 [SPARK-54450][INFRA] Allow run-tests to accept unittest style string for test names
    omit c6e8dbe23193 [SPARK-54247][PYTHON][FOLLOWUP] Ensure socket is not leaked and add regression tests
    omit 7c0b39248477 [SPARK-54116][SQL] Add off-heap mode support for LongHashedRelation
    omit 8f747eef9a38 [SPARK-54477][CONNECT][PYTHON][TESTS] Mark tests to reeanble for 4.0 client <> master server job
    omit f5a7389e0ad1 [SPARK-54475][PYTHON] Add master-server, branch-4.0-client, Python 3.11
    omit 3f5a2b9fc334 [SPARK-54491][SQL] Fix insert into temp view on DSv2 table failure
    omit 8efec56331d9 [SPARK-54221][FOLLOW-UP] Add `pypy3.11` in `Build Pipeline Status`
    omit ef83255e4749 [SPARK-54247][PYTHON] Explictly close socket for util._load_from_socket
    omit 2316411a19bc [SPARK-54219][CORE] Support `spark.cleaner.referenceTracking.blocking.timeout` config
    omit 7917e2d77c72 [SPARK-54397][PYTHON] Make `UserDefinedType` hashable
    omit 21b61660bcd0 [SPARK-54488][BUILD] Fix Connect JVM client leaks `protobuf-java-util` in shading rules
    omit 5a546decf715 [SPARK-54456][PYTHON] Import worker module after fork to avoid deadlock
    omit 99f3af8d57a2 [SPARK-54473][SQL] Add Avro read and write support for TIME type
    omit 9846dd826e16 [SPARK-54289][SQL][FOLLOW-UP] Make Merge Into update assignment by field default for UPDATE SET * and align configs
    omit d14209c6ffba [SPARK-54115][TESTS][FOLLOW-UP] Refine `org.apache.spark.util.UtilsSuite`
    omit 0ba9a9bd8d6d [SPARK-54474][PYTHON] Discard the XML report on tests that are supposed to fail
    omit 92c948f41376 [SPARK-54022][SQL] Make DSv2 table resolution aware of cached tables
    omit 4d1c79fe8f13 [SPARK-54472][SQL] Add ORC read and write support for TIME type
    omit 0fa72f72b359 [SPARK-54425][INFRA] Add two missing test directories to coverage omit
    omit 4506f2fe9bff [SPARK-54458][INFRA] Use official download-artifact to fix the test_report action
    omit 021f89ae3d13 [SPARK-54471][INFRA] Remove `build_python_3.11_macos.yml` CI
    omit 5f1d62472f03 [SPARK-54470][CORE][TESTS] Fix `BlockManagerDecommissionIntegrationSuite` to wait shuffle migrations
    omit cda51211f8d4 [SPARK-54469][DSTREAM][TESTS] Fix `StreamingContextSuite.stop slow receiver gracefully` test to clean up `SprakContext`
    omit 997525cae9b0 [SPARK-54452] Fix empty response from SparkConnect server for `spark.sql(...)` inside FlowFunction
    omit 913b162f81ea [SPARK-54467][INFRA][FOLLOWUP] Disable `actions/cache` on `python_hosted_runner_test.yml` too
    omit 3c50fda6f29d [SPARK-54467][INFRA] Disable `actions/cache` on MacOS CIs
    omit 3861ba498f6d [SPARK-50072][SQL] Handle ArithmeticException in interval parsing with large values
    omit 50b7d56eb2c2 [SPARK-54465][INFRA] Remove `build_python_connect35.yml` GitHub Action job
    omit 7f5478c7d389 [SPARK-54463][SQL] Add CSV serialization and deserialization support for TIME type
    omit 8182bc3e0015 [SPARK-54464][SQL] Remove duplicate `output.reserve` calls in `assembleVariantBatch`
    omit 12bce6b7f8f5 [SPARK-54412][SQL][TESTS] Clean up `v` properly in `identifier-clause.sql` SQL golden file
    omit 5ad630bb5614 [SPARK-54364][INFRA][FOLLOWUP] Disable pyspark_install on apache master commits
    omit f5a554067413 [SPARK-54363][REPL][TESTS] Access to buffer in SparkShellSuite should be synchronized between threads
    omit ea74e57b260d [SPARK-53635][SQL] Support Scala UDFs with input args of type Seq[Row]
    omit d16b688637c5 [SPARK-54461][SQL] Add XML serialization and deserialization support for TIME type
    omit 01fa49b9755f [SPARK-54447][BUILD] Upgrade icu4j to 78.1
    omit 09678f7eba37 [SPARK-54460][SQL][TESTS] Mark `StateStoreInstanceMetricSuite` as `ExtendedSQLTest`
    omit 8cfd78d95fa2 [SPARK-54459][SQL][TESTS] Increase `StateStoreSuite.maintenance` test timeout to 2 minutes
    omit 966e0539c67b [SPARK-54289][SQL] Allow MERGE INTO to preserve existing struct fields for UPDATE SET * when source struct has less nested fields than target struct
    omit 371b00ad4225 [SPARK-54455][CORE][TESTS] Disable a flaky `unmanaged memory tracking with off-heap memory enabled` test in MacOS CI
    omit e6fa02ee3b8d [SPARK-51765][DOCS] Document `SQL Scripting` Syntax
    omit 9b0b1ce2d628 [SPARK-54153][PYTHON][TESTS][FOLLOWUP] Skip `test_perf_profiler_data_source` if `pyarrow` is absent
    omit 05602d57d977 [SPARK-54439][SQL] KeyGroupedPartitioning and join key size mismatch
    omit 8aafae744dcc [SPARK-54451][SQL] Add JSON serialization and deserialization support for TIME type
    omit 0d99d6d852cd [SPARK-54427][SQL] Allow ColumnarRow to call `copy` with variant types
    omit a870ea459712 [SPARK-53797][SQL] Fix `FileStreamSource.takeFilesUntilMax` to use `zipWithIndex` to avoid `indices` usage
    omit 89d2f3675012 [SPARK-54395][SHUFFLE] RemoteBlockPushResolver class repeatedly initializes ObjectMapper
    omit 7c279e54257d [SPARK-54087][CORE] Spark Executor launch task failed should return task killed message
    omit da7389bb81ab [SPARK-54410][SQL] Fix read support for the variant logical type annotation
    omit 8c9d9269ba4f [SPARK-54389][SS] Fix RocksDB State Store Invalid Stamp error when task is marked as failed during initialization
    omit 6c6dd0f5c7fe [SPARK-54448][SQL][DOCS] Fix docs and error message because CREATE TEMP VIEW does not support IF NOT EXISTS
    omit ac69d935ddac [SPARK-54354][SQL] Fix Spark hanging when there's not enough JVM heap memory for broadcast hashed relation
    omit 464ef622b086 [SPARK-54442][SQL] Add numeric conversion functions for TIME type
    omit dc3964a52f3b [SPARK-54324] Add test for client-user-context-extensions
    omit b6a1eb8861e0 [SPARK-54445][SQL][TESTS] Fix configuration related to `ThresholdFilter` in `log4j2.properties` within `hive-thriftserver` module
    omit 00dda2c8d03c [MINOR][PYTHON][TEST] test_install_spark` switch to Spark 3.5.7
    omit 8eae3d7e4171 [SPARK-54364][INFRA] Separate test_install_spark from the rest of the tests
    omit 1fdc139112b0 [SPARK-54330][PYTHON][FOLLOW-UP] Further avoid unnecessary py4j calls
    omit 59a2a7474e81 [SPARK-54441][INFRA] Enable coverage data for workers
    omit d3346eb6dba3 [SPARK-54440][SDP] Give default pipeline spec file more idiomatic name, `spark-pipeline.yml`
    omit 6578b9b96e41 [SPARK-51771][SQL][FOLLOWUP] Rename currentVersion to version in DSv2 Table
    omit 4ac02ab225f3 [SPARK-54418][SDP] Fix error messages and code formatting
    omit 98e36730c405 [SPARK-54063][SS][FOLLOWUP] Remove `withTempDirFix` from `forceSnapshot metric populated when shouldForceSnapshotOnCommit is true` test
    omit 8cab0745e404 [SPARK-54436][SQL] Fix error formatting for incompatible table metadata checks
    omit 6811eabeb147 [SPARK-54435][SDP] spark-pipelines init should avoid overwriting existing directory
    omit 3dcb661df7c0 [SPARK-54417][SQL] Fix error message for scalar subquery in IDENTIFIER clause
    omit d1af9a305718 [SPARK-54300][SPARK-54396][PYTHON][FOLLOW-UP] Pass `prefers_large_var_types` to `_collect_as_arrow`
    omit e3533a62ca0a [SPARK-49133][CORE] Make member `MemoryConsumer#used` atomic to avoid user code causing deadlock
    omit 18291ea0253a [SPARK-54413][UI] Upgrade Bootstrap v4.4.1 to v4.6.2
    omit d51b4331e419 [SPARK-54312][CORE] Avoid repeatedly scheduling tasks for SendHeartbeat/WorkDirClean in standalone worker
    omit 10d1b4c38359 [SPARK-54358][SDP][FOLLOWUP] Add `tableIdentifierToPathString` helper method for SDP checkpoint path construction
    omit 9efd2eee79b8 [MINOR][SQL] Remove unused method from Observation
    omit be281fba3a1b [SPARK-54426][INFRA] Fix `release-build.sh` to detect `REPO_ID` correctly
    omit 0e2e34937305 [SPARK-54425][INFRA] Use .coveragerc for omitting sources and ignore test files for coverage
    omit f328b5ef14c9 [SPARK-54416][DOCS] Remove references to Pod Security Policies from Kubernetes docs
    omit 2168df7467ef [SPARK-54422][K8S] Increase `spark.kubernetes.allocation.batch.size` to 20
    omit 158a132e6289 [SPARK-54399][GEO][SQL][PYTHON] Implement the st_setsrid function in Scala and PySpark
    omit ee0f69245ecd [SPARK-54151][GEO][PYTHON][FOLLOWUP] Update examples for ST functions in PySpark
    omit 61668ad02671 [SPARK-54030][SQL][FOLLOWUP] Refactor test to use withBasicCatalog for consistency
    omit d29968498a5d [SPARK-54253][GEO][SQL] Add a guarding config for geospatial support
    omit c5468b906692 [SPARK-54373][UI] Enlarge the SVG viewBox attribute of Job DAG initialization
    omit 5fb072e4f25f [SPARK-54285][PYTHON] Cache timezone info to avoid expensive timestamp conversion
    omit d823ccfc3c5d [SPARK-44571][SQL][FOLLOWUP][DOCS] Update Scaladoc of `MergeSubplans`
    omit 1aa8d5aaa11d [SPARK-54375][CONNECT][TESTS][FOLLOWUP] Make `PythonPipelineSuite` perform a default check for `PythonTestDepsChecker.isConnectDepsAvailable`
    omit dff06206e211 [SPARK-53322][SQL] Select a KeyGroupedShuffleSpec only when join key positions can be fully pushed down
    omit a6fda315f8e1 [MINOR][PYTHON][DOCS] Refine the examples in window.py
    omit 2d7e9a77db80 [SPARK-54396][PYTHON] Optimize Py4J calls in Dataframe.toArrow
    omit cd1601bb8b1b [SPARK-54346][SS] Introduce state repartition API and repartition runner
    omit 6227fbab0308 [SPARK-54349][PYTHON] Refactor code a bit to simplify faulthandler integration extension
    omit 894a7e8993ed [SPARK-54400][SQL] Replace `HtmlUtils.parseQueryString` in	`ThriftHttpServlet.java` with `HttpServletRequest.getQueryString`
    omit 13fea4fa02c6 [SPARK-54383][SQL] Add precomputed schema variant for InternalRowComparableWrapper util
    omit 1012a5ffa51b [SPARK-54163][SQL] Scan canonicalization for partitioning and ordering info
    omit 05bc5d408fa9 [SPARK-54350][SQL][STS] SparkGetColumnsOperation ORDINAL_POSITION should be 1-based
    omit dce992b7708b [SPARK-54387][SQL] Fix recaching of DSv2 tables
    omit aa387f32158a [SPARK-52767][SQL] Optimize maxRows and maxRowsPerPartition for join and union
    omit 78fcc934d314 [SPARK-44571][SQL] Merge subplans with one row result
    omit e8f0a67e248d [SPARK-54377][SQL] Fix COMMENT ON TABLE IS NULL to properly remove table comment
    omit b03352b99bb9 [SPARK-54394][CORE] Move `isJavaVersionAtMost17` and `isJavaVersionAtLeast21` from `core` to `common/utils`
    omit fd683ce6dbd3 [SPARK-53924][FOLLOWUP][TESTS] Add tests for cached temp view detecting schema changes
    omit 35a99f84a352 [SPARK-54393][BUILD][TESTS] Upgrade oracle free to 23.26.0
    omit b7127004358a [SPARK-54317][PYTHON][CONNECT] Unify Arrow conversion logic for Classic and Connect toPandas
    omit 78d1d52601e4 [SPARK-54339][SQL] Fix AttributeMap non-determinism
    omit a9c1fba603a2 [SPARK-54381][BUILD] Upgrade Junit to 6.0.1
    omit 1c283c01493a [SPARK-54063][SS] Trigger snapshot for next batch when upload lag
    omit 19ff85950c35 [SPARK-54379] [SQL] Move lambda binding to separate `LambdaBinder` object
    omit ed23cc39c32e [SPARK-54344][PYTHON] Kill the worker if flush fails in daemon.py
    omit 21150230f455 [SPARK-54384][PYTHON] Modernize the _batched method for BatchedSerializer
    omit 1db267e3bd02 [SPARK-54376][SDP] Mark most pipeline configuration options as internal
    omit 5222f1a0e0a6 [SPARK-54378][SQL] Remove `CreateXmlParser.scala` from `catalyst` module
    omit 722bcc0f0d15 [SPARK-54375][CONNECT][TESTS] Add `assume` to cases in `PythonPipelineSuite` to skip tests when PyConnect dependencies is not available
    omit 3757091e1c51 [SPARK-54319][SQL] BHJ LeftAnti update numOutputRows wrong when codegen is disabled
    omit 96e0d4b94247 [SPARK-54352][SQL] Introduce `SQLConf.canonicalize` to centralize normalization of strings
    omit 0a42f557a8d2 [SPARK-54370][INFRA] Limit the Maven GitHub Action job timeout to 150 minutes
    omit 13545a4e5078 [SPARK-54369][CONNECT][TESTS] Fix `PythonPipelineSuite` flakiness via `Set` instead of `Seq`
    omit 2e4708ed10b7 [SPARK-54371][INFRA] Fix `spark-rm` Dockefile to install `pkgdown` version at the end
    omit 407e79c0ac52 [SPARK-53924] Reload DSv2 tables in views created using plans on each view access
    omit 0311f44e33e5 [SPARK-54366][INFRA] Add `free_disk_space` step to K8s integration test GitHub Action job
    omit d02a6d490f1f [SPARK-54368][INFRA] Remove `generate-llms-txt.py` from `release-build.sh`
    omit e3e986373735 [SPARK-54355][CONNECT] Make `spark.connect.session.planCompression.defaultAlgorithm` to support `NONE`
    omit e09c99949107 [SPARK-54358][SDP] Checkpoint dirs collide when streaming tables in different schemas have same name
    omit cc72c647b6f9 [SPARK-54020] Support `spark.sql(...)` Python API inside query functions for Spark Declarative Pipeline
    omit bb089184426d [SPARK-54361][SQL] Fix Spark version to intended value of 4.2.0 for spark.sql.parser.singleCharacterPipeOperator.enabled
    omit c5c65d2bf358 [SPARK-54292][SQL] Support aggregate functions and GROUP BY in |> SELECT pipe operators
    omit 1214830dc16d [SPARK-53666][FOLLOWUP] Update generated llms.txt file location
    omit cf513611bf79 [SPARK-50906][SQL] Fix Avro nullability check for reordered struct fields
    omit e739b7d349db [SPARK-53337][UI] XSS: Ensure the application name in historypage get escaped
    omit dc5bb9641b6e [SPARK-54172][SQL][FOLLOW-UP] Simplify attribute re-resolution and validate action assignment value resolution for schema evolution
    omit 32a3244c45da [SPARK-51518][SQL] Support | as an alternative to |> for the SQL pipe operator token
    omit a916690d6101 [SPARK-54348][INFRA] Recover Python unit tests CI by installing `zstandard==0.25.0`
    omit 053617c4ba1e [SPARK-54347][PYTHON] Optimize Py4J calls in classic dataframe
    omit 7599b2ffb28a [SPARK-54356][SDP] Fix EndToEndAPISuite caused by missing storage root schema
    omit 13e2765eb7ff [SPARK-54343][DOCS] Docs for parameter markers, identifier improvements, string coalescing
    omit 59759f3646cf [SPARK-54157][SQL] Fix refresh of DSv2 tables in Dataset
    omit be98a29875ce [SPARK-54114][CONNECT] Support getColumns for SparkConnectDatabaseMetaData
    omit 83d25b9ae588 [SPARK-54351][BUILD] Upgrade Dropwizard metrics to 4.2.37
    omit 6cb88c10126b [SPARK-54194][CONNECT][FOLLOWUP] Spark Connect Proto Plan Compression - Scala Client
    omit 551b922a53ac [SPARK-54330][PYTHON] Optimize Py4J calls in `spark.createDataFrame`
    omit 9f1bd47bab15 [SPARK-54332][PYTHON][CONNECT] No need to attach PlanId in grouping column names in rollup/cube/groupingSets
    omit 116240c39507 [SPARK-54341][SQL] Remember TimeTravelSpec for tables loaded via TableProvider
    omit 87b3b9423243 [SPARK-54209][CONNECT] [Followup] Fix linter error in `SparkConnectJdbcDataTypeSuite`
    omit 9f1dc1ccf8b7 [SPARK-54209][CONNECT] Support TIMESTAMP type in SparkConnectResultSet
    omit 4020794c719e [SPARK-54280][SDP] Require pipeline checkpoint storage dir to be absolute path
    omit e162b9b0007b [SPARK-54340][PYTHON] Add the capability to use viztracer on pyspark daemon/workers
    omit 7a6d3e18b132 [SPARK-54338][INFRA] Enable `pyspark-pandas` in `build_coverage.yml`
    omit 2a6e6e55cea4 [SPARK-54208][CONNECT] Support TIME type in SparkConnectResultSet
    omit 602a4bd94881 [SPARK-54331][PYTHON][TESTS] Optimize `pyspark.sql.tests.connect.test_connect_plan`
    omit 06b3d4c4e716 [SPARK-54336][SQL] Fix `BloomFilterMightContain` input type check with `ScalarSubqueryReference`
    omit 5e46f4c5fec4 [SPARK-54333][BUILD] Upgrade `commons-io` to 2.21.0
    omit 840349ee51c8 [SPARK-54334][SQL] Move the validation of subquery expressions under lambda and higher order functions to `SubqueryExpressionInLambdaOrHigherOrderFunctionValidator`
    omit 63bcc871bdec [SPARK-54194][PYTHON][FOLLOWUP] Fix `connectutils.py` to import `pb2` conditionally
    omit aad6c51d27ad [SPARK-54182][SQL][PYTHON] Optimize non-arrow conversion of `df.toPandas`
    omit 674ed486781f [SPARK-54318][PYTHON][DOCS] Fix doctests in `pyspark.sql.dataframe`
    omit d2188059d195 [SPARK-54329][INFRA] Use shorter prefix before build name
    omit daa29f6ede29 [SPARK-53573][SQL] IDENTIFIER everywhere
    omit 5c16a73c61c3 [SPARK-54326][INFRA] Recover MacOS CIs by installing `zstandard==0.25.0`
    omit 65b69a0d9fd5 [SPARK-54315][PYTHON][TESTS] Optimize test `ApplyInArrowTests.test_arrow_batch_slicing`
    omit 75fdec3d9a84 [SPARK-54310][SQL] Add `numSourceRows` metric for `MergeIntoExec`
    omit fb8dbd7071b1 [SPARK-54323][PYTHON] Change the way to access logs to TVF instead of system view
    omit 05b054315b84 [SPARK-54240] Translate get array item catalyst expression to connector expression
    omit 4eb56bb65419 [MINOR][DOCS] Document minimum executor memory requirement
    omit ecaec3d1b013 [SPARK-54206][CONNECT] Support BINARY type data in SparkConnectResultSet
    omit 03eb023c3a99 [SPARK-54172][SQL] Merge Into Schema Evolution should only add referenced columns
    omit 1a802e36ed76 [SPARK-54320][UI] Fix Job DAG overlapping
    omit 5270c997b56c [SPARK-54306] Annotate Variant columns with Variant logical type annotation
    omit 346d80379208 [SPARK-54301][SQL][TESTS] Enhance Spark SQL test suites for easier integration with other projects
    omit 40ba971b7319 [SPARK-53917][CONNECT] Support large local relations - follow-ups
    omit 0c715b4de87f [SPARK-54130][SQL] Add detailed error messages for catalog assertion failures
    omit 361d0c9cc571 [SPARK-54183][PYTHON][CONNECT] Avoid one intermediate temp data frame during spark connect toPandas()
    omit a8ca6f4ba099 [SPARK-54207][CONNECT] Supports Date type data in SparkConnectResultSet
    omit a3a8d320581a [SPARK-54113][CONNECT] Support getTables for SparkConnectDatabaseMetaData
    omit 88671ca265ce [SPARK-54307][SS] Throw an error if streaming query is restarted with stateful op but there is empty state dir
    omit ca9c2565c34c [SPARK-54300][PYTHON] Optimize Py4J calls in `df.toPandas`
    omit c21d5a45073f [SPARK-54270][CONNECT] SparkConnectResultSet get* methods should call checkOpen and check index boundary
    omit 03a0e05acfcf [SPARK-52439][SQL] Support check constraint with null value
    omit f3f1449a9aac [SPARK-54297][PYTHON][SQL][TESTS] Optimize tests for iterator of dataframe API in `applyInPandas`
    omit 7ebf5556d9e7 [SPARK-54303][SQL] Canonicalize error condition MISSING_CATALOG_ABILITY
    omit 63efbec843be [SPARK-54298][PYTHON][DOCS] Fix doctest of UDFRegistration
    omit 908b8a324e34 [SPARK-54296][PYTHON][TESTS] Optimize the arrow batch slicing tests for cogrouped
    omit 067678a90392 [SPARK-54295][INFRA] Remove redundant ASFLv2 full-text in `NOTICE-binary`
    omit 8f9385a8b9c0 [SPARK-54294][CONNECT] Normalize Connect server printed IP address
    omit 32f3a29e21c5 [SPARK-54264][SS] DeDup Operator can use RocksDB's keyExists()
    omit 811482a79cbc [SPARK-54290][SS] Skip checksum creation if path already exists without checksum
    omit c3b102bf3ea4 [SPARK-54304][SQL][TESTS] MergeScalarSubqueriesSuite code cleanup
    omit 9ff0fba06e75 [SPARK-53448][PYTHON] Conversion of a pyspark DataFrame with a Variant column to pandas fails with an error
    omit 9a37c3d59b8e [SPARK-53898][CORE] Fix race conditions between query cancellation and task completion triggered by eager shuffle cleanup
    omit 0ccaacf54ce3 [SPARK-54194][CONNECT] Spark Connect Proto Plan Compression
    omit 2d16baea577c [SPARK-54291][BUILD] Upgrade Jackson to 2.20.1
    omit d4bc277690f2 [SPARK-54173][K8S] Add support for Deployment API on K8s
    omit 67547efe6b7b [SPARK-54287][PYTHON] Add Python 3.14 support in `pyspark-client` and `pyspark-connect`
    omit e7b3ba50b7c2 [SPARK-54288][INFRA] Add a daily Python 3.14 CI for branch-4.1
    omit 52fe51b6e9b3 [SPARK-54112][CONNECT] Support getSchemas for SparkConnectDatabaseMetaData
    omit dbc643ee84f8 [SPARK-54278][PYTHON][TESTS] Simplify `pyspark/util.py` doctests by removing Python 3.9+ condition
    omit c477ee6bd3c7 [SPARK-54275][SQL] Clean up  unused code from `pipeline` module
    omit acfedacd564b [SPARK-54277][INFRA] Make `dev/run-tests` to ban `Python 3.9` and older versions
    omit e4cf867ed464 [SPARK-54273][UI][TESTS] Upgrade jest to 30.0.0
    omit fc49dbd868e0 [SPARK-54211][PYTHON][FOLLOW-UP] Fix doctests of mapInArrow
    omit 21c31226c3f7 Revert "[SPARK-54218][PYTHON][SQL][TESTS] Add integrated tests for Scalar Pandas Iterator UDF" Again
    omit 8cc76fe19606 [SPARK-54065][PYTHON][TESTS][FOLLOWUP] Remove unused import module from `test_python_datasource.py`
    omit 25e80534df4e [SPARK-54218][PYTHON][SQL][TESTS] Add integrated tests for Scalar Pandas Iterator UDF
    omit 904a5048b057 [SPARK-54271][UI][BUILD] Upgrade node version for UI test module to the latest LTS 24.11.0
    omit 6211f7be5913 [SPARK-54259][CORE] Downgrade "Shutdown hook called" messages from INFO to DEBUG
    omit 79d0c427878f [SPARK-54065][PYTHON][TESTS] Enable `test_in_memory_data_source` in Python 3.14
    omit 9754e1750614 [SPARK-54269][PYTHON] Upgrade `cloudpickle` to 3.1.2 for Python 3.14
    omit 020c1bd2b635 [SPARK-54226][SQL][FOLLOWUP] Remove zstd compression level value check and update config names
    omit 28052dccd662 [SPARK-54267][BUILD] Upgrade actions/setup-node@v4 to v6
    omit 73b50e9163a1 [SPARK-54205][CONNECT] Supports Decimal type data in SparkConnectResultSet
    omit 6466ed579d5a [SPARK-54263][INFRA] Fix `run-in-container` script to use Python 3.11 instead of 3.9
    omit 5c11509d4ef3 [SPARK-49862][INFRA][FOLLOWUP] Update `dev/infra/Dockerfile` too
    omit 5a454f86cd5f [SPARK-54260][CORE][K8S] Unify K8s cluster checks to use the single regex via `SparkMasterRegex.isK8s` methods
    omit cc57743a229a [SPARK-53944][K8S][CORE][FOLLOWUP] Override DRIVER_HOST_ADDRESS in SparkContext instead of SparkEnv when useDriverPodIP is true
    omit 9289c4349ed3 [SPARK-54239][BUILD] Bump Guava 33.4.8
    omit 6a369f953b0d [SPARK-54153][PYTHON] Support profiling iterator based Python UDFs
    omit d304a0313ff1 [SPARK-54166][GEO][PYTHON] Introduce type encoders for geospatial types in PySpark
    omit 9b68d3c92b4c [SPARK-54244][GEO][SQL] Introduce type coercion support for GEOMETRY data types
    omit b14496589149 [SPARK-54246][PYTHON][DOCS] Add the user guide for python worker logging
    omit f2d594de6195 [SPARK-54243][SQL] Introduce type coercion support for GEOGRAPHY data types
    omit a32559a3284e [SPARK-54046][INFRA] Upgrade PyArrow to 22.0.0
    omit d16b128d8215 [SPARK-54245][INFRA] Install `mlflow` at Python 3.14 Docker image
    omit 4ec6668a688c [MINOR][INFRA] Fix the install version comment to Python 3.10 in python-minimum Dockerfile
    omit 1a724ba6feab [SPARK-54191][SDP] Add once to Defineflow Proto
    omit a871ba4464e0 [SPARK-54136][SQL] Extract plan merging logic from `MergeScalarSubqueries` to `PlanMerger`
    omit 178f0f428020 [SPARK-53823][SS] Implement allow list for real time mode
    omit f9191248900c [SPARK-54232][GEO][CONNECT] Enable Arrow serialization for Geography and Geometry types
    omit bab98afd490b [SPARK-54231][SDP] Fill gaps in SDP docs
    omit cc7f5124057f [SPARK-54229][PYTHON] Make PySparkLogger in UDFs store one log entry per log function call
    omit c04e095779f4 [SPARK-54220][SQL] NullType/VOID/UNKNOWN Type Support in Parquet
    omit 7f81bd7436fe [SPARK-54242][BUILD] Skip `Checkstyle` if `NOLINT_ON_COMPILE` is true
    omit 050d3978d332 [SPARK-54241][INFRA] Enable `NOLINT_ON_COMPILE` for all PySpark GitHub Action jobs
    omit 16fd02321cd2 [SPARK-54238][CORE][TESTS] Clean up unused `private val regenerateCommand` from `SparkThrowableSuite`
    omit 37997cef6dff [SPARK-54234][PYTHON][CONNECT] Not need to attach PlanId in grouping column names in df.groupBy
    omit 1f7bbeba3a79 [SPARK-53525][CONNECT][FOLLOWUP] Allow io.grpc classes in Spark Connect testing
    omit 7bc3e75c03e9 [SPARK-54202][GEO][SQL] Allow casting from GeometryType(srid) to GeometryType(ANY)
    omit 96ed48db3879 [SPARK-54226][SQL] Extend Arrow compression to Pandas UDF
    omit 0cc2319044a9 [SPARK-54235][UI][DOCS] Upgrade jQuery from 3.5.1 to 3.7.1
    omit a8d128c7acaa [SPARK-54190][BUILD] Guava dependency governance
    omit c69e999f82e1 [SPARK-54117][SS] Throw better error to indicate that TWS is only supported with RocksDB state store provider
    omit a8a4a7743e17 [SPARK-54233][PYTHON][DOCS] Fix the doctest of cogrouped applyInArrow and applyInPandas
    omit 6da54d9a1a42 [SPARK-53666][INFRA][FOLLOWUP] Use absolute output path for `generate-llms-txt.py`
    omit 81be5fb02970 Revert "[SPARK-54218][PYTHON][SQL][TESTS] Add integrated tests for Scalar Pandas Iterator UDF"
    omit 47a8d2a2ce58 [SPARK-54230][CORE][SQL][TESTS] Clean up the usage of deprecated selenium API
    omit 941c24bbebe2 [SPARK-54201][GEO][SQL] Allow casting from GeographyType(srid) to GeographyType(ANY)
    omit 7a1b4812ac7a [SPARK-54227][INFRA] Upload test reports generated by the surefire plug-in too
    omit 9894abcca68a [MINOR][TESTS][DOCS] Make bfill/backfill tests not dependent on natural order
    omit 87da039dce8f [SPARK-54218][PYTHON][SQL][TESTS] Add integrated tests for Scalar Pandas Iterator UDF
    omit 76afa1cf2731 [SPARK-54221][INFRA] Add PyPy 3.11 GitHub Action job
    omit 2f04e783b7e3 [MINOR][DOCS][CONNECT] Replace pyspark with pyspark-client in the Connect overview
    omit f191d652f039 [SPARK-54199][SQL] Add DataFrame API support for new KLL quantiles sketch functions
    omit 37b19950eb6c [SPARK-53944][K8S][CORE][FOLLOWUP] Fix SparkEnv to use `DRIVER_BIND_ADDRESS` in case of `useDriverPodIP=true`
    omit a927a14a59e2 [SPARK-52463][SDP] Add support for cluster_by in Python Pipelines APIs
    omit d6f1e224df8f [SPARK-54169][GEO][SQL] Introduce Geography and Geometry types in Arrow writer
    omit 2ec7439f51f3 [SPARK-54198][K8S] Delete Kubernetes executor pods only once per event processing interval
    omit b3abaecf30ec [SPARK-54177][BUILD][FOLLOWUP] Restore Guava exclusion from connect-common shading artifacts
    omit 2f9581781ff6 [SPARK-54200][SS] Call close() against underlying InputPartition when LowLatencyReaderWrap.close() is called
    omit 852daf595540 [SPARK-54215][SQL] Add the SessionStateHelper trait to FilePartition
    omit 148d552ecdf6 [SPARK-54212][PYTHON][DOCS] Fix the doctest of PandasGroupedOpsMixin.applyInArrow
    omit b284a2c50e96 [SPARK-54101][GEO][FOLLOWUP] Re-enable Scala/Python parity check for ST functions
    omit 5a480912d7ae [SPARK-54214][INFRA] Increase pyspark job execution time to 150 minutes on MacOS
    omit 4ba1189dabc4 [SPARK-54213][PYTHON][CONNECT] Remove Python 3.9 from Spark Connect
    omit f9fadf2d59c6 [SPARK-54211][PYTHON][DOCS] Update exmple of mapInArrow to use arrow builtin function
    omit aed30de8174d [SPARK-54187][PYTHON][CONNECT] Get all configs in a batch in toPandas
    omit 1b442c4fa688 [SPARK-54204][INFRA] Run `build_branch40*.yml` every two days
    omit 4173bee455c3 [SPARK-54192][BUILD] Upgrade roaringbitmap from 1.3.0 to 1.5.3
    omit 37689bfea45f [SPARK-54159][INFRA] Fix `pip install` commands to use proper quotation marks
    omit 3b368ca9715c [SPARK-54197][K8S] Improve `ExecutorsPodsLifecycleManager` not to request to delete if `deletionTimestamp` exists
    omit bc58b6ec27f9 [SPARK-54078][SS] New test for `StateStoreSuite SPARK-40492: maintenance before unload` and remove infra from old test
    omit 40a512624a21 [SPARK-54178][SQL] Improve error for ResolveSQLOnFile
    omit 6b91d83787f3 [SPARK-54160][GEO][SQL] Implement the ST_SetSrid expression in SQL
    omit e7b90e7a97fe [SPARK-54176][GEO][PYTHON] Introduce Geography and Geometry data types to PySpark Connect
    omit 23c006c3abdb [SPARK-54156][PROTOBUF] Classify errors for ProtobufOptions casting failure
    omit 1af7d0defd7b [SPARK-54180][SQL] Override the toString of BinaryFileFormat
    omit 0a034a0bac89 [SPARK-54193][SHUFFLE] Deprecated spark.shuffle.server.finalizeShuffleMergeThreadsPercent
    omit 07cab004409c [SPARK-54014][CONNECT] Support max rows for SparkConnectStatement
    omit 20af57c1340a [SPARK-54147][SQL] Set OMP_NUM_THREADS to spark.task.cpus by default in BaseScriptTransformationExec
    omit 3f41adcc579e [SPARK-54186][PYTHON][TESTS] Fix doctests for `PandasCogroupedOps.applyInPandas`
    omit 8b1ee1caea86 [SPARK-54189][K8S] Remove `spark.kubernetes.pyspark.pythonVersion`
    omit 1ec4b71568ec [SPARK-54185][SHUFFLE] Deprecated spark.shuffle.server.chunkFetchHandlerThreadsPercent
    omit ada19082cfe7 [SPARK-54184][K8S] Support `spark.kubernetes.executor.deletedExecutorsCacheTimeout`
    omit fecb2f8f42c4 [SPARK-53595][DOCS][PYTHON][SQL] Add see-also for XML functions
    omit a2f0c8375ab1 [SPARK-54188][K8S] Improve `ExecutorPodsWatchSnapshotSource` to watch only active executors
    omit 84a3fa07ed5f [SPARK-54162][GEO][SQL] Allow casting from GeographyType to GeometryType
    omit a8e35c407bc5 [SPARK-54106][SS] Recheckin State store row checksum implementation
    omit fb3b7ce78fbe [SPARK-54181][SQL][TESTS] Fix typo `withConvertMetastore`
    omit 83e49b7aca35 [SPARK-54177][BUILD] Upgrade gRPC to 1.76 and protobuf to 6.33
    omit 4966fe99d3e3 [SPARK-53967][PYTHON] Avoid intermediate pandas dataframe creation in `df.toPandas`
    omit 9de0a27396d0 [SPARK-53731][PYTHON] Update the type hints of iterator APIs
    omit 32060bf99fc8 [SPARK-54175][GEO][CONNECT] Add Geography and Geometry types to Spark Connect proto
    omit e3a43ba665c0 [SPARK-53991][SQL] Add SQL support for KLL quantiles functions based on DataSketches
    omit 8100636ab870 [SPARK-38498][DSTREAM] Support customized StreamingListener by configuration
    omit afc45f08c0de [SPARK-54121][SS] Automatic Snapshot Repair for State store
    omit a1428d40970f [SPARK-54056][SQL] resolve substitution for SQLConf settings in Catalogs
    omit b722b1c0d6f0 [SPARK-54170][CORE] Use StructuredLogging message in Scala side
    omit 031af163646e [SPARK-54149][SQL][SS][CONNECT] Enable tail-recursion wherever possible
    omit 4a1b7b282aa2 Revert "[SPARK-54177][BUILD] Upgrade gRPC to 1.76 and protobuf to 6.33"
    omit e55333c9506b [SPARK-54177][BUILD] Upgrade gRPC to 1.76 and protobuf to 6.33
    omit 8c76795fa958 [SPARK-54142][GEO][SQL][PYTHON] Implement the st_srid function in Scala and PySpark
    omit 78e9f70dd6e3 [SPARK-54174][CORE][CONNECT][TESTS] Remove override of deprecated method `SparkThrowable#getErrorClass` in Spark code
    omit bdac5cd192af [SPARK-54115][CORE][UI] Escalate display ordering priority of connect server operation threads in thread dump page
    omit 7df66f9aae11 [MINOR][PYTHON][TESTS] Enable `test_pivot_exceed_max_values`
    omit ff455377f4b8 [SPARK-54161][BUILD] Upgrade `extra-enforcer-rules` to 1.11.0
    omit c8cb3e7506c4 [SPARK-54049][BUILD] Shade com.google.thirdparty package to fix Guava class conflicts in spark 4.0
    omit c8d8cf29b3ef [SPARK-54164][INFRA] Add `branch-4.1` to daily `Publish Snapshot` GitHub Action job
    omit d8f2f95a2bb2 [SPARK-53666][DOCS] Add script to generate llms.txt file for Spark main website
    omit c4ad38117e4b [SPARK-53128][CORE] Include unmanaged memory bytes in the usage log before execution memory OOM
    omit 94e00ca8f5fc [SPARK-54145][PYTHON][CONNECT] Fix column check of nested type in numeric aggregation
    omit 34c3dde17df2 [SPARK-54155][BUILD][TESTS] Upgrade `mssql-jdbc` to `13.2.1.jre11`
    omit 02ba89f02993 [SPARK-54134][SQL] Optimize Arrow memory usage
    omit be6ab9cfafbe [SPARK-54138][SQL] Enforce constant configuration parameter for Theta and HLL
    omit 24110b6f6b1b [SPARK-54151][GEO][PYTHON] Introduce the framework for adding ST functions in PySpark
    omit fc3a590d9847 [SPARK-54131][PYTHON][TESTS] Update `Pandas` version 2.3.3
    omit 2f9431340937 [SPARK-54110][GEO][SQL] Introduce type encoders for Geography and Geometry types
    omit a5e866f7ff57 [SPARK-54132][SQL][TESTS] Cover HashedRelation#close in HashedRelationSuite
    omit daa83fc209da [SPARK-53525][CONNECT][FOLLOWUP] Spark Connect ArrowBatch Result Chunking - Scala Client
    omit 6b8814906650 [SPARK-54154][INFRA] Run `build_branch35*.yml` every two days
    omit 46e7bfb58141 [SPARK-54146][CORE][SQL] Clean up the usage of deprecated Jackson API
    omit edfd3578d586 [SPARK-54108][CONNECT] Revise execute* methods of SparkConnectStatement
    omit 2be1eb72e1ca [SPARK-54111][CONNECT] Support getCatalogs for SparkConnectDatabaseMetaData
    omit f5bd5d09dfd3 [SPARK-54148][INFRA] Add `libwebp-dev` to all `dev/spark-test-image/*/Dockerfile`
    omit 6e4936d0fe8f [SPARK-54144][PYTHON] Short Circuit Eval Type Inferences
    omit a830dd251979 [SPARK-54128][CONNECT][SQL] Convert IllegalArgumentException to SparkException with proper error classes in Spark Connect server
    omit 3d92bdf6ac16 [SPARK-54127][BUILD] Fix sbt inconsistent shading package
    omit 2063c365860b [MINOR][PYTHON][DOCS] Fix the doctest of `pivot`
    omit 544eccc6a7f6 [SPARK-54143][INFRA] Add `branch-4.1` Daily GitHub Action jobs
    omit d69b9c4e0301 [SPARK-54079][GEO][SQL][FOLLOWUP] Fix catalyst expressions ST package and imports
    omit e7b7acfd44f0 [SPARK-39328][SQL][TESTS] Fix flaky test `SPARK-37753: Inhibit broadcast in left outer join when there are many empty partitions on outer/left side`
    omit 689851eae71a [SPARK-54141][INFRA] Add `libwebp-dev` to fix `dev/infra/Dockerfile` building
    omit 88eef06c221d [SPARK-54099][SQL] XML variant parser should fall back to string on decimal parsing errors
    omit 25aa49a3cdfb [SPARK-54140][INFRA] Add `libwebp-dev` to fix `dev/spark-test-image/lint/Dockerfile` building
    omit e837ff9707ef [SPARK-54123][PYTHON] Add timezone to make the timestamp an absolute time
    omit cce32a8876bd [SPARK-54052][PYTHON] Add a bridge object to workaround Py4J limitation
    omit 0ec382b1382b [SPARK-54133][BUILD] Setting version to 4.2.0-SNAPSHOT
    omit 477e6e890e2d Revert "[SPARK-54106][SS] State store row checksum implementation"
    omit 0732e44977ea [SPARK-53809][SQL] Add canonicalization for DataSourceV2ScanRelation
    omit 928f25386c35 [SPARK-54027] Kafka Source RTM support
    omit 7dd973d87cb1 [SPARK-54118][SS] Improve the put/merge operation in ListState when there are multiple values
    omit ac717dd7aec6 [SPARK-54106][SS] State store row checksum implementation
    omit 3d292dc7b1c5 [SPARK-54028][SQL] Use empty schema when altering a view which is not Hive compatible
    omit 6d5f111cc86a [SPARK-51972][SS] Introduce State Store file integrity verification using checksum
    omit 610bb52242c7 [SPARK-54104][GEO][SQL] Disallow casting geospatial types to/from other data types
    omit 6e41ac5d7015 [SPARK-53978][PYTHON] Support logging in driver-side workers
    omit 7624476c18bf [SPARK-54109][PYTHON][TESTS] Avoid function conflicts in test_pandas_grouped_map
    omit f2c81cdd6fb0 [SPARK-54101][GEO][SQL] Introduce the framework for adding ST functions in Scala
    omit aea64e580cf1 [SPARK-54039][SS] Add TaskContext information to KafkaDataConsumer release() logs for better debugging
    omit 06fe1ab21265 [SPARK-54096][GEO][PYTHON] Support Spatial Reference System mapping in PySpark
    omit 4047efddd8f6 [SPARK-54033][GEO][SQL][FOLLOWUP] Fix Java test suite names for geospatial execution classes
    omit 653e6995b110 [SPARK-54103][GEO][SQL] Introduce client-side Geography and Geometry classes
    omit 4202f239c45a [SPARK-53614][PYTHON] Add `Iterator[pandas.DataFrame]` support to `applyInPandas`
    omit 1eae69306450 [SPARK-54091][GEO][SQL] Implement the ST_Srid expression in SQL
    omit 9fae65bb220e [SPARK-53959][CONNECT] Throw a client-side error when creating a dataframe from a pandas dataframe with an index but no data
    omit 79f3d74175d2 [SPARK-53977][PYTHON] Support logging in UDTFs
    omit adc160168af8 [SPARK-53891][SQL][FOLLOW-UP] Clarify javadocs for case when merge summary metric is not found
    omit ec6122f63b12 [SPARK-54100][BUILD] Remove `ignore.symbol.file` Javac option
    omit 7c5a9a322038 [SPARK-54094][SQL] Extract common methods to KafkaOffsetReaderBase
    omit 1d575f2ae9ba [SPARK-54067][CORE] Improve `SparkSubmit` to invoke `exitFn` with the root cause instead of `SparkUserAppException`
    omit f37cd07534e4 [SPARK-53976][PYTHON] Support logging in Pandas/Arrow UDFs
    omit aead8224ae01 [SPARK-54078][SS] Deflake StateStoreSuite `SPARK-40492: maintenance before unload`
    omit 146c2baebd84 [SPARK-54085][CORE] Fix `initialize` to add `CREATE` option additionally in `DriverRunner`
    omit 26aa2e49f8c4 [SPARK-53535][SQL][TESTS][FOLLOWUP] Remove duplicated tests
    omit 84882fa060bd [SPARK-54030][SQL] Add user-friendly assertion message for view metadata corruption
    omit 52103e8ef9d1 [SPARK-53921][GEO][PYTHON][FOLLOWUP] Add GeographyType and GeometryType to `__all__` in types
    omit 067969ff9467 [SPARK-54038][CONNECT] Support getSQLKeywords for SparkConnectDatabaseMetaData
    omit 166b190fee3d [SPARK-54061] Wrap IllegalArgumentException with proper error code for invalid datetime patterns
    omit 58e5768a09b7 [SPARK-53482][SQL] MERGE INTO support for when source has less nested field than target
    omit 56eaa8d2ac8c [SPARK-54079][GEO][SQL] Introduce the framework for adding ST expressions in Catalyst
    omit 1368983b58f1 [SPARK-52134][SQL][FOLLOW UP] SQL Script execution code - refactor follow up
    omit 27a0d27b7acb [SPARK-54029][SQL] Add detailed error message for table metadata corruption to ease debugging
    omit 4cf7772a84fa [SPARK-53573][SQL] Allow coalescing string literals everywhere
    omit e991caf1672f [SPARK-54072][SS] Ensure that we don't upload empty files in RocksDB snapshot
    omit efd066e5394a [SPARK-54002][DEPLOY] Support integrating BeeLine with Connect JDBC driver
    omit 665a428cb6cd [SPARK-53905][SQL] Refactor RelationResolution to enable code reuse
    omit 571b802db743 [SPARK-54075][SQL] Make `ResolvedCollation` evaluable
    omit 576f9a515976 [SPARK-54076][SQL][TESTS] Use `sarutak/oracle-free` image to use `PASSWORD_INIT_TIMEOUT` for `OracleDatabaseOnDocker`
    omit 7d114e32a2c1 [SPARK-54074][SS][TESTS] Remove `kafka.metadata.max.age.ms` from an unavailable offset relevant test in `KafkaMicroBatchV1SourceWithConsumerSuite`
    omit 9b7ab21c9ef0 [SPARK-53975][PYTHON] Adds basic Python worker logging support
    omit 68928757c209 [SPARK-54062][SQL] MergeScalarSubqueries code cleanup
    omit 19408689a2bd [SPARK-51658][GEO][SQL] Introduce Geometry and Geography in-memory wrapper formats
    omit 40885ea19555 [SPARK-53957][GEO][SQL] Support GEOGRAPHY and GEOMETRY in the SpatialReferenceSystemMapper
    omit 803ea9590145 [SPARK-54041][SQL] Refactor ParameterizedQuery arguments validation
    omit 029393a0ef30 [SPARK-54054][CONNECT] Support row position for SparkConnectResultSet
    omit 30ccf8dae736 [SPARK-53921][GEO][PYTHON][FOLLOWUP] Fix invalid algorithm error for Geography JSON parsing
    omit 8860509a65bd [SPARK-53610][PYTHON][TESTS][FOLLOW-UP] Add tests for negative maxRecordsPerBatch
    omit d14c3558405f [SPARK-54069][PYTHON][TESTS] Skip `test_to_feather` in Python 3.14
    omit 3b10407fbf7e [SPARK-54066][PYTHON][TESTS] Skip `test_in_memory_data_source` in Python 3.14
    omit cb857243662a [SPARK-54051][INFRA][PYTHON] Keep coverage data when running pip tests
    omit 3773e3fd7890 [SPARK-54047][PYTHON] Use a difference error when kill-on-idle-timeout
    omit 9f3d905ad671 [SPARK-53941][FOLLOWUP][SS] Introduce a SQL config to control AQE for stateless streaming workload specifically
    omit face10652a00 [SPARK-54058][SS][TESTS] Increase `kafka.metadata.max.age.ms` from `1` to `500` to make a test in `KafkaMicroBatchV1SourceWithConsumerSuite` stable
    omit 4622e8584e37 [SPARK-52510][GEO][CONNECT][FOLLOWUP] Reserve expression proto numbers for Geometry and Geography
    omit 317773a5fdd5 [SPARK-54033][GEO][SQL] Introduce Catalyst server-side geospatial execution classes
    omit b4d88beb65f5 [SPARK-54059][CORE] Reduce default page size by LONG_ARRAY_OFFSET if ZGC or ShenandoahGC and ON_HEAP are used
    omit 5f83b892b5d1 [SPARK-54021][GEO][SQL] Implement Geography and Geometry accessors across Catalyst
    omit a8a81eb5edf3 [SPARK-54057][BUILD] Retrieve dependency version from `pom.xml` for `DependencyOverrides` in `SparkBuild.scala`
    omit 65b466c241a6 [SPARK-54031][SQL] Add new golden file tests for analysis edge-cases
    omit 7ac2b5423362 [SPARK-53535][SQL][FOLLOWUP] Fix findCheapestGroupField returning invalid Map
    omit 27180f0413ca [SPARK-54013][CONNECT] Implement SparkConnectDatabaseMetaData simple methods
    omit 036f591448ec [SPARK-54050][PYTHON][DOCS] Update the documents of arrow-batching related configures
    omit b6a98131956c [SPARK-53835][INFRA] Install `pyarrow/torch/torchvision` packages to `Python 3.14` Dockefile
    omit 89167c0ec1d4 [SPARK-53915] Add RealTimeScanExec and ability to execute long running batches
    omit 0393b5ebb327 [SPARK-54048][PYTHON] Update `dev/requirements.txt` to install `torch(vision)` in Python 3.14
    omit 1716b292dec2 [SPARK-53732][SQL] Remember TimeTravelSpec in DataSourceV2Relation
    omit eaacb360ec1f [SPARK-54040][BUILD] Remove unused commons-collections 3.x
    omit eac033884a04 [SPARK-53962][BUILD] Upgrade ASM to 9.9
    omit ec944cc52c8b [SPARK-53659][SQL] Infer Variant shredding schema when writing to Parquet
    omit 65755bde3cb8 [SPARK-54036][INFRA] Add `build_python_3.11_macos26.yml` GitHub Action job
    omit 9c9af608ac0f [SPARK-54032][CORE] Prefer to use native Netty transports by default
    omit c79662dcdf5b [MINOR][DOCS] Fix MacOS 26 CI link in README.md
    omit e2b107196f59 [SPARK-54034][CORE] Fix `Utils.isBindCollision` to detect port conflict `NativeIoException` correctly
    omit 4a937f899726 [SPARK-54018][K8S][INFRA][DOCS] Upgrade Volcano to 1.13.0
    omit e42fede87a3b [MINOR][TESTS] skip two duplicate tests related to mapping BinaryType to bytes in PySpark
    omit 1824dd3821bf [SPARK-53406][SQL] Avoid unnecessary shuffle join in direct passthrough shuffle id
    omit c5b58b50e142 [SPARK-54023][CORE] Support `AUTO` IO Mode
    omit 06674bab1db3 [SPARK-53942][SS] Support changing stateless shuffle partitions upon restart of streaming query
    omit a1f3dcb628eb [SPARK-52509][CORE] Cleanup individual shuffles from fallback storage on `RemoveShuffle` event
    omit b7f2ab25989e [SPARK-53335][K8S] Support `spark.kubernetes.driver.annotateExitException`
    omit 45c0a469890e [SPARK-53941][SS] Support AQE in stateless streaming workloads
    omit bcb0817904ac [SPARK-52798][SQL][TESTS][FOLLOWUP] Fix a unit test failure due to the ANSI assumption
    omit 85b3e220b1d0 [SPARK-54008] Skip QO for DESCRIBE QUERY
    omit 2fdf5744cec5 [SPARK-53980][CORE] Add `SparkConf.getAllWithPrefix(String, String => K)` API
    omit c9db27934bbb [SPARK-54015][PYTHON] Relax Py4J requirement to py4j>=0.10.9.7,<0.10.9.10
    omit bb34bc6a847f [SPARK-54011][CORE] Switch direct ctors to IoHandlerFactories for EventLoopGroups
    omit a94e709dcbab [SPARK-53986][PYTHON][SS][TESTS] Reorganize Python streaming TWS test
    omit 2b2a2a2e18e7 [SPARK-54004][SQL] Fix uncaching table by name without cascading
    omit 5059bab37f71 [SPARK-54007][CORE][SQL] Use Java `Set.of` instead of `Collections.emptySet`
    omit 6d3c35c88ae7 [SPARK-54005][CORE][TESTS] Remove no-op `spark.shuffle.blockTransferService` configuration
    omit 7bd18e3852f1 [SPARK-53610][PYTHON] Limit Arrow batch sizes in CoGrouped applyInPandas and applyInArrow
    omit 3e1c3bdc7ee1 [SPARK-54009][CORE] Support `spark.io.mode.default`
    omit a0cc8ccfb12c [SPARK-53880][SQL] Fix DSv2 in PushVariantIntoScan by adding SupportsPushDownVariants
    omit 56661c2ad195 [SPARK-53891][SQL] Model DSV2 Commit Write Summary API
    omit b46cf270abab [SPARK-53324][K8S][FOLLOWUP] Use `invokePrivate` to access `numOutstandingPods`
    omit 9b465a2a9adf [SPARK-53324][K8S] Introduce pending pod limit per ResourceProfile
    omit 1005e00177f7 [MINOR][CORE] Remove invalid `NioBlockTransferService` comment
    omit 8c9fad7bb77b [SPARK-53934][CONNECT] Initial implement Connect JDBC driver
    omit 59b8a4489c87 [SPARK-54001][SQL] Optimize memory usage in session cloning with ref-counted cached local relations
    omit cf5c7b684cec [SPARK-53966][CORE][FOLLOWUP] Support `is(Serial|Parallel)GC`
    omit 92ac08ca4ac6 [SPARK-50205][SQL][TEST] Re-enable `SparkSessionJobTaggingAndCancellationSuite.Cancellation APIs in SparkSession are isolated`
    omit c9ae4ab7ead2 [SPARK-53999][CORE] Native KQueue Transport support on BSD/MacOS
    omit f72435a87320 [SPARK-53966][CORE] Add utility functions to detect JVM GCs
    omit fcbafc3fbff1 [SPARK-53914][BUILD][FOLLOWUP] Fix branch-4.0 daily maven test
    omit 76d4718d2258 [SPARK-53920][GEO][SQL] Introduce GeometryType and GeographyType to Java API
    omit 96093bdacba1 [SPARK-53922][GEO][SQL] Introduce physical Geometry and Geography types
    omit c70c728eb1ea [SPARK-53981][BUILD] Upgrade Netty to 4.2.7.Final
    omit c707f597e9c4 Add PipelineAnalysisContext message to support pipeline analysis during Spark Connect query execution
    omit 0e10341eec9a [SPARK-53930][PYTHON] Support TIME in the make_timestamp function in PySpark
    omit a6d17f71f8d0 [SPARK-53956][PYTHON] Support TIME in the try_make_timestamp function in PySpark
    omit 0be5f96e7a7d [SPARK-53972][SS] Fix streaming query recentProgress regression in classic pyspark
    omit d65ed4a1d573 [SPARK-53687][SQL][SS][SDP] Introduce WATERMARK clause in SQL statement
    omit 4a62f7509c50 [SPARK-53319][SQL] Support the time type by try_make_timestamp_ltz()
    omit db81309dd80a [SPARK-53917][CONNECT] Support large local relations
    omit a96e9ca81518 [SPARK-53968][SQL] Store decimal precision loss conf in arithmetic expressions
    omit 3109488d1f5a [SPARK-53974][BUILD] Bump Jackson 2.20.0
    omit 7427ff46db45 [SPARK-53979][PYTHON][TESTS] Drop temporary functions in Pandas UDF tests
    omit 3d18fe1927f1 [SPARK-53973][AVRO] Classify errors for AvroOptions boolean casting failure
    omit 41d3619338a0 [SPARK-53921][GEO][PYTHON] Introduce GeometryType and GeographyType to PySpark API
    omit e58a3db23eb5 [SPARK-53954][BUILD] Bump Avro 1.12.1
    omit 748de5f3710e [SPARK-53969][PYTHON][TESTS] Drop temporary functions in Arrow UDF tests
    omit 6a77ec491436 [SPARK-53971][BUILD] Bump zstd-jni 1.5.7-6
    omit a8cfe0c682c5 [SPARK-53960][SQL] Let approx_top_k_accumulate/combine/estimate handle NULLs
    omit 0f4263205ccb [SPARK-53636][CORE] Fix thread-safety issue in SortShuffleManager.unregisterShuffle
    omit ed9957b00f9d [SPARK-53964][BUILD] Simplify Java Home finding for SBT unidoc
    omit 85bf6ec8a3f7 [SPARK-53958][BUILD] Simplify Jackson deps management by using BOM
    omit e963eb72e29e [SPARK-53965][CONNECT][SS][PYTHON] Upgrade buf plugins to `v29.5`
    omit f33d8aa90630 [SPARK-53738][SQL] Fix planned write when query output contains foldable orderings
    omit 9f325422459d [SPARK-53963][PYTHON][TESTS] Drop temporary functions in regular UDF tests
    omit 128fb1369493 [SPARK-53914][BUILD][CONNECT] Add connect-client-jdbc module
    omit 6739e4f98c86 [MINOR][PYTHON][TESTS] Update `test_arrow_udf_output_timestamps_ltz` to check the timezone conversion
    omit 8430dbfd9482 [SPARK-53961][SQL][TESTS] Fix `FileStreamSinkSuite` flakiness by using `walkFileTree` instead of `walk`
    omit 94cccad9e3f2 [SPARK-53755][CORE] Add log support in BlockManager
    omit 37ee99256b76 [SPARK-53535][SQL] Fix missing structs always being assumed as nulls
    omit 1f21a8b35872 [SPARK-53951][BUILD] Upgrade `protobuf-java` to 4.33.0
    omit b0327f37bfb0 [SPARK-53947][SQL] Count null in approx_top_k
    omit 57b4cd2ccc64 [SPARK-53696][PYTHON][CONNECT][SQL] Default to bytes for BinaryType in PySpark
    omit 9c40c126b799 [SPARK-53946][BUILD] Upgrade SBT to 1.11.7
    omit 2bb73fbdeb19 [SPARK-53950][BUILD] Upgrade scala-xml to 2.4.0
    omit b1f54285c907 [SPARK-42857][PYTHON][TESTS] Enable parity test `test_supported_types`
    omit 4ab7d5a850ce [SPARK-53949][CONNECT] Use `Utils.getRootCause` instead of `Throwables.getRootCause`
    omit 9c38696848e1 [SPARK-53945][BUILD] Upgrade `semanticdb-shared` to `4.13.10`
    omit 5ae573f176cd [SPARK-53656][SS] Refactor MemoryStream to use SparkSession instead of SQLContext
    omit 5edebd2e2bf0 [SPARK-53938][PYTHON][CONNECT] Fix decimal rescaling in LocalDataToArrowConversion
    omit dd8d13b69e8e [SPARK-53944][K8S] Support `spark.kubernetes.executor.useDriverPodIP`
    omit 2c7bc89aadb3 [SPARK-53760][GEO][SQL][FOLLOWUP] Fix error message and comment for Spatial types
    omit 1c0bca91d270 [SPARK-52798][SQL] Add function approx_top_k_combine
    omit 8499a62fb6b1 [SPARK-53785][SS] Memory Source for RTM
    omit c0416713e8e1 [SPARK-53943][PYTHON][DOCS] Add examples for function unwrap_udt
    omit 2f574594acfc [MINOR][PYTHON][DOCS] Fix the docstring of `assert_true`
    omit d799aa7354b0 [SPARK-53149][CORE] Fix testing whether BeeLine process run in background
    omit 136201ab4e75 [SPARK-53927][BUILD][DSTREAM] Upgrade kinesis client and fix kinesis integration tests
    omit f689ff75a037 [SPARK-53939][PYTHON] Use batch.num_columns instead of len(batch.columns)
    omit 983d384222d3 [SPARK-53573][SQL] Use Pre-processor for generalized parameter marker handling
    omit eb117a642bcd [SPARK-53908][CONNECT] Fix observations on Spark Connect with plan cache
    omit ea71991e07a0 [SPARK-53936][BUILD] Upgrade sbt-pom-reader from 2.4.0 to 2.5.0
    omit 24a6abf34d25 [SPARK-53935][BUILD] SBT assembly should handle META-INF correctly
    omit 83be7e7cfea4 [SPARK-53929][SQL] Support TIME in the make_timestamp and try_make_timestamp functions in Scala
    omit deb7b62189f7 [SPARK-53925][FOLLOW-UP][DOCS] Fix the link in README
    omit 9823daf2cf90 [SPARK-53931][INFRA][PYTHON] Fix scheduled job for numpy 2.1.3
    omit 61a024c1dd49 [SPARK-53916][PYTHON] Deduplicate the variables in PythonArrowInput
    omit 032dcf89c19b [SPARK-53926][DOCS] Document newly added `core` module configurations
    omit bc8020fd4278 [SPARK-53919][BUILD] Make Maven plugins up-to-date
    omit 2ae81cd9495b [SPARK-53925][INFRA] Use `MacOS 26` in `build_maven_java21_macos15.yml`
    omit fcd83713889e [SPARK-53923][CORE] Rename `spark.executor.(log -> logs).redirectConsoleOutputs`
    omit b0285f8bbf82 [SPARK-53902][SQL] Add tree node pattern bits for supported expressions in `ParameterizedQuery` argument list
    omit 9e14f5f959d4 [SPARK-53111][SQL][PYTHON][CONNECT] Implement the time_diff function in PySpark
    omit 17d15a4c6cc7 [SPARK-53762][SQL] Add date and time conversions simplifier rule to optimizer
    omit abe785379335 [SPARK-53789][SQL][CONNECT] Canonicalize error condition CANNOT_MODIFY_STATIC_CONFIG
    omit 98010b3d10dd [SPARK-53913][DOCS] Document newly added K8s configurations
    omit e3e69821af92 [SPARK-53867][PYTHON][FOLLOW-UP] Fix `pa.concat_batches` for old pyarrow versions
    omit 8ff9feb31f08 [SPARK-53760][GEO][SQL] Introduce GeometryType and GeographyType
    omit 694cc72a71fb [SPARK-53895][SS][TESTS] Add `ContinuousMemorySuite`
    omit 2d220645087e [SPARK-53897][CONNECT][TESTS] Add dependency checks for Python-related tests in the `connect` module
    omit 9ae819840bbf [SPARK-53611][PYTHON] Limit Arrow batch sizes in window agg UDFs
    omit 4eacd0b6e924 [SPARK-53907][K8S] Support `spark.kubernetes.allocation.maximum`
    omit e05c75e105d5 [SPARK-53906][K8S] Protect `ExecutorPodsAllocator.numOutstandingPods` as `protected val`
    omit ff0f1ab95238 [SPARK-53900][CONNECT] Fix unintentional `Thread.wait(0)` under rare conditions in `ExecuteGrpcResponseSender`
    omit 800309e3a463 [SPARK-53857] Enable messageTemplate propagation to SparkThrowable
    omit cba28ea9783b [SPARK-53841][PYTHON][CONNECT] Implement `transform()` in Column API
    omit 85c9fd18f200 [SPARK-53867][PYTHON] Limit Arrow batch sizes in SQL_GROUPED_AGG_ARROW_UDF
    omit 37564db05c03 [SPARK-51426][PYTHON][SQL] Fix 'Setting metadata to empty dict does not work'
    omit e6a76dfc0f00 [SPARK-53896][CORE] Enable `spark.io.compression.lzf.parallel.enabled` by default
    omit 1e515d323097 [MINOR][DOCS] Fix 404 for Python Package Management link in rdd-programming-guide.md
    omit 6f0f5873e348 [SPARK-53894][BUILD][TESTS] Upgrade `docker-java` to 3.6.0
    omit 8898ec9fadd8 [SPARK-53893][TESTS] Regenerate benchmark results after upgrading to Scala 2.13.17
    omit 7d028c69311c [SPARK-53609][PYTHON] Limit Arrow batch sizes in SQL_GROUPED_AGG_PANDAS_UDF
    omit 710c6071e8a3 [SPARK-53892][SS] Use `DescribeTopicsResult.allTopicNames` instead of the deprecated `all` API
    omit 0cb933e31fad Revert "[MINOR][PYTHON][TESTS] Retry `test_observe_with_map_type`"
    omit 6bae835ccbc8 [SPARK-53720][SQL] Simplify extracting Table from DataSourceV2Relation (#52460)
    omit f92816cc8507 [SPARK-53878][SQL][CONNECT] Fix race condition issue related to ObservedMetrics
    omit d72e6a30a37c [SPARK-53884][BUILD] Upgrade `ZSTD-JNI` to 1.5.7-5
    omit e38a65180564 [SPARK-53845] SDP Sinks
    omit 56f8b3b2cb97 [SPARK-53877] Introduce BITMAP_AND_AGG function
    omit 343a25b72b22 [SPARK-53868][SQL] Use array length check instead of direct reference check in V2ExpressionBuilder
    omit 264ca4dc320e [SPARK-53455][CONNECT] Add `CloneSession` RPC
    omit 3ac4a48fb8c5 [SPARK-53585][BUILD] Upgrade Scala to 2.13.17
    omit 00d2a5403e45 [SPARK-53881][BUILD][TESTS] Upgrade `Selenium` to 4.32.0
    omit 47e4108ef7bb [SPARK-53879] Upgrade `Ammonite` to 3.0.3
    omit 3f663bf58313 [SPARK-53870][PYTHON][SS] Fix partial read bug for large proto messages in TransformWithStateInPySparkStateServer
    omit 81d9d1f82850 [SPARK-53847] Add ContinuousMemorySink for Real-time Mode testing
    omit 9aab260c2b3c [SPARK-53868][SQL] Only use signature with Expression[]  of `visitAggregateFunction` in V2ExpressionSQBuilder
    omit 418cf56ed4dc [SPARK-53796][SDP] Add `extension` field to a few pipeline protos to support forward compatibility
    omit 72fc87b46734 [SPARK-53690][SS] Fix exponential formatting of avgOffsetsBehindLatest in Kafka sources object in progress json
    omit a35c9f36da63 [SPARK-53805][SQL] Push Variant into DSv2 scan
    omit 0702d58074c5 [SPARK-53822][PYTHON][SS][TESTS] Add Python TransformWithState test case for composite/nested output schemas
    omit 6eb4d3c9d38f [SPARK-53865][SQL] Extract common logic from ResolveGenerate rule
    omit b6160682b7f4 [SPARK-53864][BUILD] Upgrade `commons-lang3` to 3.19.0
    omit 474731f6b445 [SPARK-53866][PYTHON][TESTS] Skip doctest `pyspark.sql.pandas.functions` without pyarrow/pandas
    omit 320d09d2461d [SPARK-53605][INFRA] Restore pyspark execution timeout to 2 hours
    omit ed1140f25241 [SPARK-53786][SQL] Default value with special column name should not conflict with real column
    omit 6032a401616c [SPARK-53779][SQL][CONNECT] Implement `transform()` in Column API
    omit cd23ad55b9f5 [SPARK-53856][CORE] Remove `blacklist` alternative config names
    omit f041118ff1f5 [SPARK-53861][PYTHON][INFRA] Factor out streaming tests from `pyspark-sql` and `pyspark-connect`
    omit ea0f4faba374 [MINOR][DOCS] Add `build_python_3.14.yml` to `Build Pipeline Status` section of `README.md`
    omit ef1eec4c6d9f [SPARK-53858][PYTHON][TESTS] Skip doctests in `pyspark.sql.functions.builtin` if pyarrow is not installed
    omit cd6d701b0877 [SPARK-53859][BUILD][TESTS] Upgrade `JUnit` to 6.0.0
    omit dc11b66bdd9c [MINOR][PYTHON][TESTS] Retry `test_observe_with_map_type`
    omit 7b7cb9a69dc7 [SPARK-53562][PYTHON][TESTS][FOLLOW-UP] Add more tests for `maxBytesPerBatch`
    omit b34bc29018de [SPARK-53860][BUILD][TESTS] Upgrade `sbt-jupiter-interface` to 0.17.0
    omit 282e7d33d673 [SPARK-53862][DSTREAM][TESTS] Fix `CheckpointSuite.'get correct sparkdriver.[host|port] from checkpoint'` test flakiness
    omit efcc8f69a01f [SPARK-53850][SDP] Define proto for Sinks and Rename DefineDataset to DefineOutput
    omit 05a1ffb31f42 [SPARK-51169][PYTHON] Add Python 3.14 support in Spark Classic
    omit 26ba0ed11eda [SPARK-53564][CORE] Avoid DAGScheduler exits due to blockManager RPC timeout in DAGSchedulerEventProcessLoop
    omit 37f8df483aa1 [SPARK-53507][CORE] Don't use case class for BreakingChangeInfo
    omit ce3437af0179 Revert "[SPARK-53738][SQL] PlannedWrite should preserve custom sort order when query output contains literal"
    omit 2a5d03a00a03 [SPARK-53854][PYTHON][TESTS] Skip `test_collect_time` test if pandas or pyarrow are unavailable
    omit 49e2c9ecb975 [SPARK-53849][BUILD] Upgrade Netty to 4.2.6.Final
    omit 18f0463a97bd [SPARK-53846][PYTHON][TESTS] Skip `test_profile_pandas_*` tests if pandas or pyarrow are unavailable
    omit 22e24dfd7e8e [SPARK-53812][SDP] Refactor DefineDataset and DefineFlow protos to group related properties and future-proof
    omit d6f713eee79b [SPARK-53844][TESTS] Remove `SPARK_JENKINS*` and related logics from `dev/run-tests.py`
    omit bf2457b6db77 [SPARK-53843][BUILD] Upgrade `netty-tcnative` to 2.0.74.Final
    omit 9a1c742098c2 [SPARK-53751][SDP] Explicit Versioned Checkpoint Location
    omit 9ebe4a2d2c1b [SPARK-53792][SS] Fix rocksdbPinnedBlocksMemoryUsage when bounded memory …
    omit 65336a0a1769 [SPARK-53810][SS][TESTS] Split large TWS python tests into multiple small tests to speedup the CI
    omit c3934193e262 [SPARK-53125][TEST] RemoteSparkSession prints whole `spark-submit` command
    omit 4cc0b51bddb5 [SPARK-53806][SQL] Allow empty input on AES decrypt to have error class
    omit b83d70120515 [SPARK-53836][INFRA] Update script `free_disk_space_container`
    omit 3b51e19e955f [SPARK-53834][INFRA] Add a separate docker file for Python 3.14 daily build
    omit 22d9709c2921 [SPARK-53738][SQL] PlannedWrite should preserve custom sort order when query output contains literal
    omit 011b2b8b2eda [SPARK-53833][PYTHON] Update `dev/requirements.txt` to skip `torch/torchvision` in Python 3.14
    omit 57a9fc9efd81 [SPARK-53832][K8S] Make `KubernetesClientUtils` Java-friendly
    omit ee15beb6ed54 [SPARK-53808][CONNECT] Allow to pass optional JVM args to `spark-connect-scala-client`
    omit 032a4e718ccb [SPARK-53831][INFRA] Update script `free_disk_space`
    omit c5f007b11630 [SPARK-53829][PYTHON] Support `datetime.time` in column operators
    omit 8bf664071856 [SPARK-53638][SS][PYTHON] Limit the byte size of arrow batch for TWS to avoid OOM
    omit e0cb5120134d [SPARK-53795][CONNECT] Remove unused parameters in LiteralValueProtoConverter
    omit e312318f5dea [SPARK-53621][CORE][FOLLOWUP] Adding Comments And Enriching Tests of Continue Handler
    omit be1de5b5733d [SPARK-53802][SDP] Support string values for user-specified schema in SDP tables
    omit c3d6eea8679e [SPARK-53788][CORE] Move VersionUtils to `common` module
    omit 6bcd09553bb6 [SPARK-53784] Additional Source APIs needed to support RTM execution
    omit fd02372a3955 [SPARK-53804][SQL] Support TIME radix sort
    omit 08bd390c1c63 [SPARK-53794][SS] Add option to limit deletions per maintenance operation associated with rocksdb state provider
    omit 1124b0959872 [SPARK-53691][PS][INFRA][TESTS] Further reorganize tests for Pandas API
    omit 65ff85a31fe8 [SPARK-52640][SDP] Propagate Python Source Code Location
    omit 7ed0e37083bb [SPARK-53536][CORE][FOLLOWUP] Fixing Flakiness of Golden File Test With Randomly Generated SQL Scripts
    omit 848e47a5d766 [SPARK-53737][SQL][SS] Add Real-time Mode trigger
    omit 0ecb519e85de [SPARK-52614][SQL] Support RowEncoder inside Product Encoder
    omit 6526293f50e8 [SPARK-53741][BUILD] Upgrade ORC to 2.2.1
    omit 2ed58ab07614 [SPARK-53773][SQL] Recover alphabetic ordering of rules in `RuleIdCollection`
    omit ca9c0549a957 [SPARK-53536][CORE] Adding a Golden File Test With Randomly Generated SQL Scripts
    omit e04fd5953708 [SPARK-52807][SDP] Proto changes to support analysis inside Declarative Pipelines query functions
    omit 5dc061c9d74e [SPARK-53593][SDP] Fix: Use unquoted for response fields
    omit 0e9e27d6a495 [SPARK-53766][DOC] Improve execute immediate docs
    omit d5729f0d52c9 [SPARK-53621][CORE] Adding Support for Executing CONTINUE HANDLER
    omit 46ac78ea367c [SPARK-53734][SQL] Prefer table column over LCA when resolving array index
    omit 3c8c714796eb [SPARK-53574][SQL][FOLLOWUP] still reset the default AnalysisContext
    omit 746db3d07860 [SPARK-53743][SS] Remove the usage of fetchWithArrow in ListState.put/appendList
    omit 1692b553fb94 [SPARK-53728][SDP] Print PipelineEvent Message with Error In Test
    omit 776ffd5effb8 [SPARK-53735][SDP] Hide server-side JVM stack traces by default in spark-pipelines output
    omit 922adad1e3cc [SPARK-53575][CORE] Retry entire consumer stages when checksum mismatch detected for a retried shuffle map task
    omit 573c7dab0423 [MINOR][INFRA] Free disk space in releasing workflow
    omit 7a0bf9e61bcf [SPARK-53127][SQL][FOLLOWUP] Clean up golden files for and add comments for LIMIT ALL in rCTEs
    omit 6cdc62e254b9 Revert "[SPARK-53015][BUILD] Upgrade log4j to 2.25.1"
    omit 9e122017df62 [SPARK-49547][SQL][PYTHON] Add iterator of `RecordBatch` API to `applyInArrow`
    omit e56ab2fdaea0 [SPARK-53593][SDP] Add response field for DefineDataset and DefineFlow RPC
    omit 65b9da5f3083 [SPARK-46679][SQL] Fix for SparkUnsupportedOperationException Not found an encoder of the type T, when using Parameterized class
    omit 0cad1cd249a6 [SPARK-53707] Improve attribute metadata handling
    omit 470d622a926b [SPARK-53112][PYTHON][TESTS][FOLLOW-UP] Remove some tests for Non-ANSI
    omit 984d578c162d [SPARK-53729][PYTHON][CONNECT] Fix serialization of `pyspark.sql.connect.window.WindowSpec`
    omit 3d19a65fb1ba [SPARK-52407][SQL][TESTS][FOLLOWUP] Regenerate thetasketch.sql_analyzer_test
    omit 75267bcb4351 [SPARK-53715][SQL] Refactor getWritePrivileges for MergeIntoTable
    omit cb283404d3b2 [SPARK-53700][SQL][TEST][FOLLOWUP] Regenerate `ProtoToParsedPlanTestSuite` answer files
    omit 2f2abd5584e0 [SPARK-53719][SQL][PYTHON][CONNECT] Enhance type checking in `_to_col` function
    omit 75cb479c9510 [SPARK-53112][SQL][PYTHON][CONNECT][FOLLOW-UP] Change date and time to be positional arguments
    omit 00928f458c09 [SPARK-53716][PYTHON][DOCS] Document vectorized UDF with `@udf`
    omit 6ff9edcaf16c [SPARK-52407][SQL] Add support for Theta Sketch
    omit 76d9a7b43465 [SPARK-53717][SQL] Revise `MapType.valueContainsNull` parameter comment more correctly
    omit b37d51afd9d8 [SPARK-53700][SQL] Remove redundancy in `DataSourceV2RelationBase.simpleString`
    omit 64b0cb6a4654 [SPARK-53372][INFRA][FOLLOWUP] Synchronize the installation of `pyyaml` into `maven_test.yml`
    omit 70ce0cfd0d01 [SPARK-53695][PYTHON][TESTS][FOLLOW-UP] Additional test for type hint
    omit ba92e8ec8515 [SPARK-51831][SQL] Column pruning with existsJoin for Datasource V2
    omit 5aa9057a33b8 [SPARK-53708][BUILD] Upgrade `ZooKeeper` to 3.9.4
    omit f7309f286f98 [SPARK-53709][BUILD] Upgrade Junit to 5.13.4
    omit a863eee27a94 [SPARK-53694][SQL][TESTS] Improve `V1WriteHiveCommandSuite` test coverage
    omit a8cdae8948c1 [SPARK-53112][SQL][PYTHON][CONNECT] Support TIME in the make_timestamp_ntz and try_make_timestamp_ntz functions in PySpark
    omit 25c550ef37bc [SPARK-53695][PYTHON][TESTS] Add tests for 0-arg grouped agg UDF
    omit f3a69b216600 [SPARK-53356][PYTHON][DOCS] Small improvements to python data source docs
    omit 969a34283eff [SPARK-53492][CONNECT] Reject second ExecutePlan with an operation id that has already completed
    omit ed326b2d13cd [SPARK-52844][PYTHON] Update `protobuf` Python package to 5.29.5
    omit 466c608b1b80 [SPARK-53667][SQL] Fix EXPLAIN for CALL with IDENTIFIER
    omit 32f7b430d098 [SPARK-53692][DOCS] Remove a unused configuration spark.appStatusStore.diskStoreDir
    omit 997e5383c680 [SPARK-53677][SQL] Improve debuggability for JDBC data source when query contains syntax error
    omit 35c72081228f [SPARK-53674][SQL] Handle single-pass analyzer LCAs when assigning aliases
    omit 4418d6ec567e [SPARK-53562][PYTHON] Limit Arrow batch sizes in `applyInArrow` and `applyInPandas`
    omit b005f5692842 [SPARK-53689][BUILD][FOLLOW-UP] Check if RELEASE_VERSION is already set properly
    omit 3f2c623969ac [SPARK-53689][BUILD][FOLLOW-UP] Respect RELEASE_VERSION environment variable if already defined
    omit b1bbf02fdee8 [SPARK-53688][PYTHON][INFRA][TESTS] Increase the timeout and skip UDF type tests in Python-only on MacOS
    omit de7ba3b2a8b6 [SPARK-53631][CORE] Optimize memory and perf on SHS bootstrap
    omit 06f7ad2fbe8f [SPARK-53682][INFRA] Refresh `spark-rm` Docker Image with `jammy-20250819`
    omit b4592c431153 [SPARK-53678][SQL] Fix NPE when subclass of ColumnVector is created with null DataType
    omit 8ac0382b41f1 [SPARK-53689][BUILD] Respect RELEASE_VERSION environment variable if already defined
    omit 7dfd9e25066a [SPARK-53681][BUILD][TESTS] Upgrade `snowflake-jdbc` to 3.26.1
    omit 661d6110e2eb [SPARK-53645][PS] Implement `skipna` parameter for ps.DataFrame `any()`
    omit fa9e787db7b3 [SPARK-53425][PYTHON][TESTS] Add more table argument tests for Arrow Python UDTFs
    omit 2a9999fe5bf0 [SPARK-53671][PYTHON] Exclude 0-args from `@udf` eval type inference
    omit e95f12b59d80 [SPARK-53633][SQL] Reuse InputStream in vectorized Parquet reader
    omit 1841dd271bd1 [SPARK-53516][CORE][TESTS][FOLLOWUP] Fix compilation errors in `SparkPipelinesSuite`
    omit c6cea73e3cbe [SPARK-53591][SDP] Simplify Pipeline Spec Pattern Glob Matching
    omit a13187c2fa46 [SPARK-53676][PYTHON][TESTS] Skip UDF type check with numpy 1.x
    omit b6993cbbcaa0 [SPARK-53516][SDP] Fix `spark.api.mode` arg process in SparkPipelines
    omit 0e42b95efa1e [SPARK-53673][CONNECT][TESTS] Fix a flaky test failure in `SparkSessionE2ESuite - interrupt tag` caused by the usage of `ForkJoinPool`
    omit fdcd1405a84b [SPARK-53629][SQL] Implement type widening for MERGE INTO WITH SCHEMA EVOLUTION
    omit cf30da241571 [SPARK-47110][INFRA] Reenble AmmoniteTest tests in Maven builds
    omit dde895c722af [SPARK-53651][SDP] Add support for persistent views in pipelines
    omit 33196fe1b725 [SPARK-53643][DOCS] Add Arrow UDF to debugging and user guide
    omit 1e7169ed8c1d [SPARK-53660][SQL][TESTS] Add unit test for Metadata equality check
    omit a2adc4386dad [SPARK-53668][BUILD] Add `--enable-native-access=ALL-UNNAMED` to `build/sbt`
    omit 69031c918167 [SPARK-53661][BUILD][TESTS] Upgrade `bouncycastle` to 1.82
    omit c37ab6e573d7 [SPARK-53653][DOC] Update `rexml` gem version to 3.4.4
    omit f9aa4c943018 [SPARK-53655][SQL][TESTS] Fix the intention of 'read parquet footers in parallel' test
    omit ed2692fe885c [SPARK-53654][SQL][PYTHON] Support `seed` in function `uuid`
    omit 984e16b60862 [SPARK-53657][PYTHON][TESTS] Enable doctests for `GroupedData.agg`
    omit f48de1015470 [SPARK-53592][PYTHON][TESTS][FOLLOW-UP] Remove unused config in the parity test
    omit 36ed5ee4a0d7 [SPARK-53429][PYTHON] Support Direct Passthrough Partitioning in the PySpark Dataframe API
    omit 71c67b079087 [SPARK-53641][DOCS] Add PARTITION BY support in Arrow Python UDTF docs
    omit 686d84453610 [SPARK-53592][PYTHON] Make `@udf` support vectorized UDF
    omit 589141e3085a [SPARK-53233][SQL][FOLLOWUP] Add compatibility class/object for org.apache.spark.sql.execution.streaming
    omit 4f10262391f8 [SPARK-53623][SQL] improve reading large table properties performance
    omit db13a38e565b [SPARK-53578][CONNECT] Simplify data type handling in LiteralValueProtoConverter
    omit a8bb8b05d41e [SPARK-53625][SS] Propagate metadata columns through projections to address ApplyCharTypePadding incompatibility
    omit 552effccce9a [SPARK-53637][BUILD] Demote bcprov-jdk18on to test scope
    omit fb46424cd112 [SPARK-53626][DOCS] Add invalid mixed-type operations to ANSI migration guide
    omit 263979277198 [SPARK-53523][SQL][FOLLOWUP] Udpate scaladocs and add tests in ProcedureSuite
    omit 49a3c132e1cb [SPARK-53632][PYTHON][DOCS][TESTS] Reenable doctest for `DataFrame.pandas_api`
    omit 4b93d4cfd49a [SPARK-53598][SQL] Check the existence of numParts before reading large table property
    omit 1795306078e0 [SPARK-53323][PYTHON][CONNECT] Enable Spark Connect tests for df.asTable() in Arrow UDTF
    omit 1ec647ea17d8 [SPARK-53630][PYTHON][DOCS][TESTS] Reenable doctest for `Dataframe.freqItems`
    omit b87f6d14a787 [MINOR][TESTS] Restore classic-only python tests
    omit 9d23f2ffaf3d [SPARK-53355][PYTHON][SQL] fix numpy 1.x repr in type tests
    omit 551e7f2e1e82 [SPARK-53619][PYTHON][DOCS][TESTS] Enable doctests for toArrow/toPandas/mapInArrow/mapInPandas
    omit 3080e61e5ff0 [SPARK-53387][PYTHON] Add support for Arrow UDTFs with PARTITION BY
    omit 010d36f21940 [SPARK-53507][CONNECT] Add breaking change info to errors
    omit f49047130fdb [SPARK-52659][SQL] Misleading modulo error message in ansi mode
    omit 3990b0f1d983 [SPARK-53620][CORE] SparkSubmit should print stacktrace when exitFn is called
    omit 87a71fabb097 [SPARK-53438][CONNECT][SQL] Use CatalystConverter in LiteralExpressionProtoConverter
    omit 79a92831cc69 [SPARK-53372][SDP] SDP End to End Testing Suite
    omit e08c15b62712 [SPARK-53606][DOCS] Fix MapInPandas/MapInArrow examples with barrier
    omit f57a473a02e4 [SPARK-53546][TESTS][FOLLOW-UP] Fix nested array schema evolution and style for InMemoryBaseTable
    omit 33e40b71b0bb [SPARK-52991][SQL][FOLLOW-UP] Revise `MergeIntoTable` to use `lazy val` and add a new test
    omit 067f295f062b [SPARK-53604][INFRA] Temporarily increase PySpark job execution time to 150 minutes
    omit aaf9308307cc [MINOR][DOCS] Update the `See Also` section for time functions
    omit 5d1ad6187733 [SPARK-53603][BUILD] Upgrade Checkstyle to 11.0.1
    omit 3d87de318b62 [SPARK-53594][PYTHON] Make arrow UDF respect user-specified eval type
    omit 53a330b2eb93 [SPARK-53602][PYTHON] Profile dump improvement and profiler doc fix
    omit 78aba006b836 [SPARK-53581][CORE] Fix potential thread-safety issue for mapTaskIds.add()
    omit 4aa934ec0401 [SPARK-53599][BUILD] Upgrade `Netty` to 4.1.127.Final
    omit 54e69bafa362 [SPARK-53601][INFRA] Use Java 25 instead of 25-ea
    omit 7781ec2d5086 [SPARK-53600][SQL] Revise `SessionHolder` last access time log message
    omit 56fa1e906583 [MINOR][PYTHON][DOCS] Correct the examples of `toPandas` and `toArrow`
    omit 86d310b97f85 [SPARK-53559][SQL][CATALYST] Fix HLL sketch updates to use raw collation key bytes
    omit f03c644ccfff Fix: SparkML-connect can't load SparkML (legacy mode) saved model
    omit dbbb54db214d [SPARK-53582][SQL] Extend `isExtractable` so it can be applied on `UnresolvedExtractValue`
    omit 24139be34966 [MINOR][PYTHON][DOCS] Update the doctests to check the default column names
    omit fa2f7d5e7449 [SPARK-53584][PYTHON] Improve process_column_param validation and column parameter docstring
    omit e3dce42d63a2 [SPARK-53590][BUILD] Add `huaweicloud-provided` profile
    omit 6c9e7503f3ac [SPARK-53586][K8S][BUILD] Upgrade `kubernetes-client` to 7.4.0
    omit 10b27f3a5ad9 [SPARK-53577][DOCS] Fix Scaladoc source links for java sources
    omit 8c422f974a68 [SPARK-53113][SQL] Support the time type by try_make_timestamp()
    omit 2844705f4885 [SPARK-53127][SQL] Enable LIMIT ALL to override recursion row limit
    omit 10c634fe8cc4 [SPARK-53572][SQL] Avoid throwing from ExtractValue.isExtractable
    omit 1a86e97664ef [SPARK-53574] Fix AnalysisContext being wiped during nested plan resolution
    omit 40993e671b87 [SPARK-53361][SS][1/2] Optimizing JVM–Python Communication in TWS by Grouping Multiple Keys into One Arrow Batch
    omit fbdad297f542 [SPARK-53546][SQL][TESTS] Fix InMemoryDataSource to return default value or null for new fields
    omit ce94cb3132f9 [SPARK-53552][SQL] Optimize substr SQL function
    omit 9bd844b87c84 [SPARK-53558][SQL] Show fully qualified table name including the catalog name in the exception message when the table is not found
    omit 19ca63f82fed [SPARK-53553][CONNECT] Fix handling of null values in LiteralValueProtoConverter
    omit bacd34398aaa [SPARK-53182][PYTHON][DOCS] Fix broken and missing links in PySpark DataFrames user guide
    omit dbd765dbfac2 [SPARK-53544][PYTHON] Support complex types on observations
    omit 8c4916590b09 [SPARK-43579][PYTHON] optim: Cache the converter between Arrow and pandas for reuse
    omit e4d60e9a9778 [SPARK-53563][PS] Optimize: sql_processor by avoiding inefficient string concatenation
    omit bc93c255bd8b [SPARK-53568][CONNECT][PYTHON] Fix several small bugs in Spark Connect Python client error handling logic
    omit 54dee4a4b59c [SPARK-53537][CORE] Adding Support for Parsing CONTINUE HANDLER
    omit f92601ce879b [SPARK-53444][SQL] Rework execute immediate
    omit 04a40ab8680e [SPARK-53157][CORE] Decouple driver and executor polling intervals
    omit 4441fa1a8695 [SPARK-53529][PYTHON][CONNECT] Fix `pyspark` connect client to support IPv6
    omit b8657a9625d5 [SPARK-53523][SQL] Named parameters respect `spark.sql.caseSensitive`
    omit c0acf45023f1 [SPARK-53550][SQL][FOLLOWUP] Union output partitioning should compare canonicalized attributes
    omit 8178cb0588dc [SPARK-53491][SS] Fix exponential formatting of inputRowsPerSecond and processedRowsPerSecond in progress metrics JSON
    omit 1dab449da5c8 [SPARK-53413][SQL] Shuffle cleanup for commands
    omit 07d987a0cc6e [MINOR][CORE][TESTS] Fix typo in MasterDecommissionSuite class name
    omit faa1aaa64f49 [SPARK-53498][SDP] Correctly Reference `pyspark/pipelines/cli.py` from `spark-pipelines` Binary
    omit 29626507fa02 [SPARK-52426][SQL][FOLLOWUP] Fix spark-sql repl w/ RedirectConsolePlugin
    omit f465eca7e954 [SPARK-53557][INFRA] Reduce automated vote email deadline from 4 days to 73 hours
    omit 4575b1da9e1d [SPARK-53560][SS][SQL] Crash looping when retrying uncommitted batch in Kafka source and AvailableNow trigger
    omit b9848ac61a71 [SPARK-53561][SS] Catch Interruption Exception in TransformWithStateInPySparkStateServer during outputStream.flush to avoid the worker crash
    omit f90333d109ba [SPARK-53549][SS] Always close the arrow allocator when list state request process is completed
    omit bb41e19ae9ee [SPARK-53525][CONNECT] Spark Connect ArrowBatch Result Chunking
    omit a8f56d4f18a4 [SPARK-52601][SQL] Support primitive types in TransformingEncoder
    omit 8edc7685b971 [SPARK-53550][SQL] Union output partitioning should compare canonicalized attributes
    omit 1817e676f1b7 [SPARK-53524][CONNECT][SQL] Fix temporal value conversion in LiteralValueProtoConverter
    omit d0177795bbbe [SPARK-53399][SQL] Merge Python UDFs
    omit 7201e60bb3e8 [SPARK-53512][SQL] Better unification of DSv2 PushDownUtils
    omit d00eb2b3272f [MINOR][PYTHON][TESTS] Skip some tests if numpy not installed
    omit 6dd4001c29af [SPARK-53490][CONNECT][SQL] Fix Protobuf conversion in observed metrics
    omit feaf659d0cc0 [SPARK-53332][SS] Enable StateDataSource with state checkpoint v2 (only snapshotStartBatchId option)
    omit 81823c28036e [SPARK-53543][SQL][TEST-ONLY] Add more test coverage for `Window` on top of `Aggregate`
    omit 81c29eef4ad3 [SPARK-52238][SDP] Rename Pipeline Spec Field "definitions" to 'libraries'
    omit d61a7b93ae02 [SPARK-53029][FOLLOWUP] Update PyArrow StructType usage to avoid .names attribute for compatibility with PyArrow
    omit f03902c06bae [SPARK-53531][CORE] Better error message for HadoopRDD.getInputFormat
    omit 0b5da0260c3d [SPARK-53541][K8S][INFRA] Update K8s IT CI to use K8s 1.34
    omit 345e7da1cb0b [SPARK-53530][SS] Clean up the useless code related to `TransformWithStateInPySparkStateServer`
    omit 9ab693c01ed8 [SPARK-51756][CORE] Computes RowBasedChecksum in ShuffleWriters
    omit 19c271f10e98 [SPARK-53538][SQL] `ExpandExec` should initialize the unsafe projections
    omit e75ca577923f [SPARK-53402][CONNECT] Support Direct Passthrough Partitioning Dataset API in Spark Connect in Scala
    omit be5d367ff234 [SPARK-53533][BUILD] Upgrade Jetty to 11.0.26
    omit 19912f85b8a5 [SPARK-53540][INFRA] Remove `pypy3` from `spark-rm` image
    omit f210e979f017 [SPARK-53539][INFRA] Add `libwebp-dev` to recover `spark-rm/Dockerfile` building
    omit 19735de5b46e [SPARK-53534][PYTHON][TESTS] Add tests for arrow udf with numpy output
    omit 50ce752a57ae [SPARK-53434][SQL] ColumnarRow's get should also check isNullAt
    omit 1d09e5a470ff [SPARK-52449][CONNECT][PYTHON][ML] Make datatypes for Expression.Literal.Map/Array optional
    omit fb0c4b8d22e8 [SPARK-53506][PS] Disallow `%` between Decimal and float under ANSI
    omit b08ac303faed [SPARK-53521] Refactor Star expression
    omit cff7baef736d [SPARK-53526][SQL] Enable SQL scripting by default
    omit 2f305b6817bc [SPARK-53527][SQL] Improve fallback of analyzeExistenceDefaultValue
    omit adef6f30fe34 [SPARK-53479][PS] Align `==` behavior with pandas when comparing against scalar under ANSI
    omit d16e92de0a3a [SPARK-53029][PYTHON] Support return type coercion for Arrow Python UDTFs
    omit b633ad30ac84 [SPARK-53518][SQL] No truncation for catalogString of User Defined Type
    omit 8575d09f80b0 [SPARK-53522][SDP][TEST] Simplify PipelineTest
    omit d3f638f3f8f4 [SPARK-48163][CONNECT][TESTS] Fix flaky tests in `SparkConnectServiceSuite` which are caused by `executorHolder` undefined
    omit 1454f252e45f [SPARK-53489][SQL] Remove using `v2ColumnsToStructType` in `ApplyDefaultCollationToStringType`
    omit c247483480d3 [SPARK-53505][CONNECT] Try the eager analysis first for withColumns and withColumnsRenamed in Spark Connect
    omit 90ef38960586 [SPARK-53488][SQL] Update CollationTypeCoercion to be compatible with single-pass Analyzer logic
    omit cd6f63042461 [SPARK-52745][SQL][FOLLOWUP] Allow the absense of exists default
    omit 3ad04e36e3ea [SPARK-53517][PYTHON][TESTS] Add more test for pyarrow datatype conversion
    omit 6f51e04dcd7f [SPARK-53515][SQL] Remove unused `private lazy val` from `SchemaOfCsv/Xml`
    omit af16aa8e11c2 [SPARK-53480][PYTHON] Refactor arrow python runner code
    omit ab9a63626018 [SPARK-53514][DOCS] Update `Gemfile` to allow Ruby 3.0+ explicitly
    omit 0dc38b02f4a7 [SPARK-53502][CONNECT][SQL] Improve the naming of methods in LiteralValueProtoConverter
    omit 254beec342f4 Revert "[SPARK-53458][INFRA] Add daily dry-run release test GitHub Action jobs for Spark 4.0/3.5"
    omit d9a23c2f447f [SPARK-50888][CONNECT][TESTS] Remove flaky assertion from `SparkConnectServiceSuite`
    omit 24409692b640 [SPARK-53458][INFRA] Add daily dry-run release test GitHub Action jobs for Spark 4.0/3.5
    omit e45ddd17242d [SPARK-53497][PS] Disallow `/`|`//`|`*` between Decimal and float under ANSI
    omit 1043d53b62bd [SPARK-53500][PYTHON][TESTS] Make `test_unpivot` deterministic
    omit 91899986a0a1 [SPARK-53481][PYTHON] Make Hybrid queue class
    omit 9d27db940673 [SPARK-53474][SQL] Add conf DATA_SOURCE_V2_EXPR_FOLDING
    omit f2ede1846915 [SPARK-53453][PYTHON][ML][FOLLOW-UP] Update `dev/requirements`
    omit 061bbebb811c [SPARK-53499][PYTHON] from_arrow_type should respect containsNull in ArrayType
    omit 61ba323fc277 [SPARK-53486][ML][PYTHON] Avoid setting `weights_only=False` in torch model load
    omit fb7719869666 [SPARK-53441][PS] Bools `|`/`&`/`^` None should fail under ANSI
    omit f78f164c5ea3 [SPARK-53496][SQL][CORE][ML][SS][K8S][YARN] Remove dangling right curly braces in log
    omit 41ee8d95b36c [SPARK-53485][DOCS][SECURITY] Add Proxy User Instructions
    omit fda37bf432dd [SPARK-52431][SDP][TESTS][FOLLOWUP] Remove unused scalatest Matchers from SparkPipelinesSuite
    omit 241f92ec527a [SPARK-53494][BUILD] Upgrade `Netty` to 4.1.126.Final
    omit fcaeb6358892 [SPARK-53487][SQL] Improve comments in `NormalizePlan.scala`
    omit 336ca8c12a16 [SPARK-49386][CORE][SQL][FOLLOWUP] More accurate memory tracking for memory based spill threshold
    omit 65c5775a3365 [SPARK-53420][BUILD] Upgrade Parquet to 1.16.0
    omit 76352046c890 [SPARK-53472][DOCS] Fix jekyll-redirect-from template and generated html files
    omit eea976727b33 [MINOR][DOCS] Update comments for ResolveCoalesceHints
    omit 13b23b247351 [SPARK-53459][CORE] Use ReadOnlySparkConf in more places
    omit c7d8c2b8695a [SPARK-53470][SQL] ExtractValue expressions should always do type checking
    omit 20c60d982de0 [SPARK-50785][SQL][FOLLOWUP] Remove overrideIfExists check in SqlScriptingLocalVariableManager
    omit 60d2078c6b07 [SPARK-53461][PYTHON][DOCS] Clarify pie plot parameter `subplots`
    omit b01747341918 [SPARK-53401][SQL] Enable Direct Passthrough Partitioning in the DataFrame API
    omit fbbc5509bbfe [SPARK-51739][PYTHON][FOLLOW-UP] Disable spark.sql.execution.arrow.pyspark.validateSchema.enabled by default
    omit 7f5db0c1c557 [SPARK-52346][SDP] Propagate partition columns from destination for BatchTableWrite
    omit 0c3dacb6497b [SPARK-53207][SDP] Send Pipeline Event to Client Asynchronously
    omit a5f76ca33fd0 [SPARK-53464][K8S][INFRA] Update `setup-minikube` to v0.0.20
    omit 6ec533ca3e37 [SPARK-53460][BUILD] Upgrade `analyticsaccelerator-s3` to 1.3.0
    omit 41c43466db5c [MINOR][BUILD] Remove `preview` postfix in documentation.md when releasing
    omit fb22d371f0dd [MINOR][BUILD] Remove todos for testing in the real releases
    omit 6476dbc92afa [MINOR][BUILD] Fix download of preview releases in the news
    omit 885bfc22cb0d [SPARK-53333][SS] Enable StateDataSource with state checkpoint v2 (only readChangeFeed)
    omit 47574ba833af [SPARK-53453][PYTHON][ML] Unblock 'torch<2.6.0'
    omit ce763c18eac8 [SPARK-53452][PYTHON] `from_arrow_type` should respect `valueContainsNull`
    omit 8c8c96641ce6 [SPARK-53443][PYTHON] Update python to `SPDX-license`
    omit 9ada1e7e68db [SPARK-51820][FOLLOWUP][CONNECT] Replace literal in `SortOrder` only under `Sort` operator
    omit 166895a9a073 [SPARK-53450] Nulls are filled unexpectedly after converting hive table scan to logical relation
    omit 916a779a8d46 [SPARK-53449][SQL] Simply options for builtin Datasource Scan related classes
    omit 5a30d68a6520 [SPARK-53451][PYTHON][TESTS] Fix scheduled job for python 3.13 no GIL
    omit 4ba38c2c3305 [SPARK-52346][SDP] Fix counter initialization and increment logic for flow retries
    omit 183f9726938e [SPARK-53433][TESTS][FOLLOW-UP] Add a test for aggregation
    omit 688a30b03b56 [SPARK-53433][TESTS][FOLLOW-UP] Make the test compatible with PyArrow 15.0.0
    omit a74d50b95970 [SPARK-53156][CORE] Track Driver Memory Metrics when the Application ends
    omit 1485295a6e49 [SPARK-53437][SQL] InterpretedUnsafeProjection shall setNull4Bytes for YearMonthIntervalType field
    omit 9c5015603acc [SPARK-53108][SQL] Implement the time_diff function in Scala
    omit f93eff3d9062 [SPARK-53329][CONNECT] Improve exception handling when adding artifacts
    omit c459d71f8134 [SPARK-53421][SPARK-53377][SDP] Propagate Logical Plan ID in SDP Analysis
    omit 871fe3ded668 [SPARK-53435][SQL] Fix race condition in CachedRDDBuilder
    omit 5a0e5b15d362 [SPARK-48547][DEPLOY] Add opt-in flag to have SparkSubmit automatically call System.exit after user code main method exits
    omit 6d36560fd4b8 [SPARK-53422][SPARK-30269][SQL][TEST] Make test case robust
    omit 38a3d326bd62 [SPARK-53417][PYTHON][TESTS][FOLLOW-UP] Add more tests for aggregation and window
    omit 7e7380ff590b [SPARK-53424][PYTHON][TESTS] Hide traceback in `assertSchemaEqual/assertDataFrameEqual/assert_rows_equal`
    omit 08e39c310d09 [SPARK-53436][BUILD] Upgrade `Netty` to 4.1.124.Final
    omit eca8c62392d3 [SPARK-53427][PS][TESTS] Test divisor 0 in truediv/floordiv/mod tests under ANSI
    omit 59b34dcb08a6 [SPARK-51168][BUILD] Upgrade to Hadoop 3.4.2
    omit 7bbc5d292af5 [SPARK-53386][SQL] Support query parameter ending with semicolon in JDBC connectors
    omit a68ac4885ddd [SPARK-53419][SQL][TEST] Move common SqlScriptingContextManager initialization code from SqlScriptingInterpreterSuite
    omit 5c1c6e367f18 [MINOR] Fix redundant brace in log
    omit b177b6515c83 [SPARK-53431][PYTHON] Fix Python UDTF with named table arguments in DataFrame API
    omit 5bf4a29fe158 [SPARK-53433][PYTHON][TESTS] Add test for Arrow UDF with VariantType
    omit 5b2c4cf9ce88 [SPARK-53341][CORE] Expand golden test coverage on multivariable DECLARE
    omit 78871d7de88b [SPARK-53394][CORE] UninterruptibleLock.isInterruptible should avoid duplicated interrupt
    omit 7b8877fd3556 [SPARK-53355][PYTHON][SQL] test python udf type behavior
    omit 0f5204c67d96 [SPARK-53403][PS] Improve add/sub tests under ANSI
    omit 9b2592c5be4e [SPARK-53423][SQL] Move all the single-pass resolver related tags to `ResolverTag`
    omit 54b53f955269 [SPARK-53398][SS] Ensure that RocksDBMemoryManager metrics reporting doesn't throw a NullPointerException
    omit 7b8186a45434 [SPARK-53418][SQL] Support `TimeType` in `ColumnAccessor`
    omit 596d03fc541c [SPARK-53415][SQL] Simply options for builtin FileFormats
    omit d233607c611d [SPARK-53412][K8S][INFRA][DOCS] Upgrade Volcano to 1.12.2
    omit 1f1bacca720b [SPARK-53143][SQL] Fix self join in DataFrame API - Join is not the only expected output from analyzer
    omit 0c5797aa3e74 [SPARK-53416][SS][TESTS] Use `createOrReplaceTempView` instead of `registerTempTable` in `StreamingQueryOptimizationCorrectnessSuite`
    omit 2de024807103 [SPARK-53417][PYTHON][TESTS] Add test for Arrow UDF with TimeType
    omit 316d06ba2f5f [SPARK-53318][SQL] Support the time type by make_timestamp_ltz()
    omit 824da27e4386 [SPARK-53391][CORE][FOLLOWUP] Add comments for PrimitiveKeyOpenHashMap removal in Mima
    omit 2167693a705a [SPARK-53408][SQL] Remove unused functions from `QueryCompilationErrors`
    omit e921a74ba2b3 [SPARK-53414][PYTHON][TESTS] Add tests for Arrow UDF with profiler
    omit 620474661f11 [SPARK-53397][PYTHON][TESTS] Fix UDTF with collations test indentation
    omit 0c9af991ca28 [SPARK-51585][SQL][FOLLOWUP] Turn on ANSI mode in DockerJDBCIntegrationV2Suite
    omit 51b5f30ae321 [SPARK-53390][PS] Raise error when bools with None `astype` to ints under ANSI
    omit 86dad8358a42 [SPARK-53348][SQL][FOLLOWUP] Don't run `AlwaysPersistedConfigsSuite` in case ANSI was set using the system var
    omit ac45be2e1587 [SPARK-53294][SS] Enable StateDataSource with state checkpoint v2 (only batchId option)
    omit 5337a5780690 [SPARK-52777][SQL] Enable shuffle cleanup mode configuration in Spark SQL
    omit 8b1a74843d8e [SPARK-53392][ML][CONNECT] Move SpecializedArray handling to connect-common
    omit fb8df345c886 [SPARK-53391][CORE] Remove unused PrimitiveKeyOpenHashMap
    omit 0c26d7a1a68c [SPARK-53395][PYTHON][CONNECT][TESTS] Add tests for combinations of different scalar UDFs
    omit 2a4c1886d8f0 [SPARK-53393][PYTHON] Disable memory profiler for Arrow Scalar Iterator UDFs
    omit 21b1d11ddd6a [SPARK-53385][SQL] Refactor Identifier evaluation out
    omit 994fc65afa6d [SPARK-53109][SQL] Support TIME in the make_timestamp_ntz and try_make_timestamp_ntz functions in Scala
    omit e7fb0704747a [SPARK-53384][SQL] Refactor variable resolution out
    omit b04245fa2c43 [SPARK-53388][SQL][TESTS] Split `collations.sql`
    omit a9d6919ef45c [SPARK-53367][PYTHON][SQL] add int to decimal coercion for Arrow UDFs
    omit 01667f1c4c54 [SPARK-53369][PYTHON] Fix error message for UDFs with `CHAR/VARCHAR` return type
    omit 1dfb6a247861 [SPARK-53330][SQL][PYTHON] Fix Arrow UDF with DayTimeIntervalType (bounds != start/end)
    omit 67f9d378e7ff [SPARK-53383][PYTHON][TESTS] Add tests to check the timezone handling in Arrow UDF
    omit ae7178c5eb78 [SPARK-53381][CONNECT] Avoid creating temporary collections in `toCatalystStruct`
    omit e74b77e656bd [SPARK-53342][SQL] Fix Arrow converter to handle multiple record batches in single IPC stream
    omit 20a6af7d00e5 [SPARK-52873][SQL][TESTS][FOLLOWUP] Fix test for non-ansi mode
    omit fc1da934c371 [SPARK-53365][SQL] Unify code for persisting of configs in views and UDFs
    omit 542451400423 [SPARK-53382][SQL] Fix rCTE bug with malformed recursion
    omit f0a3a2ea1611 [SPARK-53349][SQL] Optimized XML parser can't handle corrupted files correctly
    omit b5840e156032 [SPARK-52930][CONNECT] Use DataType.Array/Map for Array/Map Literals
    omit 79a0ca72daca [SPARK-53366][CONNECT] Apply formatting rules to sql/connect/shims
    omit f441da461398 [SPARK-52110][SDP][SQL][FOLLOWUP] Move optionsClause to before tableAlias
    omit bc36a7db43f2 [SPARK-53275][SQL] Handle stateful expressions when ordering in interpreted mode
    omit c13c10fa194a [SPARK-53362][ML][CONNECT] Fix IDFModel local loader bug
    omit ef9322f31956 [SPARK-53357][PYTHON] Update `pandas` to 2.3.2
    omit 50a2ebe87f5c [SPARK-53344][DOCS] Add user guide for Arrow Python UDTFs
    omit 1ff987ff7a53 [SPARK-53352][PYTHON] Refine the error message for unsupported return type
    omit f5f590b45025 [SPARK-53359][PYTHON] Fix Arrow UDTF to handle the results as iterator
    omit 8178f88d7b90 [SPARK-53353][PYTHON] Fail Scalar Iterator Arrow UDF with 0-arg
    omit 2a0192546617 [SPARK-53354][CONNECT] Simplify LiteralValueProtoConverter.toCatalystStruct
    omit e28f42700611 [SPARK-53360][SQL] Once strategy with ConstantFolding's idempotence should not be broken
    omit c86371783102 [SPARK-53358] Improve arrow Python UDTF output type mismatch error message
    omit dab3464701e2 [SPARK-52982][PYTHON] Disallow lateral join with Arrow Python UDTFs
    omit 6ab0df9287c5 [SPARK-53044] Change Declarative Pipelines import alias convention from "sdp" to "dp"
    omit e2eb540cb8db [SPARK-53103][SS] Throw an error if state directory is not empty when query starts
    omit ce646b3dac93 [SPARK-52991][SQL] Implement MERGE INTO with SCHEMA EVOLUTION for V2 Data Source
    omit 1d8481021cde [SPARK-52873][SQL] Further restrict when SHJ semi/anti join can ignore duplicate keys on the build side
    omit a322e0c5e15c [SPARK-53348][SQL] Always persist ANSI value when creating a view or assume it when querying if not stored
    omit 7007e1c7ad64 [SPARK-53345][SS][TESTS] Use withTempDir for consistent directory across restarts in streaming test
    omit f0e89994fd41 [SPARK-53265][PYTHON][DOCS] Add Arrow Python UDF Type Coercion Tables in Arrow Python UDF Docs
    omit 8530444e25b8 [SPARK-53346][CONNECT] Avoid creating temporary collections in toCatalystArray/toCatalystMap
    omit 12c87cef0d0f [SPARK-53328][ML][CONNECT] Improve debuggability for SparkML-connect
    omit 7e73d0ec8763 [SPARK-53336][ML][CONNECT] Reset `MLCache.totalMLCacheSizeBytes` when `MLCache.clear()` is called
    omit 44cdd2653737 [SPARK-52482][DOCS][FOLLOW-UP] Mention behavior changes in migration guide
    omit 2e6f0ec2103f [MINOR][PYTHON][DOCS] Add TimeType to API reference
    omit 5660dbadf90e [SPARK-53334][CONNECT] `LiteralValueProtoConverter` should keep the order of input map
    omit 33df1b6d237c [SPARK-51874][SQL][FOLLOW-UP] Revert API changes of rebase methods in DataSourceUtils and AvroOptions
    omit ee619d3dc2e9 [SPARK-53260][SQL] Reducing number of JDBC overhead connections creation
    omit 967f2b6b2e60 [SPARK-53308][SQL] Don't remove aliases in RemoveRedundantAliases that would cause duplicates
    omit 549c30acfebd [SPARK-52582][SQL] Improve the memory usage of XML parser
    omit 594d26cef814 [SPARK-53015][BUILD] Upgrade log4j to 2.25.1
    omit 4aa8b677b0ad [SPARK-53331][PS] Re-enable SPARK_ANSI_SQL_MODE during doc generation
    omit 5ce657f189c4 [SPARK-53287][PS] Add ANSI Migration Guide
    omit 9920b2276326 [SPARK-53030][PYTHON] Support Arrow writer for streaming Python data sources
    omit 77413d443f23 [SPARK-51874][SQL][FOLLOW-UP] Revert ParquetOptions rebase methods to return string type
    omit af0b444dfd34 [SPARK-53144][CONNECT][SQL] Make CreateViewCommand in SparkConnectPlanner side effect free
    omit f41c538e0b6d [SPARK-52837][SQL][PYTHON][FOLLOW-UP] Specify the BitWidth of Arrow TimeType
    omit d2e550fcd280 [SPARK-53326][BUILD] Upgrade ORC Format to 1.1.1
    omit 2a5b097e281b [SPARK-53295][PS] Turn on ANSI by default for Pandas API on Spark
    omit 9a62f7d25333 [SPARK-53311][SQL][PYTHON][CORE] Make PullOutNonDeterministic use canonicalized expressions
    omit 076618a46cab [SPARK-49872][CORE] Remove jackson JSON string length limitation
    omit 9f63d1dbd4a0 [SPARK-53303][SS][CONNECT] Use the empty state encoder when the initial state is not provided in TWS
    omit 7782a70889ab [SPARK-53301][PYTHON] Differentiate type hints of Pandas UDF and Arrow UDF
    omit 83e7b4faf731 [SPARK-53012][PYHTON] Support Arrow Python UDTF in Spark Connect
    omit a58c5e1cee9b [SPARK-53288][SS] Fix assertion error with streaming global limit
    omit b34b95088f2b [SPARK-53306][SQL][CONNECT][YARN][TESTS] Fix wrong package statements
    omit 26dbf651bf8c Revert "[SPARK-52709][SQL] Fix parsing of  STRUCT<>"
    omit f9518003d232 [SPARK-53305][PYTHON] Support TimeType in createDataFrame
    omit 8ede68bf2860 [SPARK-53307][CONNECT][CLIENT][PYTHON][SCALA] Remove RetriesExceeded error from Spark Connect Python and Scala clients
    omit 05101e9928da [SPARK-53304][BUILD] Upgrade commons-text to 1.14.0
    omit c86093f11e28 [SPARK-53146][CONNECT][SQL] Make MergeIntoTable in SparkConnectPlanner side effect free
    omit 9e06a506a947 [SPARK-51920][SS][PYTHON] Fix composite/nested type in value state for python TWS
    omit 59556b1a69f6 [SPARK-53302][PYTHON][TESTS] Make doctest of df.unpivot deterministic
    omit 2be3e5416f66 [SPARK-53300][PYTHON][TESTS] Fix field names in test_unpivot
    omit 87fc2ffe4d14 [SPARK-53299][INFRA] Rebalance the test modules of pandas API on connect
    omit 783167143860 [SPARK-53297][SDP] Fix StreamingTable Declarative Pipelines API docstring
    omit fd77ec6a2af2 [SPARK-53291][SQL] Fix nullability for value column
    omit 923d70f3fe6b [SPARK-52307][PYTHON][FOLLOW-UP] Fix type hint for Scalar Arrow Iterator UDF
    omit c68cf9405030 [SPARK-53290][SQL][CONNECT] Fix Metadata backward-compatibility breaking
    omit 959f424abb92 Revert "[SPARK-49872][CORE] allow unlimited json size again"
    omit efd7c85da42a [SPARK-53268][BUILD][TESTS] Update Oracle free version from 23.7 to 23.9
    omit f98394043b62 [SPARK-52998][CORE] Multiple variables inside declare
    omit cd8fdbce052c [SPARK-53274][SQL] Support left and right join pushdown in JDBCScanBuilder
    omit 337a67ff58ad [SPARK-52741][SQL] RemoveFiles ShuffleCleanup mode doesnt work with non-adaptive execution
    omit 4b72478e0578 [SPARK-53282][PYTHON][TESTS] Add test for arrow udf type hints
    omit 9414e46a562e [SPARK-53251][PYTHON] Enable DataFrame API testing with asTable() for Arrow Python UDTFs
    omit e5ed2264e89d [SPARK-53285][INFRA] Run `Java 17/25` Maven install tests if necessary
    omit 45440905e52f [SPARK-53284][PS] Adjust imports of Spark config in tests
    omit 7f3c7047ae4c [SPARK-53269][PYTHON][FOLLOWUP] Fix GRPC and GRPCStatus check
    omit d4539024a927 [SPARK-53050][PS] Enable MultiIndex.to_series() to return struct for each entry
    omit 2f27838cadad [SPARK-53276][SS] Checking if we own the stamp before closing RocksDB
    omit 02898338cffc [SPARK-52988][SQL] Fix race conditions at CREATE TABLE and FUNCTION when IF NOT EXISTS is used
    omit bdc424345aa9 [SPARK-49984][CORE] Fix `supplementJava(Module|IPv6)Options` to update `extraJavaOptions` only
    omit eba5381e4ce5 [MINOR][DOCS] Fix an Arrow UDF example
    omit a3a394e31f8c [SPARK-53280][CORE] Use Java `instanceof` instead of `Throwables.throwIf*` methods
    omit e20c21a212da Revert "[SPARK-53277][INFRA] Improve `merge_spark_pr.py` to stop early in case of the closed PRs"
    omit ca75a0ebea8c [SPARK-53269][PYTHON][TESTS] Centralize connect dependency checks
    omit 4aa3a361226c [SPARK-53279][INFRA] Improve `determine_modules_for_files` to ignore `Scalastyle/Checkstyle` rule files
    omit b7a9b42e425f [SPARK-53277][INFRA] Improve `merge_spark_pr.py` to stop early in case of the closed PRs
    omit 81850af17fda [SPARK-53278][INFRA] Improve `merge_spark_pr.py` to accept PR numbers as a CLI argument
    omit 9297712a66d3 [SPARK-53272][SQL] Refactor SPJ pushdown logic out of BatchScanExec
    omit 14f300406133 [SPARK-53271][PYTHON][INFRA] Show Python Versions in PySpark Jobs
    omit 977bc7c6aa65 [SPARK-53106][SS] Add schema evolution tests for TWS Scala spark connect suites
    omit 43f650eaa43f [SPARK-52844][PYTHON] Update numpy to 1.22
    omit 72c1d414331f [SPARK-53270][SQL][TESTS] Disable oracle datetime pushdown tests in non-ANSI mode
    omit a65659639f96 [MINOR][PYTHON][DOCS] Update an UDF example with specified eval type
    omit cf43735bd1af [SPARK-53267][DOCS] Update the javadoc for Arrow UDF physical plans
    omit 2d4f7c39dece Revert "[SPARK-53265][PYTHON][DOCS] Add Arrow Python UDF Type Coercion Tables in Arrow Python UDF Docs"
    omit 5c52a00e8f4f [SPARK-53265][PYTHON][DOCS] Add Arrow Python UDF Type Coercion Tables in Arrow Python UDF Docs
    omit b3f12af6b8c8 [SPARK-53266][TESTS] Regenerate benchmark results
    omit ebf4dd1889f3 [MINOR][PYTHON][TESTS] Use different temp table name in foreachBatch tests
    omit 8163fa4f581d [SPARK-53124][SQL] Prune unnecessary fields from JsonTuple
    omit 645ed16f92b0 [SPARK-53173][SQL][TESTS] Improve `Owner` regex pattern in the replaceNotIncludedMsg method
    omit aa1f7f11d939 [SPARK-53261][CORE][SQL] Use Java `String.join|StringJoiner` instead of Guava `Joiner`
    omit 68fdc9bb2925 [SPARK-53263][PYTHON] Support TimeType in df.toArrow
    omit 2fef901e1224 [MINOR][DOCS] Updated the docstring of DataStreamWriter.foreach() method
    omit 19c8f90bbbff [SPARK-53257][PYTHON][TESTS] Deduplicate have_graphviz and graphviz_requirement_message
    omit 04a2d0020472 [SPARK-53110][SQL][PYTHON][CONNECT] Implement the time_trunc function in PySpark
    omit d2b49662fa24 [SPARK-52981][PYTHON] Add table argument support for Arrow Python UDTFs
    omit 554f6b64f1e2 [SPARK-53246][TEST] remove class files for ReplSuite
    omit 205ed98513ca [SPARK-53259][PYTHON] Correct the message for INVALID_UDF_EVAL_TYPE
    omit d8dcfe778f33 [SPARK-53233][SQL][SS][MLLIB][CONNECT] Make the code related to `streaming` uses the correct package name
    omit 69185dce1e71 [SPARK-53258][CORE][SQL] Use `JavaUtils`'s `check(Argument|State)`
    omit 96a4f5097ad9 [SPARK-53074][SQL] Avoid partial clustering in SPJ to meet a child's required distribution
    omit c91066772b59 [SPARK-52482][SQL][CORE] Improve exception handling for reading certain corrupt zstd files
    omit df3d8e4dbecd [SPARK-53256][CORE] Promote `check(Argument|State)` to `JavaUtils`
    omit b248ba5e0089 [SPARK-53255][SQL] Ban `org.apache.parquet.Preconditions`
    omit 6537153e2c7b [SPARK-53138][CORE][BUILD] Split common-utils Java code into a new module
    omit 07c85a531f55 [SPARK-53253][PYTHON] Fix register UDF of type `SQL_SCALAR_ARROW_ITER_UDF`
    omit 1ae3d681983b [SPARK-53252][TESTS] Use Java `IntStream` instead of `ParSeq` in `CollationFactorySuite`
    omit 3f9917aa3d69 [SPARK-53250][BUILD] Remove unused `Guava` dependency from `unsafe` module
    omit 12e700c48f7b [SPARK-53249][INFRA] Run `build_maven_java21_arm.yml` every two days
    omit 7c2c84a20e08 [SPARK-53243][PYTHON][SQL] List the supported eval types in arrow nodes
    omit 7fe2f5ed8f50 [SPARK-53248][CORE] Support `checkedCast` in `JavaUtils`
    omit b82957c77b65 [SPARK-53176][DEPLOY] Spark launcher should respect `--load-spark-defaults`
    omit 0c0fd94da83e [SPARK-53247][CORE][SQL][MLLIB][TESTS] Use `createArray` for large test array creations
    omit 66ff752d8055 [SPARK-52008][FOLLOWUP] Fixing StateStoreCoordinator `warn` compilation messages from batch commit tracking
    omit 8d5e60279b10 [SPARK-53242][CORE][DSTREAM] Move `stackTraceToString` to `JavaUtils` and use it to replace `Throwables.getStackTraceAsString`
    omit 73f8a8483cdc [SPARK-53184][PS] `melt` when "value" has MultiIndex column labels
    omit 1bc8ce0b1f0a [SPARK-53244][SQL] Don't store dual-run enabled and tentative mode enabled confs during view creation
    omit b7ada565cac6 [MINOR][PYTHON] Remove two obsolete TODO items
    omit 221e076bdfb9 [SPARK-53241][CORE] Support `createArray` in `SparkCollectionUtils`
    omit f9093f241218 [SPARK-53147][SQL] Log generated JDBC query in JDBC connector
    omit 027d5c71575e [SPARK-52996][TESTS] Update brace-expansion to 1.1.12
    omit 3c4c2ee63d53 [SPARK-53240][SQL] Ban `com.google.common.collect.(ArrayList)?Multimap`
    omit d2dc05575294 [SPARK-53238][CORE][TESTS] Improve `SorterBenchmark` to include `Arrays.parallelSort`
    omit ba1496507044 [SPARK-53236][CORE][EXAMPLE] Use Java `ArrayList` constructors instead of `Lists.newArrayList` in Java code
    omit 7d96524892bb [SPARK-53239][SQL] Improve `MapSort` and `SortArray` performance via `parallelSort`
    omit 6eb078099739 [SPARK-53237][SQL] Use Java `Base64` instead of `org.apache.commons.codec.binary.Base64` instance
    omit e915312f5ab5 [SPARK-53192][CONNECT] Always cache a DataSource in the Spark Connect Plan Cache
    omit 6b71b22ca45c [SPARK-53235][CORE][TESTS] Use Java `Set.of` instead of `ImmutableSet.of`
    omit 54443540d6dc [SPARK-53234][CORE][SQL][MLLIB][YARN] Use `java.util.Objects` instead of `com.google.common.base.Objects`
    omit f4d10122fa22 [SPARK-53232][SQL][TESTS] Use Java `Map.copyOf` instead of `ImmutableMap.copyOf`
    omit 3bdcb2c1e295 [SPARK-53231][TESTS] Ban `com.google.common.collect.Sets`
    omit 08fe88aad043 [SPARK-53224][BUILD] Upgrade `joda-time` to 2.14.0
    omit 99b45b34624b [SPARK-53221][BUILD] Upgrade `commons-codec` to 1.19.0
    omit ca1990298d86 [SPARK-53229][CORE][SQL][EXAMPLES][TESTS] Use Java `Map.of` instead of `ImmutableMap.of`
    omit e994667644f0 [SPARK-53228][CORE][SQL] Use Java `Map` constructors instead of `Maps.new*HashMap()`
    omit 69a65b9daafe [SPARK-53223][BUILD] Upgrade `jersey` to 3.0.18
    omit 0b0944c5d902 [SPARK-53222][BUILD] Upgrade `commons-compress` to 1.28.0
    omit cb28e2c12169 [SPARK-53219][BUILD] Upgrade `Dropwizard` metrics to 4.2.33
    omit 0e34870ff878 [SPARK-53220][BUILD] Upgrade `dev.ludovic.netlib` to 3.0.4
    omit d5905afeaeba [SPARK-53227][SQL][TESTS] Use Java `HashMap.equals` instead of `Maps.difference.areEqual`
    omit e6e79b021ece [SPARK-53217][CORE][DSTREAM] Use Java `Set.of` instead of `Sets.newHashSet`
    omit 8f843a9a1632 [SPARK-53215][CORE][TESTS] Use `JavaUtils.listFiles` in `CleanupNonShuffleServiceServedFilesSuite`
    omit 4838f9979e0f [SPARK-53216][CORE] Move `is*(Blank|Empty)` from `object SparkStringUtils` to `trait SparkStringUtils`
    omit 423bca63dec4 [SPARK-53214][CORE][SQL][K8S] Use Java `HexFormat` instead of `Hex.encodeHexString`
    omit e36f9de67693 [SPARK-53213][CORE][SQL][K8S] Use Java `Base64` instead of `Base64.(decodeBase64|encodeBase64String)`
    omit f2510072c217 [SPARK-53211][TESTS] Ban `com.google.common.io.Files`
    omit 4693e09f4aff [SPARK-53218][BUILD] Upgrade `bouncycastle` to 1.81
    omit a8b8381f4473 [SPARK-53197][CORE][SQL] Use `java.util.Objects#requireNonNull` instead of `com.google.common.base.Preconditions#checkNotNull`
    omit ccff9980762f [SPARK-53210][CORE][SQL][DSTREAM][YARN] Use Java `Files.write(String)?` instead of `com.google.common.io.Files.write`
    omit d3f2054fa0ef [SPARK-53181][PS] Enable doc tests under ANSI
    omit f2568a855dbe [SPARK-53208][SQL][TESTS] Use `Hex.unhex` instead of `o.a.commons.codec.binary.Hex.decodeHex`
    omit 5c99c2b01e2e [SPARK-53205][CORE][SQL] Support `createParentDirs` in `SparkFileUtils`
    omit babc78d1ffbf [SPARK-53206][CORE] Use `SparkFileUtils.move` instead of `com.google.common.io.Files.move`
    omit 87af200bd7da [SPARK-53201][CORE] Use `SparkFileUtils.contentEquals` instead of `Files.equal`
    omit bc4493349936 [SPARK-53202][SQL][TESTS] Use `SparkFileUtils.touch` instead of `Files.touch`
    omit 16b70baa67d0 [SPARK-53200][CORE] Use Java `Files.newInputStream` instead of `Files.asByteSource().openStream()`
    omit 5cab61257b50 [SPARK-53199][SQL][TESTS] Use Java `Files.copy` instead of `com.google.common.io.Files.copy`
    omit 50948e4c47d6 [SPARK-53196][CORE] Use Java `OutputStream.nullOutputStream` instead of `ByteStreams.nullOutputStream`
    omit 8eaad0069e3f [SPARK-53195][CORE] Use Java `InputStream.readNBytes` instead of `ByteStreams.read`
    omit f7eee4ecf5cd [SPARK-53194][INFRA] Set -XX:ErrorFile to build/target directory for tests
    omit 96edcce564f6 [SPARK-53191][CORE][SQL][MLLIB][YARN] Use Java `InputStream.readAllBytes` instead of `ByteStreams.toByteArray`
    omit 8b80ea04d0b1 [SPARK-53190][CORE] Use Java `InputStream.transferTo` instead of `ByteStreams.copy`
    omit 76d2878b61bd [SPARK-53188][CORE][SQL] Support `readFully` in `SparkStreamUtils` and `JavaUtils`
    omit 69b45c654b58 [SPARK-53183][SQL] Use Java `Files.readString` instead of `o.a.s.sql.catalyst.util.fileToString`
    omit 6b1f1a6932f2 [SPARK-52976][PYTHON] Fix Python UDF not accepting collated string as input param/return type
    omit 44fd3587868e [SPARK-53185][CORE][YARN][TESTS] Use `SparkStreamUtils.toString` instead of `ByteStreams.toByteArray`
    omit c77f316cae60 [SPARK-53180][CORE] Use Java `InputStream.skipNBytes` instead of `ByteStreams.skipFully`
    omit 5d295c80b570 [SPARK-53178][BUILD] Upgrade `curator` to 5.9.0
    omit 106841389239 [SPARK-53179][CORE][TESTS] Use `SparkStreamUtils.toString` instead of `CharStreams.toString`
    omit 84858f3d8be0 [SPARK-52215][PYTHON][TESTS][FOLLOW-UP] Fix `test_arrow_udf_output_nested_arrays`
    omit b968d2a50596 [SPARK-53177][K8S] Use Java `Base64` instead of `com.google.common.io.BaseEncoding`
    omit a609db3fd992 [SPARK-53169][SQL] Remove comments related to "`Set the logger level of File Appender to`" from `log4j2.properties`
    omit dd925c091556 [SPARK-53168][CORE][TESTS] Change default value of the input parameter `level` for `SparkFunSuite#withLogAppender` from `None` to `Some(Level.INFO)`
    omit cffb8f6bcf37 [SPARK-53141][CORE] Add APIs to get overhead memory size and offheap memory size from resource profile
    omit 3aa8c9dbc1c0 [SPARK-53155][SQL] Global lower agggregation should not be replaced with a project
    omit 2420fe91ce98 [SPARK-53167][DEPLOY] Spark launcher isRemote also respects properties file
    omit 71afa4688447 [SPARK-53163][PYTHON][INFRA] Upgrade PyArrow to 21.0.0
    omit 89973d057699 [SPARK-53171][CORE] Improvement UTF8String repeat
    omit 598f310aed5d [SPARK-53170][CORE] Improve `SparkUserAppException` to have `cause` parameter
    omit 4430cd8544b1 [SPARK-53164][CORE][K8S][DSTREAM] Use Java `Files.readAllBytes` instead of `Files.toByteArray`
    omit 88dbe4297103 [SPARK-53066][SQL] Improve EXPLAIN output for DSv2 Join pushdown
    omit f62724d2d8dc [SPARK-52828][SQL] Make hashing for collated strings collation agnostic
    omit 7e7f967928a1 [SPARK-53166][CORE] Use `SparkExitCode.EXIT_FAILURE` in `SparkPipelines` object
    omit 7214029db67b [SPARK-53165][CORE] Add `SparkExitCode.CLASS_NOT_FOUND`
    omit 46fd5258a165 [SPARK-53162][SQL][TESTS] Mark `DynamicPartitionPruningHive*Suite*` as `SlowHiveTest`
    omit a72fd418242d [SPARK-52394][PS] Fix autocorr divide-by-zero error under ANSI mode
    omit e4ded0810119 [MINOR][CORE] Remove a wrong comment inside `SparkTestUtils.SparkTestUtils`
    omit acfe6d461400 [SPARK-52980][FOLLOWUP] Fix Python Arrow UDTF test cases
    omit 111c7c6d9ff3 [SPARK-53134][PS] Clean up unused ANSI imports in tests
    omit 6b4c3138c85e [SPARK-53160][CORE][SQL][MLLIB][K8S][YARN][SS] Use Java `Files.writeString` instead of `Files.asCharSink`
    omit a09ea772e374 [SPARK-53098][SQL][TEST-ONLY][FOLLOWUP] Fix a test when returning single-pass result
    omit 49d542d1d951 [SPARK-52593][PS] Avoid CAST_INVALID_INPUT of `MultiIndex.to_series`, `Series.dot` and `DataFrame.dot` in ANSI mode
    omit 9a458eb8b6f0 [SPARK-53152][CORE][K8S][YARN] Use Java `Files.readString` instead of `Files.asCharSource`
    omit 7db6c8287b08 [SPARK-53153][CORE][TESTS] Use Java `Files.readAllLines` in `PluginContainerSuite`
    omit 01d695365d3d [SPARK-53150][CORE] Improve `list(File|Path)s` to handle non-existent, non-directory, symlink input
    omit 19ea6ff95ef3 [SPARK-53130][SQL][PYTHON] Fix `toJson` behavior of collated string types
    omit f6cd38598078 [SPARK-52943][PYTHON] Enable arrow_cast for all pandas UDF eval types
    omit ea8b6fdde977 [SPARK-53107][SQL] Implement the time_trunc function in Scala
    omit 70c20087b709 [SPARK-53097][CONNECT][SQL] Make WriteOperationV2 in SparkConnectPlanner side effect free
    omit cf3ab766c8c2 [SPARK-53136][CORE] tryWithResource & tryInitializeResource shall close resource quietly
    omit 971d318cae8e [SPARK-51585][SQL] Oracle dialect supports pushdown datetime functions
    omit 95a96891ac6e [SPARK-53140][TESTS] Ban `org.apache.commons.io.FileUtils`
    omit 2f1fde501227 [SPARK-53137][CORE][SQL] Support `forceDeleteOnExit` in `SparkFileUtils` and `JavaUtils`
    omit bca5f3a0f09e [SPARK-53132][CORE][TESTS][FOLLOWUP] Use `Utils.listFiles` in `BlockManagerDecommissionIntegrationSuite`
    omit e964086649f1 [SPARK-52942][YARN][BUILD] YARN External Shuffle Service jar should include `scala-library`
    omit 08e734e6fb71 [SPARK-43100][CORE] Push Based Shuffle metrics must be deserialized correctly
    omit d2187f4b4ea1 [SPARK-53135][CORE][SQL] Support `copyURLToFile` in `SparkFileUtils` and `JavaUtils`
    omit d53de53fff55 [SPARK-52989][SS] Add explicit close() API to State Store iterators
    omit a895d553fe57 [SPARK-53133][SQL][TESTS] Use Java `System.getProperty` instead of `FileUtils.getTempDirectory`
    omit f4640734ef8b [SPARK-53132][CORE][SQL][MLLIB] Support `list(File|Path)s` in `SparkFileUtils` and `JavaUtils`
    omit ca02481b6be2 [SPARK-52971][PYTHON] Limit idle Python worker queue size
    omit 3f0c4506f43a [SPARK-53094][SQL] Fix CUBE with aggregate containing HAVING clauses
    omit af4e1d91b76a [SPARK-53104][PS] Introduce ansi_mode_context to avoid multiple config checks per API call
    omit 5180c0c51de1 [SPARK-53069][SS] Fix incorrect state store metrics with virtual column families
    omit 0491d7e32bb9 [SPARK-53131][SHELL] Improve `SparkShell` to import `java.nio.file._` by default
    omit 48367c125b9d [SPARK-53129][SHELL] Improve `SparkShell` to import `java.net._` by default
    omit 0c885bfd14b1 [SPARK-53126][SQL][CONNECT] Refactor `SparkErrorUtils#stackTraceToString` to handle `null` input and use it to replace `ExceptionUtils#getStackTrace`
    omit dc46b1479450 [SPARK-53064][CORE] Rewrite MDC LogKey in Java
    omit d72e02897617 [SPARK-53077][CORE][TESTS][FOLLOWUP] Reduce insertion count in SparkBloomFilterSuite
    omit 26d668d6ec27 [SPARK-51462][SPARK-51564][SQL][FOLLOWUP][TESTS] Add E2E SQL tests for TIME literals in 24hr and 12hr formats
    omit 3bdcf90a7f54 [SPARK-53123][CORE][SQL] Support `getRootCause` in `SparkErrorUtils`
    omit ef30c3aaa63e [SPARK-53067][BUILD] Increase GCLockerRetryAllocationCount for SBT
    omit 4a2ce9f7b0df [SPARK-53098][SQL] `DeduplicateRelations` shouldn't remap expressions if old `ExprId` still exists in output
    omit 9932e8ccf715 [SPARK-53122][CORE] Support `moveFile` in `SparkFileUtils` and `JavaUtils`
    omit faa6723125d0 [SPARK-53121][SQL] Use `deleteRecursively` instead of `FileUtils.forceDelete`
    omit 84becfbb73a6 [SPARK-53119][CORE] Support `touch` in `SparkFileUtils`
    omit bccbb5130d17 [SPARK-53120][DOCS] Recover _source directory for PySpark documentation
    omit 10492a00633c [SPARK-53118][CORE][TESTS] Use `Files.write` in `GenericFileInputStreamSuite`
    omit 93fc2ed9fb1b [SPARK-52980][PYTHON] Support Arrow Python UDTFs
    omit 1f4f5401d072 [SPARK-53117][CORE][SQL] Support `moveDirectory` in `SparkFileUtils` and `JavaUtils`
    omit 50882d2d8886 [SPARK-53115][TESTS] Ban `org.apache.commons.lang3.StringUtils`
    omit ff06fa2b6e27 [SPARK-53114][CORE][SQL] Support `join` in `JavaUtils`
    omit 8e99e4ee3d5d [SPARK-53101][CORE][SQL][MLLIB] Support `(left|right)Pad` in `SparkStringUtils`
    omit 14c62c0dd6d4 [SPARK-53072][TESTS] Only run Lock Hardening tests in RocksDBStateStoreLockHardeningSuite
    omit 35c299a1e3e3 [SPARK-53002][SS] Show accurate memory usage statistics when bounded memory is enabled for RocksDB
    omit e3cab3ee434a [SPARK-53102][BUILD] Upgrade `commons-cli` to 1.10.0
    omit c28b51b4dae3 [SPARK-53095][SQL] Support HMS v4.1
    omit ba85b25e3867 [SPARK-53100][SQL] Use Java `String.substring` instead of `StringUtils.substring`
    omit 8342d7438114 [SPARK-52974][SQL] Don't broadcast dynamicpruning InSubqueries
    omit e38c5b582440 [SPARK-53040][SQL] Ban self references inside topmost CTEs in Recursive CTEs
    omit 27aba95bdf76 [SPARK-52451][CONNECT][SQL] Make WriteOperation in SparkConnectPlanner side effect free
    omit 87016ce2dd53 [SPARK-53092][CORE][CONNECT] Ban `org.apache.commons.lang3.SystemUtils`
    omit 635c24295c85 [SPARK-53096][K8S] Support `spark.kubernetes.executor.terminationGracePeriodSeconds`
    omit ced87308a23e [SPARK-53077][CORE][TESTS] Add an env to control the execution of the `SparkBloomFilterSuite`
    omit 6ef9a9d34053 [SPARK-51555][SQL][FOLLOWUP] Fix the error message for invalid unit in the time_diff() function
    omit 09d2765e56ca [SPARK-53045][SQL] DESCRIBE EXTENDED should be resilient to corrupt metadata
    omit c9b85a2d6008 [SPARK-53063][CORE] Implement and call new APIs in FileCommitProtocol instead of the deprecated
    omit e19ef4994ce7 [SPARK-53093][K8S][DOCS] Drop K8s v1.31 Support
    omit 34d7a3c5ccf6 [SPARK-52975][SQL] Simplify field names in pushdown join sql
    omit 712ef0858525 [SPARK-53091][SQL] Ban `org.apache.commons.lang3.Strings`
    omit 1de64a4f16b1 [SPARK-53032] Fix parquet format of shredded timestamp values within arrays
    omit a7e9d93b2a2c [SPARK-53089][BUILD] Upgrade `Ammonite` to 3.0.2
    omit a32bfd7be493 [SPARK-53008][SQL] Infer SQL Data Access from SQL UDF body
    omit f3a08e7036a7 [SPARK-53088][BUILD] Remove `scala-collection-compat` dependency
    omit a8e98340be51 [SPARK-53090][CORE][SS][TESTS] Use Java `OutputStream.write` instead of `IOUtils.write`
    omit c23ce6840453 [SPARK-53087][BUILD] Upgrade `Jackson` to 2.19.2
    omit 1641b30c7247 [SPARK-53086][TESTS] Revise Scalastyle RegexChecker patterns
    omit 21fd32536265 [SPARK-53079][K8S][DOCS] Update `YuniKorn` docs with `1.7.0`
    omit bb2a25ed93fc [SPARK-53080][CORE][CONNECT] Support `cleanDirectory` in `SparkFileUtils` and `JavaUtils`
    omit 3ed7de4cb885 [SPARK-53081][CORE][SQL][CONNECT] Support `contentEquals` in `SparkFileUtils`
    omit cb9cb7cd10c5 [SPARK-53083][CORE][SQL][K8S][SS][TESTS] Use Java `Files.writeString` instead of `FileUtils.write`
    omit b206304b897a [SPARK-53082][CORE][SQL][TESTS] Use Java `Files.readString` instead of `FileUtils.readFileToString`
    omit cd6cda135def [SPARK-53078][CORE][TESTS] Remove `commons-(io|lang3)` test dependencies from `kvstore`
    omit 36a77f872007 [SPARK-53076][CORE][TESTS] Disable `SparkBloomFilterSuite`
    omit 4a8c4be79918 [SPARK-53073][CORE][SQL][YARN][SS] Support `copyDirectory` in `SparkFileUtils` and `JavaUtils`
    omit 322b3d06d3f8 [SPARK-53075][CORE][TESTS] Use Java `Files.readAllLines/write` instead of `FileUtils.(read|write)Lines`
    omit 2513f4123539 [SPARK-53071][CORE][SQL][CONNECT] Support `copyFile` in `SparkFileUtils`
    omit 443e170397a5 [SPARK-53070][CORE] Support `is(Not)?Empty` in `SparkCollectionUtils`
    omit fd5ed10d01dc [SPARK-52503][SQL][CONNECT][TESTS][FOLLOW-UP] Add more unit tests for `df.drop` on semantic equality
    omit bd76faf83bff [SPARK-53068][SQL][TESTS] Mark `TransformWith*Suite` as `SlowSQLTest` in `sql/core`
    omit 6e7c67fef7af [SPARK-53058][SQL][DOCS] Clean up old comments related to HashClusteredDistribution
    omit 0225d4cf76cc [SPARK-53001][CORE][SQL][FOLLOW-UP] Disable `spark.memory.unmanagedMemoryPollingInterval` by default
    omit 0cd2f9019a09 [SPARK-53035][TESTS][FOLLOWUP] Use `String.repeat` in `tests` too
    omit a0eecf6e5dc5 [SPARK-52968][SS] Emit additional state store metrics
    omit 7c96b3e36e3d [SPARK-53061][CORE][SQL] Support `copyFileToDirectory` in `SparkFileUtils`
    omit 49dc0a7ab72a [SPARK-53059][PYTHON] Arrow UDF no need to depend on pandas
    omit 6b0d2ca00121 [SPARK-53062][CORE] Support `deleteQuietly` in `SparkFileUtils` and `JavaUtils`
    omit 032677cb329d [SPARK-52928][PYTHON][FOLLOW-UP] Remove unreachable code after upgrading pyarrow minimum version to 15.0.0
    omit 3839fed74c20 [SPARK-52772][SQL] Inconsistent table attribute handling during updates
    omit 0dd0e022b411 [SPARK-53054][CONNECT] Fix the connect.DataFrameReader default format behavior
    omit 92df232addc5 [SPARK-53057][CORE] Support `sizeOf` in `SparkFileUtils` and `JavaUtils`
    omit fbebc20e61d1 [SPARK-53056][BUILD] Upgrade `objenesis` to 3.4
    omit d0a8b18c1adb [SPARK-53014][PYTHON][DOCS][FOLLOW-UP] Add `arrow_udf` to API reference
    omit 57dfc0a21bf2 [SPARK-53039][PYTHON][TESTS] Add unit test for complex arrow UDF used in window
    omit 133d9f60e72f [SPARK-53055][PS][TESTS] Remove sort_values from pyspark.pandas.tests.indexes.test_insert
    omit 8c90e6fa77c4 [SPARK-53049][CORE][SQL][SS] Support `toString` in `SparkStreamUtils`
    omit a75e1aa1cac2 [SPARK-53001] Integrate RocksDB Memory Usage with the Unified Memory Manager
    omit 3024c7c21db8 [SPARK-53051][CORE][TESTS] Ban `org.apache.hadoop.io.IOUtils`
    omit fd17a8d06e23 Revert "[SPARK-52877][PYTHON][FOLLOW-UP] Use columns instead of itercolumns in RecordBatch"
    omit 707d9eef6bfa [SPARK-53046][CORE][SQL] Use Java `readAllBytes` instead of `IOUtils.toByteArray`
    omit c5789579de20 [SPARK-53036][PYTHON][DOCS][TESTS] Enable doctest `pyspark.sql.pandas.functions`
    omit f45106286f38 [SPARK-53043][CORE][SQL][K8S] Use Java `transferTo` instead of `IOUtils.copy`
    omit 9c35c43eaf3b [SPARK-52580][PS] Avoid CAST_INVALID_INPUT of `replace` in ANSI mode
    omit e9a285eaf273 Revert "[SPARK-52610][BUILD] Upgrade rocksdbjni to 10.2.1"
    omit 635fa9be4281 [MINOR][CORE][TESTS] Rename `TestMemoryManager.(markconsequentOOM -> markConsequentOOM)`
    omit 1ce28ccfb7de [SPARK-53034][CORE][TESTS][FOLLOWUP] Use `JavaUtils.deleteRecursively` in Java code
    omit 6a6317d6fd2f [SPARK-53037][CORE][SQL][K8S] Support `closeQuietly` in `SparkErrorUtils`
    omit 293342da1185 [SPARK-53017][SQL][FOLLOWUP] Add logs in DSv2 Join pushdown rule
    omit 605e5203a16f [SPARK-52926][SQL] Added SQLMetric for remote schema fetching time duration
    omit 800d3729c9c2 [SPARK-53035][CORE][SQL][K8S][MLLIB] Use `String.repeat` instead of Scala string multiplication
    omit d892ca40fa66 [SPARK-53034][CORE][SQL][MLLIB][YARN] Use `deleteRecursively` instead of `FileUtils.deleteDirectory`
    omit b03a86923c0d [SPARK-51555][SQL] Add the time_diff() function
    omit 2e3e0f571986 [SPARK-52940][PYTHON][TESTS][FOLLOW-UP] Correct the return type in test UDFs
    omit 388335d0d72c [MINOR][PYTHON][DOCS] Fix a pandas UDF example
    omit 7c6693ddec00 [SPARK-53031][CORE] Support `getFile` in `SparkFileUtils`
    omit 1f425c7433b4 [SPARK-53028][SQL][K8S][EXAMPLE] Use `StandardCharsets.UTF_8` instead of `Charset.defaultCharset`
    omit 3738cb3dd701 [SPARK-53025][CORE][SQL][TESTS] Use Java `Files.writeString` instead of `FileUtils.writeStringToFile`
    omit dd6525acd927 [SPARK-53023][SQL] Remove `commons-io` dependency from `sql/api` module
    omit b4c5291fbcb8 [SPARK-53014][PYTHON][DOCS] Make Arrow UDF public
    omit 39f8331cda44 [SPARK-53013][PYTHON] Fix Arrow-optimized Python UDTF returning no rows on lateral join
    omit 6a7a3debda07 [SPARK-53024][BUILD] Upgrade `commons-io` to 2.20.0
    omit 61a79ca6ba9b [SPARK-52622][PS] Avoid CAST_INVALID_INPUT of `DataFrame.melt` in ANSI mode
    omit dc8fba647ac1 [SPARK-53021][BUILD] Switch `common/utils` module to have `commons-io` test dependency
    omit 9a1c1ab4710a [SPARK-52008] Throwing an error if State Stores do not commit at the end of a batch when ForeachBatch is used
    omit e242bcfc1183 [SPARK-53020][DEPLOY] JPMS args should also apply to non-SparkSubmit process
    omit ba2346e19bee [SPARK-52967][BUILD] Upgrade ORC to 2.2.0
    omit 94309f216db4 [SPARK-53018][PYTHON] ArrowStreamArrowUDFSerializer should respect argument arrow_cast
    omit 5cf3c3204440 [SPARK-52954][PYTHON][TESTS][FOLLOW-UP] Alway set safe_check=True in Arrow UDFs
    omit 3654fd359750 [SPARK-51554][SQL] Add the time_trunc() function
    omit 9b927449825a [SPARK-53003][CORE][FOLLOWUP] Handle null input values
    omit 296d0f9bed00 [SPARK-53004][CORE] Support `abbreviate` in `SparkStringUtils`
    omit 95abf2fc0972 [SPARK-53010][MLLIB][YARN] Ban `com.google.common.base.Strings`
    omit 783beb648d0e [SPARK-53006][BUILD][TESTS] Upgrade `bytebuddy` to 1.17.6
    omit 161f447d6432 [SPARK-53003][CORE] Support `strip` in `SparkStringUtils`
    omit 42553b94e368 [SPARK-52992][PYTHON][DOCS] Restore API reference of `pandas_udf`
    omit 349c8637f367 [SPARK-52985][PS] Raise TypeError for pandas numpy operand in comparison operators
    omit 2d1f77f9f744 [SPARK-52646][PS] Avoid CAST_INVALID_INPUT of `__eq__` in ANSI mode
    omit c3fa01032fcc [SPARK-53000][SQL] Rename `StringUtils.scala` to `StringConcat.scala` in `sql/api` module
    omit 475d72f61c92 [SPARK-52999][K8S][TESTS] Clean up the deprecated APIs usage in `kubernetes-integration-tests` module
    omit 3bf3696461ab [SPARK-51745][SQL][SS] Enforce State Machine for RocksDBStateStore
    omit 2e9d64295aa9 [SPARK-52995][YARN] Use Buffered I/O for creating spark jar archive
    omit 1dccc5341d98 [SPARK-52959][PYTHON] Support UDT in Arrow-optimized Python UDTF
    omit 1c2d0620390e [SPARK-52990][CORE] Support `StringSubstitutor`
    omit aeeb68bd99e3 [SPARK-52952][PYTHON] Add PySpark UDF Type Coercion Dev Script
    omit ab1f6ed2d06e [SPARK-52889][PYTHON] Implement the current_time function in PySpark
    omit 7a6a6e66df20 [SPARK-52960][SQL] Show subtree string in LogicalQueryStage toString
    omit 8274b7c78455 [SPARK-52993][BUILD] Bump Snappy 1.1.10.8
    omit 26ff6dff8b09 [SPARK-52977][TESTS] Fix npm vulnerabilities by `npm audit fix`
    omit 633ffe4f3744 [SPARK-52972][PYTHON][INFRA] Remove the testing image for python 3.9
    omit 8348f2a845f1 [SPARK-52983][BUILD] Upgrade Netty to 4.1.123.Final
    omit 9eee6bf6be85 [SPARK-52936][INFRA][TESTS] Benchmark result update automation
    omit c7d780b0bb3f [SPARK-52987][SQL][K8S] Use Java `String.(equals|replace)` method instead of `commons-lang3`
    omit e65341397b2e [SPARK-52956][SQL] Preserve alias metadata when collapsing projects
    omit 13c43bc4fd2c [SPARK-52890][SPARK-52891][PYTHON] Implement the to_time and try_to_time functions in PySpark
    omit cf3a34e19dfc [SPARK-52921][SQL] Specify outputPartitioning for UnionExec for same output partitoning as children operators
    omit 9a452f81dbdd [SPARK-52945][SQL][TESTS] Split `CastSuiteBase#checkInvalidCastFromNumericType` into three methods and guarantee assertions are valid
    omit 2817654d439b [SPARK-52892][SPARK-52893][SPARK-52894][PYTHON] Support TIME in the hour, minute, and second functions in PySpark
    omit cdb4f713402c [SPARK-52888][PYTHON] Implement the make_time function in PySpark
    omit f523fdb451c3 [SPARK-52973][TESTS] Fix the execution failure of StateStoreBasicOperationsBenchmark
    omit 570b32500844 [SPARK-52689][SQL] Send DML Metrics to V2Write
    omit 3ff28ae4ef43 [SPARK-52146][SQL] Detect cyclic function references in SQL UDFs
    omit 8819828a4013 [SPARK-52853][TESTS][FOLLOW-UP] Import SDP module when connect dependencies are available
    omit dd36f61decd4 [SPARK-52970][CONNECT][INFRA] Upgrade PyArrow version in 3.5 client <> master server build
    omit 0c4a36f392b0 [SPARK-52954][PYTHON] Arrow UDF support return type coercion
    omit f23703de658b [SPARK-51415][SQL] Support the time type by make_timestamp()
    omit 4dc3f0fcf987 [SPARK-52933][K8S] Verify if the executor cpu request exceeds limit
    omit cdc89aea9ac6 [SPARK-52961][PYTHON] Fix Arrow-optimized Python UDTF with 0-arg eval on lateral join
    omit 5a72a6ae7c81 [SPARK-52904][PYTHON] Enable convertToArrowArraySafely by default
    omit afd595a57f1d [SPARK-52944][CORE][TESTS][FOLLOWUP] Avoid hard-coding the checksum algorithm name
    omit a823f95c5220 [SPARK-52962][SQL] BroadcastExchangeExec should not reset metrics
    omit aeae9ff7bfbb [SPARK-52737][CORE] Pushdown predicate and number of apps to FsHistoryProvider when listing applications
    omit 25892a77ff9c [SPARK-52949][PYTHON] Avoid roundtrip between RecordBatch and Table in Arrow-optimized Python UDTF
    omit 8b6b28b5af3b [SPARK-52141][SQL] Display constraints in DESC commands
    omit 90fd991d992b [SPARK-52944][CORE][SQL][YARN] Fix invalid assertions in tests
    omit d35399acb3d3 [SPARK-52955] Change return types of WindowResolution.resolveOrder and WindowResolution.resolveFrame to WindowExpression
    omit 30585c1d69dd [SPARK-49968][SQL] The split function produces incorrect results with an empty regex and a limit
    omit 131898f51c20 [SPARK-50614][FOLLOW-UP] Add assert(false) to test in catch block
    omit 3ae3e344da07 [SPARK-52950][SQL][TESTS] Enable ANSI mode in DataSourceV2StrategySuite
    omit d148e9be24f4 [SPARK-52877][PYTHON][FOLLOW-UP] Use columns instead of itercolumns in RecordBatch
    omit 8c538921a8ae [SPARK-52840][PYTHON][DOCS][FOLLOW-UP] Increase Pandas minimum version to 2.2.0
    omit e8015e9a89f9 [SPARK-52928][PYTHON][INFRA] Upgrade the minimum version of pyarrow to 15.0.0
    omit d57dc7de62a1 [SPARK-52948][PS] Enable test_np_spark_compat_frame under ANSI
    omit f9347bc18ddf [SPARK-52905][PYTHON][FOLLOW-UP] ArrowWindowPythonExec code clean up
    omit eb63949298b3 [SPARK-52908][CORE] Prevent for iterator variable name clashing with names of labels in the path to the root of AST
    omit 37137257049a [SPARK-52947][SDP] Fix image path in declarative pipelines programming guide
    omit b8c6854d7867 [SPARK-50889][CONNECT][TESTS] Fix Flaky Test: `SparkSessionE2ESuite.interrupt operation` (Hang)
    omit ff980cc4aefa [SPARK-52940][PYTHON][TESTS] Add arrow aggregation unit tests for complex return types
    omit a82b4158d448 [SPARK-52946][PYTHON] Fix Arrow-optimized Python UDTF to support large var types
    omit dd2ee2400e85 [SPARK-52934][PYTHON] Allow yielding scalar values with Arrow-optimized Python UDTF
    omit acdec9bafb8a [SPARK-52592][PS] Support creating a ps.Series from a ps.Series
    omit dc687d4c83b8 [SPARK-52853][SDP] Prevent imperative PySpark methods in declarative pipelines
    omit 5a9929c32ef8 [SPARK-52918][SQL][TESTS] Batch JDBC database statements in JDBC suites
    omit 4dc426085d20 [SPARK-52939][TESTS] Add unit tests for ArrowWriter with User Defined Type
    omit 0802097c4767 [SPARK-52837][CONNECT][PYTHON] Support TimeType in pyspark
    omit 3bba8c892e66 [SPARK-52903][SQL][FOLLOWUP] Fix query failure with LCAs on structs
    omit f003453a6117 [SPARK-52882][SQL] Implement the current_time function in Scala
    omit 8b889ce1cbf3 [SPARK-52897][PYTHON] Update `pandas` to 2.3.1
    omit 276fede8c686 [SPARK-7008][INFRA][PS] Upgrade pyarrow to 15.0 in image python-ps-minimum
    omit c4c0db027cfe [SPARK-52686][SQL][FOLLOWUP] Don't push `Project` through `Union` if there are duplicates in the project list
    omit ae75efeb407f [SPARK-51505][SQL] Always show empty partition number metrics in AQEShuffleReadExec
    omit 867250f68a1d [SPARK-52925][SQL] Return correct error message for anchor self references in rCTEs
    omit 64cada1d25aa [SPARK-52709][SQL] Fix parsing of  STRUCT<>
    omit 479410594fb5 [SPARK-52935][PYTHON][CONNECT][TESTS] Enable Arrow window tests in connect
    omit 40f3ea7c6258 [SPARK-52927][PYTHON][INFRA] Upgrade pyarrow to 15.0 in image `python-minimum`
    omit 79ba12afdbfe [SPARK-52929][SQL] Support MySQL and SQLServer connector for DSv2 Join pushdown
    omit e824f88c40a9 [SPARK-52821][PYTHON] add int->DecimalType pyspark udf return type coercion
    omit 634362cbe2d5 [SPARK-52147][SQL][TESTS] Block temporary object references in persistent SQL UDFs
    omit 97f04618f0b5 [SPARK-52914][CORE] Support `On-Demand Log Loading` for rolling logs in `History Server`
    omit a81d79256027 [SPARK-52680][DOCS][FOLLOW-UP] Update migration guide about Python 3.9 dropped
    omit 03cb4d9d6874 [SPARK-52883][SPARK-52884][SQL] Implement the to_time and try_to_time functions in Scala
    omit e888e37ee2eb [SPARK-52924][CORE] Support ZSTD_strategy for compression
    omit a08d8b093c0e [SPARK-47547][CORE] Add `BloomFilter` V2 and use it as default
    omit 23a19e6b5b03 [SPARK-52905][PYTHON] Arrow UDF for window
    omit f34563442a7c [SPARK-52919][SQL] Fix DSv2 Join pushdown to use previously aliased column
    omit 19cedecc36f8 [SPARK-52751][PYTHON][CONNECT] Don't eagerly validate column name in `dataframe['col_name']`
    omit c7c1021693a0 [SPARK-52300][SQL][TEST] Fix invalid AnalysisConfOverrideSuite
    omit e31ea9f1645f [SPARK-52920][DOCS] Improve memory profiler doc to clarify generator UDFs are unsupported
    omit 5182eb4c6a51 [SPARK-52791][PS] Fix error when inferring a UDT with a null first element
    omit 125c79aec851 [SPARK-52424][TESTS] Drop table created in PySpark test_streaming_foreach_batch_external_column
    omit 4de866146228 [SPARK-52877][PYTHON] Improve Python UDF Arrow Serializer Performance
    omit a8111b222340 [SPARK-52875][SQL] Simplify V2 expression translation if the input is context-independent-foldable
    omit 5b622c53dcfa [SPARK-52917][SQL] Read support to enable round-trip for binary in xml format
    omit 47b08a0e9588 [SPARK-52832][SQL][FOLLOWUP] Fix comment in `MySQLDialect::quoteIdentifier`
    omit c2ff983145a4 [SPARK-48230][BUILD] Remove unused `jodd-core`
    omit 628422027be0 [SPARK-52906 ][SQL] Support DSv2 Join pushdown for Postgres connector
    omit 77dc7f3deb15 [SPARK-52903][SQL] Trim non-top-level aliases before LCA resolution
    omit 75721ad9629e [SPARK-52866][SQL] Add support for `try_to_date`
    omit 1c5908e84639 [SPARK-50614][FOLLOW-UP] Fix bug where shredded timestamp values did not conform to the Parquet Variant Shredding spec
    omit d029adad91cd [SPARK-52916][BUILD] Exclude slf4j-simple from SBT
    omit 3e0d2ebb8d7d [SPARK-52860][SQL][TEST] Support V2 write schema evolution in InMemoryTable
    omit 1a2977e289ac [SPARK-52881][SQL] Implement the make_time function in Scala
    omit ea83a17a43f4 [SPARK-52823][SQL] Support DSv2 Join pushdown for Oracle connector
    omit a2a9d4092e00 [SPARK-52852][SDP] Remove unused spark_conf in create_streaming_table
    omit a50fbf76ba56 [SPARK-52869][SQL] Add FrameLessOffsetWindowFunction validation to validateResolvedWindowExpression for reuse in single-pass analyzer
    omit 1a8c26c3f67e [SPARK-52829][PYTHON][FOLLOWUP] Remove unnecessary special handling
    omit b14eb1e8d6cb [SPARK-52804][BUILD][FOLLOWUP] Revert Java minimum version check for Maven
    omit ddf84c87972a [SPARK-52912][CORE] Improve `SparkStringUtils` to support `is(Not)?(Blank|Empty)`
    omit 7255e2cc8395 [SPARK-52771][PS] Fix float32 type widening in `truediv`/`floordiv`
    omit 39fbf594fa73 [SPARK-52448][CONNECT] Add simplified Struct Expression.Literal
    omit 4a4505457544 [SPARK-52851][SDP] Remove public APIs for append once flows
    omit 9a60a5408f1d [SPARK-52911][SQL] Remove `StringUtils.(split|chop)` usage
    omit 5d0556bae2c3 [SPARK-52902][K8S] Support `SPARK_VERSION` placeholder in container image names
    omit 27dcbcd4b075 [SPARK-52880][CORE] Improve `toString` by `JEP-280` instead of `ToStringBuilder`
    omit 0177265b6cb9 [SPARK-52432][SDP][SQL] Scope DataflowGraphRegistry to Session
    omit 689e4580e143 [SPARK-52846][SQL] Add a metric in JDBCRDD for how long it takes to fetch the resultset
    omit 386e4646cff4 [SPARK-52870][SQL] Properly quote variable names in `FOR` statement
    omit f8c2671ada36 [SPARK-52872][SQL][TESTS] Improve test coverage for `HigherOrderFunctions`
    omit c67a774f084c [SPARK-52895][SQL] Don't add duplicate elements in `resolveExprsWithAggregate`
    omit 45374d8950fb [SPARK-52899][SQL] Fix QueryExecutionErrorsSuite test to register H2Dialect back
    omit ddea7dd2be71 [SPARK-52885][SPARK-52886][SPARK-52887][SQL] Implement the hour, minute, and second functions in Scala for the TIME type
    omit 02e0ab31a8d8 [SPARK-52901][SQL][K8S] Fix deprecated nested class shadowing warnings
    omit 216394081072 [SPARK-52900][CORE] Use `SparkStringUtils.stringToSeq` in `FsHistoryProvider`
    omit 04645dc57885 [SPARK-52862][PYTHON][CONNECT] Fix the nullability check of `Decimal('NaN')`
    omit 79cee25d1d9a [MINOR] Remove unnecessary df.show() in tests
    omit c7176246a21a [SPARK-52800][SQL][K8S] Remove several deprecated commons-lang3 methods
    omit a0fb5f0fff80 [SPARK-51564] TIME parsing in the 12hr clock format
    omit 656b97fc752f [SPARK-52876][CORE] Fix a typo `buffer` to `body` in `ChunkFetchSuccess.toString`
    omit 984a2ec6eec3 [SPARK-52142][SQL] Display table constraints in SHOW CREATE TABLE COMMAND
    omit 3d99b0b397e6 [SPARK-51955] Adding release() to ReadStateStore interface and reusing ReadStore for Streaming Aggregations
    omit 72ce64ef9daf [SPARK-52817][SQL] Fix `Like` Expression performance
    omit 7cc28969bf27 [SPARK-52874][CORE] Support `o.a.s.util.Pair` Java Record
    omit 51494a7f1216 [SPARK-51596][SS] Fix concurrent StateStoreProvider maintenance and closing
    omit 537f8136c571 [SPARK-52861][PYTHON] Skip Row object creation in Arrow-optimized UDTF execution
    omit 2ba0eab9db25 [SPARK-51562][SQL] Add `time` function
    omit c57556c9769f [SPARK-52787][SS] Reorganize streaming execution dir around runtime and checkpoint areas
    omit b7df1874b269 [SPARK-52871][SQL] Merge `o.a.s.sql.catalyst.util.SparkStringUtils` to `o.a.s.util.SparkStringUtils`
    omit f05a9080fae5 [SPARK-52841][SQL][TESTS] Fix PlanStabilitySuite id normalization
    omit 63e58d24269f [SPARK-52859] Add `SparkSystemUtils` trait
    omit 2bc575785465 [SPARK-52832][SQL] Fix JDBC dialect identifier quoting
    omit 3d77f26823b2 [SPARK-52850][PYTHON] Skip calling conversions if identity function
    omit 7bbdda2482e2 [SPARK-52863][PYTHON] Clean up code paths for old pandas versions
    omit eadf1a4431ea [SPARK-52848][SQL] Avoid cast to `Double` in casting TIME/TIMESTAMP to DECIMAL
    omit 197c9d6051ef [SPARK-52516][SQL] Don't hold previous iterator reference after advancing to next file in ParquetPartitionReaderFactory
    omit 2297cf436fc4 [SPARK-52842][SQL] New functionality and bugfixes for single-pass analyzer
    omit 93748ccc5979 [SPARK-52511][SDP] Support dry-run mode in spark-pipelines command
    omit 1570206f58bc [SPARK-52849][CORE] Add `stringifyException` to `o.a.s.util.Utils`
    omit 35596e81408a [SPARK-52833][SQL] Fix `VariantBuilder.appendFloat`
    omit faae05a4a2c2 [SPARK-52843][PYTHON][TESTS] Support retry timeout tests
    omit 3457df6433cc [SPARK-52641][SQL] Primary key columns should not be nullable
    omit 098204127c32 [SPARK-52637][SS] Fix version ID mismatch issue for RocksDB compaction leading to incorrect file mapping
    omit 9204b0558945 [SPARK--52810][SDP][SQL] Spark Pipelines CLI Selection Options
    omit 169b47fd6b4d [SPARK-52575][SQL] Introduce contextIndependentFoldable attribute for Expressions
    omit 28f2a5cc6af0 [SPARK-52171][SS] StateDataSource join implementation for state v3
    omit afa9b0c3aee5 [SPARK-52620][SQL] Support casting of TIME to DECIMAL
    omit 7a9310887378 [SPARK-52845][BUILD] Upgrade maven to 3.9.11
    omit a9da24c3cb91 [SPARK-52840][PYTHON][DOCS] Increase Pandas minimum version to 2.2.0
    omit e65bd5f5fe6e [SPARK-52839][PYTHON][CONNECT][TESTS] Enable Arrow aggregation tests in connect
    omit efbb06f40d02 [MINOR][CORE] Fix a missing space in log
    omit 4cc8101cb5e0 [SPARK-52830][K8S] Support `spark.kubernetes.(driver|executor).pod.excludedFeatureSteps`
    omit c095b95decc8 [SPARK-52824][SS] CheckpointFileManager error classification
    omit 09dd5dd063b6 [SPARK-52214][PYTHON] Arrow UDF for aggregation
    omit 4bcf743c60ae [SPARK-52795][SS] Include runId in state store logs
    omit 9f7a0d3b1575 [SPARK-52820][SQL] Capture full plans in golden files
    omit 5a05f672f710 [SPARK-52792][CORE][FOLLOWUP] Remove redundant condition check
    omit 5e6b4df13265 [SPARK-52829][PYTHON] Fix LocalDataToArrowConversion.convert to handle empty rows properly
    omit 5e9c02161aab [SPARK-52811][PYTHON] Optimize ArrowTableToRowsConversion.convert to improve its performance
    omit 91502eb89a29 [SPARK-52619][SQL][FOLLOWUP] Simplify casting of TIME to INT
    omit b9dcdde251d2 [SPARK-52804][BUILD] Add a check for the Java version used to build Spark
    omit 5724c714ed9f [SPARK-52814][BUILD] Upgrade zstd-jni  to 1.5.7-4
    omit b77749801ac3 [SPARK-52619][SQL] Cast TimeType to IntegralType
    omit 315c62510da8 [SPARK-52815][CORE] Improve `SparkClassUtils` to support `getAllInterfaces`
    omit fac4433f2399 [SPARK-52779][SQL][CONNECT] Support TimeType literal in Connect
    omit 5cd50eb6d3c7 [MINOR][PYTHON][TESTS] De-flake unit tests of `collect_set` and `collect_list` in `test_connect_function`
    omit cea0051f71e8 [SPARK-52809][SQL] Don't hold reader and iterator references for all partitions in task completion listeners for metric update
    omit 3fda679ec33d [SPARK-52801][SQL] Make name-precedence-in-order-by-and-having-with-conflicting-attributes suite deterministic
    omit 1112a870e0d7 [SPARK-52783][SQL] Refactor windowFunction validation logic from checkAnalysis for reuse in single-pass analyzer
    omit b4fc939223d7 [MINOR][DOCS] Fix example 4 in DataFrame.na.replace()
    omit ec9d8c0bbbca [SPARK-52808][CORE] `spark.history.retainedApplications` should be positive
    omit 13b3d65003ff [SPARK-52796][PYTHON] Optimize LocalDataToArrowConversion.convert to improve its performance
    omit eaf201788015 Revert "[SPARK-52576][SDP] Drop/recreate on full refresh and MV update"
    omit 454f7de59c1a [SPARK-52805][INFRA] Add `Java 17` build and install test pipeline
    omit d15ddd676a07 [SPARK-52788][SQL] Fix error of converting binary value in BinaryType to XML
    omit 4bb51328434c [SPARK-52789][BUILD][TESTS] Upgrade PostgreSQL to 42.7.7
    omit 09efee759b68 [SPARK-52686][FOLLOWUP][SQL][MINOR][TESTS] Fix a failing test
    omit a66c74fd1d6f [SPARK-52187][SQL] Introduce Join pushdown for DSv2
    omit 0d1375fe0e90 [SPARK-47404][SQL] Add configurable size limits for ANTLR DFA cache
    omit c84629ed173b [SPARK-52770][CONNECT] Support `TIME` type in connect proto
    omit 9565d164f389 [SPARK-52799][TESTS] Fix ThriftServerQueryTestSuite result comparison
    omit d90a2741c7e7 [SPARK-52577][SDP] Add tests for Declarative Pipelines DatasetManager with Hive catalog
    omit c51f5c3deb57 [SPARK-52784][SQL] Add TimeZoneAware casting in WindowFrameTypeCoercion.createBoundaryCast
    omit 7b0a5fe9c34b [MINOR][PYTHON][TESTS] Make a group of tests more deterministic in `test_connect_function.py`
    omit c4127d2ec8ba [SPARK-52759][SDP][SQL] Throw exception if pipeline has no tables or persisted views
    omit 417599d4f8eb [SPARK-52578][SQL][FOLLOW-UP] Small capitalization fixes for merge merge metrics
    omit 996181b49d14 [SPARK-52797] Use `Utils.isWindows` instead of re-evaluating in `PythonWorkerFactory`
    omit 83cc1e65c552 [SPARK-52792][CORE] Remove `commons-lang3` dependency from `network-common`
    omit c531e3376395 [SPARK-52793][CORE] Support `isUnix` at `o.a.s.util.Utils`
    omit a3cdd16c3a58 [SPARK-52588][SQL] Approx_top_k: accumulate and estimate
    omit cf097a5ab7fc [SPARK-52776][CORE] Do not split the comm field in ProcfsMetricsGetter
    omit 64f1c603befa [SPARK-52565][SQL] Enforce ordinal resolution before other sort order expressions
    omit 59303f7e54eb [SPARK-52673][CONNECT][CLIENT] Add grpc RetryInfo handling to Spark Connect retry policies
    omit f40fa761b573 [SPARK-52782][SQL] Return NULL from +/- on datetime with NULL
    omit 1f965cdf0afc [SPARK-52786][BUILD] Make pyspark-client package to upload with preview naming
    omit a76be7686d26 [SPARK-52781][PS] Fix dtyeps typo in internal.py
    omit 6c6ca1f0b6f4 [SPARK-52778][BUILD] Upgrade `commons-lang3` to 3.18.0
    omit 46f31844ebec [SPARK-52722][CORE] Deprecate JdbcRDD class
    omit d91a5da06920 [SPARK-52740][SS] Fix NPE in HDFSBackedStateStoreProvider accessing StateStoreConf.sqlConf when checkpoint format version is >=2
    omit f0356ab41e6f [SPARK-52711][PS] Fix float32 type widening in `mul`/`rmul` under ANSI
    omit a5a3b7ce2209 [SPARK-52724][SQL] Enhance broadcast join OOM error handling with SHUFFLE_MERGE hint support
    omit 9677b44e7de8 [SPARK-52727][SQL] Refactor Window resolution in order to reuse it in single-pass analyzer
    omit 823be3b75001 [SPARK-52763][SQL] Support TIME subtract
    omit d0880821bb7b [SPARK-52753][SQL] Make parseDataType binary compatible with previous versions
    omit 0b57f9c95490 [SPARK-52716][SDP][SQL] Remove comment from Flow trait and references
    omit 42787ccd56d9 [SPARK-52768][PS] Fix typo "moveing" in pandas/series.py
    omit 8c6170a9daf7 [SPARK-52765][K8S] Use `StandardCharsets.UTF_8` instead of `Charsets.UTF_8`
    omit a4955bb1882b [SPARK-52766][INFRA] Allow manual dispatches of dry runs in the forked repository
    omit 14d5c6759dc1 [MINOR][PYTHON][ML][TESTS] Further skip flaky `test_parity_regression`
    omit 57eccea6488e [SPARK-52725][CORE] Delay resource profile manager initialization until plugin is loaded
    omit d883ce58467a [SPARK-52745][SQL] Ensure one of the `schema` and `columns` in the Table interface is implemented and `columns` is preferable
    omit 360df7c6c073 [SPARK-52735][FOLLOWUP] Drop created UDF UDFs in SQL query tests
    omit f145b2f2789a [SPARK-52730][SQL] Store underlying driver and database version in JDBCRDD
    omit 11fff05032ec [SPARK-52750][TESTS] Add eventually in flaky test LocalTableScanExec
    omit 2571992aabb4 [SPARK-52760][PS] Fix float32 type widening in `sub`/`rsub` under ANSI
    omit 7208875483d0 [SPARK-52757][CONNECT] Rename "plan" field in DefineFlow to "relation"
    omit aeaca3005fc4 [SPARK-52720][PS] Fix float32 type widening in `add`/`radd` under ANSI
    omit bebb105adf8c [SPARK-52694][SQL] Add `o.a.s.sql.Encoders#udt`API
    omit 0aace88342e6 [SPARK-52731][SS][PYTHON][TESTS] Fix flaky test_listener_events in 'pyspark.sql.tests.streaming.test_streaming_listener'
    omit d88298a06aff [SPARK-52738][SQL] Support aggregating the TIME type with a UDAF when the underlying buffer is an `UnsafeRow`
    omit 35baa9755440 [SPARK-52749][BUILD] Replace preview1 to dev1 in its PyPI package name in the vote email
    omit 45865a6c3dc1 [SPARK-52723][SQL][CONNECT] Server side column name validation
    omit b0ea3e5a0c03 [SPARK-52561][PYTHON][DOC][FOLLOW-UP] Further upgrade the minimum python version
    omit 5cc8a7935769 [SPARK-52739][PYTHON][TEST] Apply a patch to unittest.main to report errors in setUpClass properly
    omit 6efa8c536933 [SPARK-52718][PS] Fix float32 type widening in `rmod`/`mod` under ANSI
    omit cdc25791f878 [SPARK-52735][SQL] Fix missing error conditions for SQL UDFs
    omit 71282402ef04 [SPARK-52736][PS] Fix typos 'moveing' in pyspark/pandas/generic.py
    omit 589e93a02725 [SPARK-52733][SQL][DOCS] Update the page about TIME literals
    omit 4faae1d88e8d [SPARK-52728][BUILD] Remove the usage of non-existent profiles
    omit e72128498cbe [SPARK-52691][BUILD] Upgrade ORC to 2.1.3
    omit 55bbe5aaac85 [SPARK-52409][SDP] Only use PipelineRunEventBuffer in tests
    omit 2460893c39f2 [SPARK-52704][SQL][FOLLOWUP] Move session state utilities from trait FileFormat to SessionStateHelper
    omit c6e864538e2d [SPARK-52721][PYTHON] Fix message parameter for CANNOT_PARSE_DATATYPE
    omit de5fa426e23b [MINOR][PYTHON][ML][TESTS] Skip two flaky tests
    omit 4321ade7ee02 [SPARK-52715][SQL] Pretty SQL string of TIME - DAY-TIME INTERVAL
    omit 308d3e899fd9 [SPARK-52686][SQL] `Union` should be resolved only if there are no duplicates
    omit 958c1abda64d [SPARK-52710][SQL] `DESCRIBE SCHEMA` should print collation
    omit aa0a5087e936 [SPARK-52717][INFRA] Escape special characters when redacting the log files in release build
    omit a46296e74408 [SPARK-52706][SQL] Fix inconsistencies and refactor primitive types in parser
    omit c506b2fb4190 [SPARK-52701][PS] Fix float32 type widening in `mod` with bool under ANSI
    omit 9e17cbafbce8 [SPARK-52714][SDP] Remove unused comment arg in append_flow decorator
    omit 18e995f3814d [SPARK-52563][PS] Fix var naming bug in _assert_pandas_almost_equal
    omit 6cfe303b06da [SPARK-51919][PYTHON] Allow overwriting statically registered Python Data Source
    omit 8915c6078527 [SPARK-52705][SQL] Refactor deterministic check for grouping expressions
    omit f8ab74e6a6c3 [SPARK-52703][INFRA][PS] Upgrade minimum python version of pandas api to 3.10
    omit 51bbae091885 [SPARK-52707][BUILD] Remove preview postfix when looking up the JIRA versions
    omit cdd7f226f751 [SPARK-52704][SQL] Simplify interoperations between SQLConf and file-format options in TextBasedFileFormats
    omit 25b0f7b1df64 [SPARK-52699][SQL] Support aggregating TIME type in interpreted mode
    omit 62becf4750e0 [SPARK-52693][SQL] Support +/- ANSI day-time intervals to/from TIME
    omit 8b437570fd1e [SPARK-52576][SDP] Drop/recreate on full refresh and MV update
    omit cda84dd096b9 [SPARK-52695][SQL] User Defined Type write support for xml file format
    omit 00cf5dac497a Revert "[SPARK-52698][PYTHON] Improve type hints for datasource module"
    omit a9b8e370893b [SPARK-52698][PYTHON] Improve type hints for datasource module
    omit f17a8d7f0d86 [SPARK-51557][SQL][TESTS] Add tests for TIME data type in Java API
    omit b0022ad15b68 [SPARK-52697][BUILD] Make Maven plugins up-to-date
    omit 46b6ccbd93c4 [SPARK-52692][SQL] Support caching the TIME type
    omit 4b862685b3d4 [SPARK-52082][PYTHON][DOCS] Improve ExtractPythonUDF docs
    omit 5a4932b3675d [SPARK-52683][SQL] Support ExternalCatalog alterTableSchema
    omit d89a4af9d761 [SPARK-52675][ML][CONNECT] Interrupt hanging ML handlers in tests
    omit 83514c972ea8 [SPARK-52688][SQL][DOCS] Fixing broken links to python reference in the getting started guide
    omit d1d29f9dd9ae [SPARK-52560][BUILD] Bump ap-loader 4.0(v10) to support for async-profiler 4.0
    omit 00a8adbf217c [SPARK-52663][SDP] Introduce name field to pipeline spec
    omit c2942b752fc2 [SPARK-52460][SQL][FOLLOWUP] Rename `timeToMicros` to `makeTime`
    omit 077b27ceb34b [SPARK-52671][SQL] RowEncoder shall not lookup a resolved UDT
    omit a053c68f4016 [SPARK-52561][PYTHON][INFRA] Upgrade the minimum version of Python to 3.10
    omit c2dd021492fa [SPARK-52632][SQL] Pretty display V2 write plan nodes
    omit 3fb4adbf3fa2 [SPARK-52685][SQL][TESTS] Add a clue for flaky test: 'SPARK-47148: AQE should avoid to submit shuffle job on cancellation'
    omit a1d55d7d1f56 [SPARK-49386][CORE][SQL] Add memory based thresholds for shuffle spill
    omit eb5af454e147 [SPARK-52672][SQL] Don't replace Sort/Having expressions with aliases if expression exists in Aggregate
    omit ff93cbb0c4b4 [SPARK-52634][SQL][DOCS] Update the ANSI compliance page regarding the TIME type
    omit 08e43d37f5fd [SPARK-52684][SQL] Make CACHE TABLE Commands atomic while encountering execution errors
    omit b44389c594ec [SPARK-52618][SQL] Casting TIME(n) to TIME(m)
    omit 80ef244e43f7 [SPARK-52682][CORE][TESTS] Migration test case `RocksDBBenchmark` to microbenchmark
    omit 5a8cd9749845 [SPARK-52680][INFRA] Remove Python 3.9 GitHub Action CI
    omit 9953c95e97d3 [SPARK-52674][SQL] Clean up the usage of deprecated APIs related to `RandomStringUtils`
    omit 3b15ee86bd29 [SPARK-52670][SQL] Make HiveResult work with UserDefinedType#stringifyValue
    omit 2b3eae0281a4 [MINOR][PYTHON] Add `scipy-stubs` to dev/requirements.txt
    omit 700d645e9539 [SPARK-52666][SQL] Map User Defined Type to correct MutableValue in SpecificInternalRow
    omit 4ec992c77396 [SPARK-52665][BUILD] Fix make-distribution.sh [: missing `]'
    omit d2cc107560e7 [MINOR][CONNECT][PYTHON] Fix a race condition in ExecutePlanResponseReattachableIterator.shutdown
    omit 1ffbfbe48737 [MINOR][TEST][CONNECT][PYTHON] Fix flaky test_client.py test
    omit b1a9158881fa [SPARK-51516][SQL] Support TIME by the Thrift server
    omit 031310bcc339 [SPARK-52660][SQL] Add time type to `CodeGenerator#javaClass`
    omit 2bc9dd3525dc [SPARK-52656][SQL] Fix current_time()
    omit ace03f419868 [SPARK-52535][SQL] Improve code readability of rule ApplyColumnarRulesAndInsertTransitions
    omit e8384d5af58e [SPARK-52613][CORE][SQL] Restore printing full stacktrace when HBase/Hive DelegationTokenProvider hit exception
    omit 17bd772291de [SPARK-52649][SQL] Trim aliases before matching Sort/Having/Filter expressions in `buildAggExprList`
    omit 03eb318f7119 [SPARK-52615][CORE] Replace File.mkdirs with Utils.createDirectory
    omit 2bd93dafb9fd [SPARK-52630][SS] Reorganize streaming operator and state mgmt code and dirs
    omit 0c6d7fd277ba [SPARK-52651][SQL] Handle User Defined Type in Nested ColumnVector
    omit 5990db0a3c1a [SPARK-52652][BUILD] Replace `os-maven-plugin` with `nisse`
    omit a1e628574b7d [SPARK-52633][SQL] Deduplicate single `Union` child output before `DeduplicateRelations`
    omit 372c3d15c02b [SPARK-51268][SQL] Make `TreeNode` lock-free
    omit ddc4005a93fc [SPARK-52503][SQL][CONNECT] Fix `drop` when the input column is not existent
    omit c5ed408ee3c2 [SPARK-52545][SQL] Standardize double-quote escaping to follow SQL specification
    omit 591e1c38dedd [SPARK-52612][INFRA] Add an env `NO_PROVIDED_SPARK_JARS` to control collection behavior of `sbt/package` for `spark-avro.jar` and `spark-protobuf.jar`
    omit 47e0888aca36 [SPARK-52642][PYTHON] Use daemonWorkers.get(worker) to avoid unexpected out-of-sync between idleWorkers and daemonWorkers
    omit 86a81a748f33 [SPARK-52643][BUILD] Upgrade `extra-enforcer-rules` to support Java 25
    omit e762eee92cf4 [SPARK-52639][K8S][INFRA][DOCS] Upgrade Volcano to 1.12.1
    omit 9a9e321831fb [SPARK-52551][SQL][FOLLOW-UP] Enable ANSI mode in PushablePredicateSuite
    omit 47782d046225 [SPARK-52539][CONNECT][FOLLOW-UP] Avoid using TypeAlias that is available from Python 3.10
    omit 771a9d77f659 [SPARK-52570][PS] Enable divide-by-zero for numeric rmod with ANSI enabled
    omit 9b271a92e1cb [SPARK-52636][K8S] Support `java_image_name` ARG in K8s `Dockerfile`
    omit 5e6e8f12a503 [SPARK-52023][SQL] Fix data corruption/segfault returning Option[Product] from udaf
    omit 646d96ed2a58 [SPARK-52515][SQL][TESTS][FOLLOW-UP] Explicitly enable ANSI for overflow error case
    omit 8b6eecaa8a1c [SPARK-52626][SQL] Allow grouping by the time type
    omit fdd7f6fb491e Revert "[SPARK-52570][PS] Enable divide-by-zero for numeric rmod with ANSI enabled"
    omit c16535549603 [SPARK-52541][SDP] Add programming guide for Declarative Pipelines
    omit bbff88291c4e [SPARK-52578][SQL] Add metrics for rows to track case and action in MergeRowsExec
    omit 12da3c5aa648 [SPARK-52596][SQL] Try make TIMESTAMP_NTZ from DATE and TIME
    omit 26a9ba6481c4 [SPARK-52609][SQL] Make sure row sorter from each task won't be overwritten by other tasks in parallel
    omit 2d4ff6eeeffd [SPARK-52610][BUILD] Upgrade rocksdbjni to 10.2.1
    omit 0afe9f5d13e2 [SPARK-52148][SQL] Fix CREATE OR REPLACE function for SQL user-defined TVFs
    omit 82860bfb30c4 [SPARK-52497][DOCS] Add docs for SQL user-defined functions
    omit 6f54429e5761 [SPARK-52551][SQL] Add a new v2 Predicate BOOLEAN_EXPRESSION
    omit 6ca94aac54d1 [SPARK-52600][CONNECT] Move CompletionIterator to common/utils
    omit 9fcacba60c5a [SPARK-52611][SQL] Fix SQLConf version for excludeSubqueryRefsFromRemoveRedundantAliases…
    omit 0c8d18d41ed2 [SPARK-52599][PYTHON] Support periodical traceback dump in Driver side workers
    omit 059b395c8cbf [SPARK-52595][SQL] Rename the `TimeAdd` expression to `TimestampAddInterval`
    omit 7d8d018e7bdc [SPARK-52381][CORE] JsonProtocol: Only accept subclasses of SparkListenerEvent
    omit dd497fbd3d2c [SPARK-52597][SS][TESTS] Fix the execution failure of `StateStoreBasicOperationsBenchmark`
    omit 16c0ce961844 [SPARK-52594][DOCS] Workaround pandas version string issue in doc generation
    omit 7f2289909383 [SPARK-52570][PS] Enable divide-by-zero for numeric rmod with ANSI enabled
    omit 69c3eb9ae65a [SPARK-52188][FOLLOWUP] Update comment in getRunId
    omit 99acf6410ed4 [SPARK-52552][SQL] Skip CHECK constraint enforcement for deletion vector deletes
    omit a1908738223b [SPARK-52591][SDP] Validate streaming-ness of DFs returned by SDP table and standalone flow definitions
    omit af5632f49260 [SPARK-52590][SQL][TESTS] Add SQL query tests for SQL functions without explicit return types
    omit 65e62283fd0c [SPARK-52494][SQL] Support colon-sign operator syntax to access Variant fields
    omit d411b28ed86e [SPARK-52547][INFRA][FOLLOW-UP] Set default GIT_BRANCH even for scheduled jobs
    omit 813467943b6e [SPARK-52586][SQL] Introduce AnyTimeType
    omit c229245c0177 [SPARK-52583][SQL] Add an Developer API for stringifying values in UserDefinedType
    omit 3f6cefb1ee00 [SPARK-52518][SQL][TEST] Fix `AddMetadataColumnsSuite` package path
    omit 4efb630db3b8 [SPARK-52587][SHELL] spark-shell 2.13 support `-i` `-I` parameter
    omit 063144d9a084 [SPARK-52515][SQL] Add approx_top_k function
    omit efb03b18087b [SPARK-52403][SQL] Add metric to MergeRowExec for rows that do not match condition
    omit 5432402f7dc6 [SPARK-52584][BUILD] Make build script to support preview releases in finalize step
    omit b63221cd3d9b [SPARK-52354][SQL] Add type coercion to UnionLoop
    omit 1a8859d24763 [SPARK-52579][PYTHON] Set periodical traceback dump for Python workers
    omit ebf5a2aef403 [SPARK-52572][PS] Avoid CAST_INVALID_INPUT of DataFrame.isin in ANSI mode
    omit 936d48147728 [SPARK-52567][SQL][HIVE][TEST] Refactor Hive_2_1_DDLSuite
    omit 36a08bd37354 [SPARK-52568][BUILD] Fix `exec-maven-plugin` version used by `dev/test-dependencies.sh`
    omit c0c017af8b49 [SPARK-52573][PYTHON][TESTS] Add tests for decimal type
    omit d24c7180b39e [SPARK-52574][SQL][TESTS] Ensure compression codec is correctly applied in Hive tables and dirs
    omit e1d1302a3632 [SPARK-52533] Support enabling only driver profiler
    omit 503681eed418 [SPARK-52569][SQL] Fix the class cast exception in `SecondsOfTimeWithFraction`
    omit bde09039fb2f [PYTHON][MINOR] Decrease default `arrowMaxBytesPerBatch` for arrow-optimized UDF
    omit 1cfe07cdbeea [SPARK-52339][SQL][FOLLOWUP] Sort paths in InMemoryFileIndex#equal only when size matches
    omit 82ab68045759 [SPARK-52562][INFRA] Automatically create the base of release notes and push
    omit 307f91ae0303 [SPARK-52547][INFRA][FOLLOW-UP] Check RELEASE_VERSION and SPARK_RC_COUNT for dryruns
    omit 4d7d2538bc7b [SPARK-52557][PS] Avoid CAST_INVALID_INPUT of to_numeric(errors='coerce') in ANSI mode
    omit 74c94c3587ce Revert "[SPARK-50849][CONNECT] Add example project to demonstrate Spark Connect Server Libraries"
    omit 9116d33c5bc1 [SPARK-52548][CORE][TESTS] Add a test case for when shuffle manager is overridden by a SparkPlugin
    omit 108bbaedb9a3 [SPARK-52470][ML][PYTHON][FOLLOW-UP] Further fix GRPC import
    omit ad9d9e7a0935 [SPARK-52460][SQL][FOLLOWUP] Fix names and test ranges of TIME values
    omit 79167994d04a [SPARK-48231][BUILD] Remove unused CodeHaus Jackson dependencies
    omit 570dc7aa34e9 [SPARK-52395][CORE] Fast fail when shuffle fetch failure happens
    omit be08d581b202 [SPARK-52553][SS] Fix NumberFormatException when reading v1 changelog
    omit cd77379f6027 [SPARK-52546][SQL] check sparkContext if has stopped when running catch code block in execute(), otherwise, it will return wrong state
    omit 618b3da9f642 [SPARK-52534][ML][CONNECT] Make MLCache and MLHandler thread-safe
    omit 81ca4fd5479c [SPARK-52499][SQL] Add more data type tests for SQL UDFs
    omit 04fd2203382c [SPARK-52536] Set AsyncProfilerLoader extractionDir to spark local dir
    omit 1f449817af93 [SPARK-52554][PS] Avoid multiple roundtrips for config check in Spark Connect
    omit 89e8e0782461 [SPARK-52549][SQL] Disable Recursive CTE self-references from window functions and inside sorts
    omit 87dddc6eea68 [SPARK-52547][INFRA] Build dry runs against master branch
    omit b9e121a144f5 [SPARK-52349][PS] Enable boolean division tests with ANSI enabled
    omit b5948486cbd5 [SPARK-52462][SQL] Enforce type coercion before children output deduplication in Union
    omit 5e1e4d31d7d9 [SPARK-52540][SQL] Make TIMESTAMP_NTZ from DATE and TIME
    omit ccded6c21b28 [SPARK-52339][SQL] Fix comparison of `InMemoryFileIndex` instances
    omit 23617db84e5f [SPARK-52531][SQL] `OuterReference` in subquery aggregate is incorrectly tied to outer query aggregate
    omit f5a45826bbff [SPARK-52532][SQL][TESTS] Add tests for hidden or main output prioritization during attribute name conflict
    omit 232f86c684d0 [SPARK-52528][PS] Enable divide-by-zero for numeric mod with ANSI enabled
    omit 1ab9ba6efd0f [SPARK-52482][SQL][CORE] ZStandard support for file data source reader
    omit bfe231c9a5a8 [SPARK-52539][CONNECT] Introduce session hooks
    omit 4914b4b689b1 [SPARK-52424][TESTS] Add unit test to verify in Spark Connect, Column object defined outside of a function can be referenced properly
    omit 6ab297b0fc7d [SPARK-52519][PS] Enable divide-by-zero for numeric floordiv with ANSI enabled
    omit 457542966651 [SPARK-52537][CORE] Print stacktrace on creating temp dir failure
    omit f502d66fa295 [SPARK-52450][CONNECT] Improve performance of schema deepcopy
    omit 509f93617c01 [SPARK-52530][SQL][TESTS] Move `SimpleTestOptimizer` to test source directory
    omit 8222670dba17 [SPARK-52470][TESTS][FOLLOWUP] Fix test failure caused by import connect in pure classic testing envs
    omit 44d9fce85524 [SPARK-52492][SQL] Make InMemoryRelation.convertToColumnarIfPossible customizable
    omit e8d6f5437e9e [SPARK-52426][CORE] Support redirecting stdout/stderr to logging system
    omit be04876578d3 [SPARK-52529][DOCS][FOLLOWUP] Fully upgrade jekyll from 4.3 to 4.4
    omit 23cf9266f323 [SPARK-52489][SQL] Forbid duplicate SQLEXCEPTION and NOT FOUND handlers inside SQL Script
    omit 105570b34ea8 [SPARK-52525][PYTHON][INFRA] Refresh testing images for pyarrow 20
    omit a38d1cef73ed [SPARK-52527][BUILD] Upgrade `junit` to 5.13.1
    omit 18faa83903b1 [SPARK-52521][SQL] `Right#replacement` should not access SQLConf dynamically
    omit 73ca5e83d20f [SPARK-52510] Reserve proto numbers for Geospatial types
    omit e17ff648f836 [SPARK-52417][SQL] Simplify Table properties handling in View Schema Evolution Mode
    omit 722d02ccbda7 [SPARK-52484][SQL] Skip child.supportsColumnar assertion from driver side in ColumnarToRowExec
    omit fa7b6b5e2668 [SPARK-52514][INFRA] Use `windows-2025` image in R GitHub Action job
    omit 3185b9e8d116 [SPARK-52488][SQL] Strip alias before wrapping outer references under HAVING
    omit 3b75442b5cab [SPARK-52402][PS] Fix divide-by-zero errors in Kendall and Pearson correlation under ANSI mode
    omit 26956365935d [SPARK-52501][BUILD] Upgrade scala-maven-plugin to 4.9.5
    omit 850640427c9d [SPARK-52502][UI] Thread count overview
    omit fd74b5ec6bd6 [SPARK-52470][ML][CONNECT] Support model summary offloading
    omit 7d0b921c465f [SPARK-42841][SQL] Assign a name to the error class _LEGACY_ERROR_TEMP_2005
    omit 37c85a608929 [SPARK-52483][INFRA][FOLLOW-UP] Fix workflow `pages.yml`
    omit 435cb4bf50a4 [SPARK-52493][SQL] Support TIMESTAMP WITHOUT TIME ZONE
    omit c4fb4a472fb1 [SPARK-42322][SQL] Assign name to_LEGACY_ERROR_TEMP_2235
    omit abecd4affbd9 [SPARK-52491][INFRA] Add `Java 25-ea` build and install test pipeline
    omit 420ac24b3466 [SPARK-33538][SQL] Directly push IN/NOT predicates to the Hive Metastore
    omit 243af2fe684e [SPARK-52483][INFRA] Upgrade to Python 3.11 in doc image
    omit 057d9fa4f986 [SPARK-52469][CORE] Use JEP 223 API to process Java version
    omit 59e6b5b7d350 [SPARK-52475][TESTS] Remove deprecated table.schema method from tests
    omit a108d29e5a66 [SPARK-52487][UI] Add Stage Submitted Time and Duration to StagePage Detail
    omit 441922044a2b [SPARK-52485][SQL] Declarative Pipelines code cleanup
    omit 5d0b2f41794b [SPARK-52065][SQL] Produce another plan tree with output columns (name, data type, nullability) in plan change logging
    omit ac1835484320 [SPARK-52433][PYTHON] Unify the string coercion in createDataFrame
    omit a7f3b5f2db08 [SPARK-52460][SQL] Store internal TIME values as nanoseconds
    omit 2e379be018d7 [SPARK-52455][SQL][TESTS] Test expression-based defaults in UPDATE and MERGE
    omit bbc80430dfce [SPARK-51563][SQL] Support the fully qualified type name TIME(n) WITHOUT TIME ZONE
    omit f025a46c9ee7 [SPARK-52476][BUILD] Upgrade dropwizard metrics to 4.2.32
    omit 7b499191d28d [SPARK-52427][SQL] Normalize aggregate expression list covered by a Project
    omit c4ffb7b31169 [SPARK-52431][SDP] Finishing touches on Declarative Pipelines runner
    omit 227515c7e2ff [SQL][MINOR] Inline recursive CTEs
    omit 2768c2e84ab4 [MINOR][INFRA] Update `.asf.yaml` with new `README.md` link
    omit 597d49f89cab [SPARK-52447][SQL] Move GetViewColumnByNameAndOrdinal normalization to SessionCatalogSuite
    omit bf9237396a89 [SPARK-52454][BUILD] Automatically remove old releases from the mirror
    omit 20cd4ad8167e [SPARK-52453][BUILD] Automatically release and drop artifacts in Apache Nexus repository
    omit 2bd193a69cd0 [MINOR][DOCS] Fix broken link for Python Package Management
    omit 1d4d06b3dd31 [SPARK-52458][CORE] Support `spark.eventLog.excludedPatterns`
    omit 269584b55158 [SPARK-52422][FOLLOWUP] Fix Github Pages workflow w/ pandas=2.2.3
    omit 68136fd50a5b [SPARK-52456][CORE] Lower the minimum limit of `spark.eventLog.rolling.maxFileSize`
    omit 348636605ff2 [SPARK-52430][SQL] Resolve bug with nullability of Union in rCTEs
    omit 8052660f49a6 [SPARK-52440][DECLARATIVE PIPELINES] Add python install extra for declarative pipelines
    omit a258f922515d [SPARK-52441][INFRA] Skip testing `sql/pipelines` module in the Maven daily test when the input branch is branch-4.0
    omit 46de57b55113 [SPARK-52423][CONNECT] Add missing golang package name to pipelines.proto
    omit c999d2e3185b [SPARK-52421][INFRA][FOLLOW-UP] Fix the link for KEYS
    omit cadf7385f826 [SPARK-52438][BUILD] Upgrade Netty to 4.1.122.Final and Netty-tcnative to 2.0.72.Final
    omit ac12869a09e2 [SPARK-52437][BUILD] Ensure `Annotation Processing` in Java compilation for Java 23+
    omit efd81c5a46ce [SPARK-52422][PYTHON] Update `pandas` to 2.3.0
    omit b44e243dbd9c [SQL][MINOR] Update comments and remove dead code in UnionLoop
    omit 45546005cfb1 [SPARK-52434][BUILD] Upgrade `gcs-connector` to `2.2.28`
    omit 0a894cc50e63 [SQL][MINOR] Fix bugs with self references inside subqueries in rCTEs
    omit 651181a817d2 [SPARK-52425][SQL] Refactor ExtractValue so it can be reused in single-pass implementation
    omit 1dfcd8f010ad [SPARK-52404][SQL] Give a separate error class to failures in `BindReferences`
    omit 8c3194fbb8dc [SPARK-52348][CONNECT] Add support for Spark Connect handlers for pipeline commands
    omit 1a3ae66c6c48 [MINOR][PYTHON][TESTS] Skip pipeline tests if yaml not installed
    omit 9f72b7a71633 [SPARK-52412][INFRA][FOLLOW-UP] Escape special characters when redacting the logs
    omit fc786bbb318e [SPARK-52421][BUILD] Automatically send the RC vote email
    omit 13bf9703f257 [SPARK-52410] Deprecate `PipelineConf` and use `SqlConf` directly
    omit c75e7c78b2d1 [SPARK-52420][PYTHON][TESTS] Make test_udtf_with_invalid_return_type compatible with Python only client
    omit b969ffe5d816 [SPARK-50752][SQL][PYSPARK][FOLLOWUP] Respect PYTHON_UDF_MAX_RECORDS_PER_BATCH in Python worker
    omit 0a05e1cd6346 [SPARK-52412][INFRA][FOLLOW-UP] Skip empty patterns when redacting
    omit 73de7ccc8314 [SPARK-52419][PYTHON][TESTS] Make Python Connect tests work with NumPy 2.3.0
    omit d31edcb5aac9 [SPARK-52346][SQL][FOLLOW-UP] Fix MaterializeTablesSuite in Maven
    omit 7bf0deea93e6 [SPARK-52238][TESTS][FOLLOW-UP] Fix test failure in classic-only envs
    omit 300e4672d598 [SPARK-52418][CORE] Add a `noElements ` state variable to the `PercentileHeap` to avoid repetitive calculations of `isEmpty()`
    omit 1b7df31fa9e8 [SPARK-52406][CORE][TESTS] Move the test case named "benchmark" from `PercentileHeapSuite` to `PercentileHeapBenchmark`
    omit 920e46f02785 [SPARK-52413][PYTHON][TEST] `test_install_spark` switch to Spark 3.5.6
    omit ad32d690d609 [SPARK-52400][SQL][TESTS][FOLLOWUP] Regenerate `nonansi/try_arithmetic.sql.out` for Java 21
    omit 3f9e0d213510 [SPARK-52415][BUILD] Upgrade Maven to 3.9.10
    omit d7043c0c7c7e [SPARK-52393][SDP] Pipeline SQL Graph Element Registration
    omit 83639f598026 [SPARK-52356][PS] Enable divide-by-zero for boolean mod/rmod with ANSI enabled
    omit 5633e9312137 [SPARK-52288][PS] Avoid INVALID_ARRAY_INDEX in `split`/`rsplit` when ANSI mode is on
    omit a380380428db [SPARK-52265][SQL][TEST] Fix regex leading to empty PROCESS_TABLES.testingVersions in HiveExternalCatalogVersionsSuite
    omit ec120fddcf64 [SPARK-51987][SPARK-52235][SQL][FOLLOW-UP] Add ANSI mode check for new tests
    omit 6a6a0818d8b3 [SPARK-52405] Extend V2JDBCTest to test multiple partition reading
    omit a9bfacb084e6 [SPARK-52391][SS] Refactor TransformWithStateExec to extract shared functions and variables into an abstract base class for Scala and Python
    omit e4e35caf7b06 [MINOR] Ensure structured_logging_style runs out of correct directory
    omit ced68a079b68 [SPARK-52412][INFRA] Redact sensitive information in log files at release workflow
    omit b196c24d8211 [SPARK-52346][SQL] Declarative Pipeline `DataflowGraph` execution and event logging
    omit ced7b0322233 [SPARK-52386][CORE][TESTS][FOLLOWUP] Add field sorting to JSON strings to ensure the consistency of the content output to golden files
    omit 6f5fb36165fb [SPARK-50815][PYTHON] Fix Variant Local Data to Arrow Conversion
    omit 92bfd02336d9 [SPARK-52400][SQL][TESTS] Cover SQL test gaps in golden files
    omit 79893b4c8515 [SPARK-52396][SQL] Artifact Root Directory should use tmpdir
    omit 23e6274f54da [SPARK-48356][FOLLOW UP][SQL] Improve FOR statement's column schema inference
    omit 273a5e79ec73 [SPARK-52397][CONNECT] Idempotent ExecutePlan: the second ExecutePlan with same operationId and plan reattaches
    omit 970ba3dbff28 Revert "[MINOR] Ensure structured_logging_style runs out of correct directory"
    omit 4c69d0df96cb [SPARK-52283][SQL] Declarative Pipelines `DataflowGraph` creation and resolution
    omit 28f05886a078 [MINOR] Ensure structured_logging_style runs out of correct directory
    omit 0804b8af2965 [SPARK-52386][CORE][TESTS] Refactor the `HistoryServerSuite` to support regenerating the `expectation.json` files using `SPARK_GENERATE_GOLDEN_FILES=1`
    omit ff79c2ad7f2e [SPARK-52383][CONNECT] Improve errors in SparkConnectPlanner
    omit ae3f720eb712 [SPARK-52385][SQL] Remove TempResolvedColumns from InheritAnalysisRules name
    omit 5ef6d4ba7051 [SPARK-52392][SQL] New single-pass Analyzer functionality
    omit 4c9bfcb5d9b7 [SPARK-52353][SQL] Fix bug with wrong constraints in LogicalRDDs referencing previous iterations in UnionLoop
    omit 1b41079f3f1e Revert "[SPARK-52283][SQL] Declarative Pipelines `DataflowGraph` creation and resolution"
    omit 47457cb218c3 [SPARK-52352][PYTHON][DOCS] Update `pyspark.sql.functions.to_json` docstring to include `VariantType` as valid input
    omit d52b8e24abea [SPARK-52355][PYTHON] Infer `VariantVal` object type as `VariantType` when creating a `DataFrame`
    omit e9a822bef051 [SPARK-52384][CONNECT] Fix bug Connect should insensitive for JDBC options
    omit 58387b47ddb2 [SPARK-52337][CONNECT] Make InvalidPlanInput a user-facing error
    omit 26370d6bcc13 [SPARK-52283][SQL] Declarative Pipelines `DataflowGraph` creation and resolution
    omit a584cc48ef63 [SPARK-52389][K8S][DOCS] Drop K8s v1.30 Support
    omit 80657974ee69 [SPARK-52238][TESTS][FOLLOW-UP] Fix test failure in daily builds
    omit b18b956f9670 [SPARK-52212][PYTHON][INFRA] Upgrade linter image to python 3.11
    omit 64982a45f6bb [SPARK-52307][PYTHON][CONNECT] Support Scalar Arrow Iterator UDF
    omit cfd695b8e768 [SPARK-52281][SQL] Change `ALTER TABLE ALTER COLUMN TYPE STRING` not to apply default collation if original data type was instance of `StringType`
    omit 3e7fd7e45759 [SPARK-52372][SQL] Fix error handling in `ApplyDefaultCollationToStringType`
    omit 4a395bc2f229 [SPARK-51987][SQL] DSv2 expressions in column defaults on write
    omit 8513b91c53b2 [SPARK-52363][SQL][TESTS] Analyzer NameScope test for nested struct fields inside map values
    omit b0c2ba357bf0 [SPARK-52249][PS] Enable divide-by-zero for numeric truediv with ANSI enabled
    omit 73f1dd2d535e [SPARK-52218][SQL] Make current datetime functions evaluable again
    omit e3321aa44ea2 [SPARK-52238][PYTHON] Python client for Declarative Pipelines
    omit 6a89f654bb54 [SPARK-52130][FOLLOW-UP][ML][CONNECT] Refine error message when model.summary is evicted
    omit a4bdb37efcf7 [SPARK-52350][SS] Fix link for SS programming guide in 4.0
    omit 6cfd80006c94 [SPARK-52233][SQL][FOLLOW-UP] Fix 0.0 and -0.0 issues in map_zip_with
    omit 5201a66f55cf [SPARK-52311][SQL] Redefine UnionLoop output to not be duplicated if the anchor output is duplicated
    omit 64222567d266 [SPARK-52312][SQL] Ignore V2WriteCommand when caching DataFrame
    omit 6bbe032657eb [SPARK-52235][SQL] Add implicit cast to DefaultValue V2 Expressions passed to DSV2
    omit d61c1ea7f700 [SPARK-52335][CONNET][SQL] Unify the 'invalid bucket count' error for both Connect and Classic
    omit 2367f5826119 [SPARK-52345][SQL] Fix NULL behavior in scripting conditions
    omit 80cd86758a06 [SPARK-48353][FOLLOW UP][SQL] Exception handling improvements
    omit abc72f2da462 [SPARK-52232][SQL] Fix non-deterministic queries to produce different results at every step
    omit 3159416c1ed1 [SPARK-52333][SS][PYTHON] Squeeze the protocol of retrieving timers for transformWithState in PySpark
    omit 98552a6b6424 [SPARK-52338][SQL] Support for inheriting default collation from schema to View
    omit 3d84039292b7 [SPARK-52228][SS][PYSPARK] Construct the benchmark purposed TWS state server with in-memory state impls and the benchmark code in python
    omit 7e48ba7bf10e [SPARK-52272][SQL] V2SessionCatalog does not alter schema on Hive Catalog
    omit f634311c7e77 [INFRA][MINOR] Update mima exclude
    omit f2079d8a05de [SPARK-52243][CONNECT] Add NERF support for schema-related InvalidPlanInput errors
    omit f423885f65d4 [SPARK-52060][SQL] Make `OneRowRelationExec` node
    omit d152ae89bb2d Small improvements to Python Spark Connect installation docs
    omit 62eac3ec8f93 [SPARK-52329][SS] Remove private sql scoping tags for new transformWithState API
    omit 7f1306a309e7 [SPARK-52313][SQL] Correctly resolve reference data type for Views with default collation
    omit f349192dec3c [SPARK-52331][PS][TESTS] Adjust test for promotion from float32 to float64 during division
    omit 940e9058fe2f [SPARK-52264][PS][TESTS] Test divide-by-zero behavior with more numeric data types
    omit cc7bf9827dab [SPARK-52169][PYTHON][DOCS] User guide for native plotting
    omit da68d10206de [SPARK-52223][CONNECT] Add SDP Spark Connect Protos
    omit 190d3916c63c [SPARK-52300][SQL] Make SQL UDTVF resolution use consistent configurations with view resolution
    omit b4b986265c1d [SPARK-52254][INFRA] Adds a GitHub Actions workflow to convert RC to the official release
    omit 2cb8057ea425 [SPARK-52305][PYTHON] Refine the docstring for isnotnull, equal_null, nullif, nullifzero, nvl, nvl2, zeroifnull
    omit cf6c26ada2eb [SPARK-52267][SQL] Match field ID in ParquetToSparkSchemaConverter
    omit dd5dea9614f4 [SPARK-52324][INFRA] Move Spark docs to the release directory
    omit c9e356ab0481 [SPARK-52316][BUILD] Upgrade Kafka to 3.9.1
    omit 56dcdd83ad07 [SPARK-52314][K8S] Upgrade `kubernetes-client` to version 7.3.1
    omit 82e3f0d5d594 [SPARK-52309][SQL][TESTS] Add `log4j2.properties` to `pipelines` module for testing
    omit 0af54a13ba7b [SPARK-52310][SQL][BUILD] Fix code change detection for `pipelines` module
    omit 7775dd66748b [SPARK-52233][SQL] Fix map_zip_with for Floating Point Types
    omit 87b5d29ad0f3 [SPARK-52303][CORE] Promote `ExternalCommandRunner` to `Stable`
    omit 70cfe7d42f60 [SPARK-52294][INFRA] Add `sql/pipelines` to maven daily test module list
    omit 3db4232d1552 [MINOR] Update known_translations
    omit 8149d9bf6c45 [SPARK-47618][CORE] Use Magic Committer for all S3 buckets by default
    omit 12e32d65a4c2 [SPARK-52240] Corrected row index usage when exploding packed arrays in vectorized reader
    omit 486d99ae6a95 [SPARK-52299][CORE] replace deprecated JsonNode.fields
    omit 73a3b1c532cc [SPARK-52290][SQL][BUILD] Fix the Maven compilation and testing issues for `sql/pipelines` module
    omit cf3ef31bdfb0 [SPARK-52287][CORE] Improve `SparkContext` not to populate `o.a.s.internal.io.cloud.*`-related setting if not exist
    omit 5b382ffdc09a [SPARK-52280][TEST] Upgrade `MiMa` to use 4.0.0 as `previousSparkVersion`
    omit 80da53ed1dfd [SPARK-52143][TESTS] Add test for table constraints with `current_*` functions
    omit 8043304b9d42 [SPARK-51430][PYTHON] Stop PySpark context logger from propagating logs to stdout
    omit 1434d6841c48 [SPARK-52263][CORE][TESTS] Move the benchmark code related to `Sorter` from `SorterSuite` to `SorterBenchmark`
    omit 858fbccb7bab [SPARK-52219][SQL] Schema level collation support for tables
    omit 37c1f39e5e15 [SPARK-52262][SQL] swap order of withConnection and classifyException in loadTable
    omit 692f1b6265be [SPARK-52278][PYTHON] Scalar Arrow UDF support named arguments
    omit b8e6dc24bad7 [SPARK-52276][BUILD] Make ANSWER not to be overwritten in release scripts
    omit 83e57b00733d [SPARK-52225][BUILD][FOLLOW-UP] Change -it to -ti in Docker execution in release script
    omit 8f699a415410 [SPARK-52259][ML][CONNECT] Fix Param class binary compatibility
    omit 7fee2912ba8b [SPARK-52224][CONNECT][PYTHON] Introduce pyyaml as a dependency for the Python client
    omit a9881a8f32ff [SPARK-52260][SQL][TESTS] Add test for Update/Merge Into/Delete From table
    omit 4325d3a02819 [SPARK-52153][SQL] Fix from_json and to_json with variant
    omit 451a7a625742 [SPARK-52236][SQL] Standardize analyze exceptions for default value
    omit de3d44d46fdc [SPARK-50137][SQL] Avoid fallback to Hive-incompatible ways when table creation fails by thrift exception
    omit e2f4f5b9a48a [SPARK-52179][INFRA][FOLLOW-UP] Skip dryruns in forked repository
    omit 1c2b1e0fc8d4 [SPARK-52258][BUILD][FOLLOW-UP] Add retries when connecting to the ASF repo
    omit 7ee8f63c7e14 [MINOR][INFRA] Limit build_python_3.11_macos execution time to up to 2 hours
    omit ac184982e3fe [SPARK-52257][PYTHON][TESTS] Add tests for chained arrow UDF
    omit 086fdfdff965 [SPARK-52258][BUILD] Increase the number of retries and decrease the parallelism in the release script
    omit 2e526b543655 [SPARK-52179][INFRA][FOLLOW-UP] Do not quote wildcard in logs
    omit 2004f333173b [SPARK-52255][BUILD] Adds a new environment variable SPARK_RC_COUNT in release script
    omit ff0d9e466b28 [SPARK-52252][SQL] ScalaUDF encoders in subquery should be resolved
    omit 6ee2101c55f4 [SPARK-52179][INFRA] GitHub workflow for releasing the official Apache Spark
    omit ebb476ab3164 [SPARK-52239][PYTHON][CONNECT] Support register an arrow UDF
    omit 9df43ac6af88 [SPARK-52216][SQL][CONNECT] Make InvalidCommandInput a user-facing error
    omit b3c7c9e84ee8 [SPARK-52241][BUILD][FOLLOW-UP] Make it fail if one of the curls fails
    omit 11090d554b11 [SPARK-52166][SDP] Add support for PipelineEvents
    omit 701e356c534f Revert "[SPARK-50137][HIVE] Avoid fallback to Hive-incompatible ways when table creation fails by thrift exception"
    omit bc27f691000b [SPARK-50137][HIVE] Avoid fallback to Hive-incompatible ways when table creation fails by thrift exception
    omit 085645f8bba4 [SPARK-52079][SQL][FOLLOWUP] Normalize internal projections under Aggregate as well
    omit b84d3e6cca95 [SPARK-51927][SPARK-52213][BUILD] Upgrade jackson to 2.19.0 and upgrade kubernetes-client to version 7.3.0
    omit 67860390bb7b [SPARK-52237][DOCS] Fix the documentation of hypot function
    omit d0b0b10d2eed [SPARK-52241][BUILD] Upload to repository in parallel in release script
    omit d4bb024d1f77 [SPARK-52215][PYTHON][CONNECT] Implement Scalar Arrow UDF
    omit dad1369df70d [SPARK-52195][PYTHON][SS] Fix initial state column dropping issue for Python TWS
    omit 315be9f475a3 [SPARK-52110][SDP][SQL] Implement SQL syntax support for pipelines
    omit 4b3a65307681 [SPARK-51820][SQL][FOLLOWUP][CONNECT] Don't add `UnresolvedOrdinal` when appending grouping to aggregate expressions
    omit 9ce5914ab028 [SPARK-52095][SQL] Alter table alter column to pass V2Expression to DSV2
    omit 243d7e878bd3 [SPARK-52234][SQL] Fix error on non-string input to schema_of_csv/xml
    omit 85be7ee78281 [SPARK-52227][PROTOBUF][BUILD] Make packaging results of the `spark-protobuf` module generated by maven and sbt remain consistent
    omit b825e7437ca5 [SPARK-52229][ML][CONNECT] Improve model size estimation
    omit b542e27ae4cf [SPARK-52231][BUILD] Propagate GITHUB_ACTIONS environment variable in release script
    omit e7627b100a53 [SPARK-52184][SQL] Wrap external engine JDBC syntax errors with a unified exception
    omit 45fe27f78522 [SPARK-52181][SQL][FOLLOWUP] Avoid OOM in variant tests
    omit 96961d98738b [SPARK-52217][SQL] Skip outer reference validation under Filter in single-pass resolver
    omit 031ce8124591 [SPARK-52221][SQL] Refactor SqlScriptingLocalVariableManager into more generic context manager
    omit aee432cb2061 [SPARK-52178][BUILD][FOLLOW-UP] Pass ANSWER environment variable to Docker container
    omit bf6fa768f574 [SPARK-52178][BUILD][FOLLOW-UP] Respect ANSWER for checking tags
    omit 14fca6551e39 Revert "[SPARK-51927][SPARK-52213][BUILD] Upgrade jackson to 2.19.0 and upgrade kubernetes-client to version 7.3.0"
    omit 6e7c45f3ec8b [SQL][MINOR] Fix error desription for RECURSION_LEVEL_LIMIT_EXCEEDED
    omit ba9e0ae576d1 [SPARK-51927][SPARK-52213][BUILD] Upgrade jackson to 2.19.0 and upgrade kubernetes-client to version 7.3.0
    omit ece5c28b3ce7 [SPARK-52203][SQL] New iteration of single-pass Analyzer functionality
    omit 436ca7a7317a [SPARK-52101][SQL] Disable Inline Forcing for rCTEs
    omit a08b3fdeff7a [SPARK-52159][SQL] Properly handle table existence check for jdbc dialects
    omit 810a547e20e6 [SPARK-52109][SQL] Add listTableSummaries API to Data Source V2 Table Catalog API
    omit 6a75387ecb11 [SPARK-52189][SQL] New golden file tests for NATURAL JOIN with GROUP BY, HAVING, ORDER BY and LCAs
    omit 7e363747b15e [SPARK-52192][ML][CONNECT] MLCache loading path check
    omit 7604f677d928 [SPARK-51272][CORE] Aborting instead of continuing partially completed indeterminate result stage at ResubmitFailedStages
    omit d7a5d4a152db [SPARK-52211][CONNECT] Strip  suffix from `SparkConnectServer` INFO log
    omit 7fbfeae25175 [SPARK-52191][ML][CONNECT] Remove Java deserializer in model local path loader
    omit 7de318e61204 [SPARK-52158][CONNECT][SQL] Add InvalidInputErrors object to centralize errors in SparkConnectPlanner
    omit 702738bb8f59 [SPARK-52165][SQL][SDP] Setup build scaffolding for the pipelines project
    omit c09ac2373e1c [SPARK-52188] Fix for StateDataSource where StreamExecution.RUN_ID_KEY is not set
    omit af6499fd1996 [SPARK-52181] Increase variant size limit to 128MiB
    omit c8327c5e290c [SPARK-52193][K8S] Add `Spark Connect` port to Spark Driver pod and service
    omit 02f196a4f2c0 [SPARK-52156][SQL] Put legacy CREATE TEMPORARY TABLE ... USING provider under the flag
    omit f1fe275e5a85 [SPARK-52174][CORE] Enable `spark.checkpoint.compress` by default
    omit 5b07e52c987c [SPARK-52180][INFRA] Create artifacts for output and log files in dry-run build
    omit 71a046b73abe [SPARK-52157][SQL] Use earlier computed name for OuterReference in single-pass resolver
    omit 458cf70c0aa5 [SPARK-50762][SQL][TESTS] Add more scalar SQL UDF SQL query tests
    omit 02389836a2ed [SPARK-52134] Move execution logic to SqlScriptingExecution and enable Spark Connect path
    omit da2c7dd83a92 [SPARK-52122][ML][CONNECT] Fix DefaultParamsReader RCE vulnerability
    omit 2d234d96151b [SPARK-51301][BUILD] Bump zstd-jni 1.5.7-3
    omit bf7ad391b04f [SPARK-52175][INFRA] Make a scheduled build for releasing Apache Spark (dry-run)
    omit f77ab1ec0f9b [SPARK-52130][ML][CONNECT] Refine error message, and hide internal spark config
    omit d7d8b243c7d7 [SPARK-52178][BUILD] Make release scripts to respect SKIP_TAG and ANSWER when already set with DEBUG_MODE
    omit 26a74598830a [SPARK-52144][PYTHON][TESTS] Refactor memory profiling tests with shared assertion utility
    omit a7d315ada3cc [SPARK-52170][PS][TESTS] Add test to verify divide-by-zero behavior in Pandas on Spark
    omit 781031cd7160 [SPARK-52145][PYTHON][TESTS] Add tests for mapInPandas schema mismatch
    omit 6f8fa6b08ff3 [SPARK-52100][SQL] Enable overriding the recursion level limit in syntax
    omit 2b5cd7dc59e8 [SPARK-52096][SQL][SS] Reclassify kafka source offset assertion error
    omit cc7f88b05a90 [SPARK-49872][CORE] allow unlimited json size again
    omit cea3ae4e8bf9 [SPARK-52127][SS][PYTHON] Squeeze the protocol of MapState KEYS / VALUES / ITERATOR for transformWithState in PySpark
    omit 6b86244834fc [SPARK-52078][TEST] Add ZStandardTPCDSDataBenchmark
    omit ee9a6dce8e48 [SPARK-52132][BUILD] Upgrade `arrow-java` to 18.3.0
    omit 83883ea71033 [SPARK-52080][DOCS][FOLLOW-UP] Add missing workflow `build_python_connect35` and `build_coverage`
    omit a52bb6dbff8a [SPARK-52080][DOCS][FOLLOWUP] Fix the heading level of `Daily Build Pipeline Status`
    omit 87d518680abb [SPARK-52131][INFRA] Upload the application directory when tests for the `yarn` module fail
    omit d4c8f79b2024 [SPARK-52052][CORE][FOLLOW-UP] Remove extra broadcast
    omit 3fad557a7528 [MINOR][PYTHON][TESTS] Fix JobCancellationTests for pypy3.10
    omit 6ac21cf9ad3a [SPARK-51903][SQL] Validate data on adding a CHECK constraint
    omit b5edc11f7993 [SPARK-52129] Improve ParserUtils to support more Scripting constructs
    omit 092223365928 [SPARK-52126][SS][PYTHON] Revert rename for TWS utility classes for forward compatibility in Spark Connect
    omit 53d1fd54bfea [SPARK-52116][SQL] Improve exception for non-deterministic default values
    omit d183eb1e1fb0 [SPARK-52120][ML][CONNECT] Pass the session to ConnectHelper
    omit 8af74e8c1cbb [SPARK-51856][FOLLOW-UP] Estimate ALS model size on one pass
    omit 7b22a9b7471f [SPARK-52114][PYTHON][TESTS] Test UDF evaluation type under useArrow with session-level Arrow config
    omit 2f7cd6e13dce [SPARK-52112][PYTHON][TESTS] Refactor UDF profiling tests with shared assertion utility
    omit 18aebd8eb86b [SPARK-52099][SQL] Enable overriding the recursion row limit by adding a LIMIT operator
    omit 14dd441b1e1a [SPARK-52092][PYTHON][TESTS] Add UDF tests for empty DataFrames
    omit f7c10b53c1fb [SQL][MINOR] SQL Scripting parser improvements
    omit 0be0279758fe Revert SPARK-52046 and SPARK-52019: Strip outer references and prettify correlated aggregate functions
    omit 897150a4ffd0 [SPARK-52014][SQL] Support FoldableUnevaluable in HiveGenericUDFEvaluator
    omit e70f39e9a671 [SPARK-52087][SQL] Add copying of tags and origin to AliasHelper.trimNonTopLevelAliases
    omit 5f5d818671d2 [SPARK-52022][PYTHON][FOLLOW-UP] Explicitly check thrown exceptions
    omit a57b8d77a0e5 [SPARK-50585][PYTHON][DOCS] Render interactive PySpark plot examples
    omit d67208a83bf5 [SPARK-46685][DOCS] Refine PySpark profiler user guide for clarity
    omit ef1553d6b7b3 [SPARK-52080][DOCS][FOLLOWUP] Fix a wrong link in `Daily Build Pipeline Status` table
    omit 37028fafc4f9 [SPARK-52079] Normalize order of attributes in inner project lists
    omit 1582d185c661 [SPARK-52080][DOCS] Display the Daily Build Pipeline Status in `README.md`
    omit dfc817502dfb [SPARK-52077][PYTHON][CONNECT][TEST] Skip ArrowUDTFParityTests in Spark Connect compatibility test
    omit dcac3f5fff31 [SPARK-52076][CONNECT] Explicitly closes ExecutePlanResponseReattachableIterator after usage
    omit b50690a89a81 [SPARK-52057][ML][CONNECT] Collect Tree size limit warning messages to client
    omit 1016f274a95e [SPARK-51906][SQL][FOLLOW-UP] Enforce ANSI mode in previous tests
    omit 4788bae2974a [MINOR][PYSPARK][SQL] fixed a typo in python/pyspark/sql/column.py
    omit aa89883746b0 [SPARK-52046][SQL] Prettify OuterReference on AggregateExpression names when using toPrettySQL
    omit 10c761444cd6 [SPARK-52055][PYTHON][CONNECT] Suppress the exception in ExecutePlanResponseReattachableIterator.__del__
    omit 20e94a54e844 [SPARK-52052][CORE] Add `.broadcast` in the companion object of `SerializableConfiguration`
    omit 568f92017fe1 [SPARK-52051][ML][CONNECT] Enable model summary when memory control is enabled
    omit 207e296870a1 [SPARK-51789][CORE][FOLLOW-UP] Set the initial Spark Connect mode properly
    omit da2b44931749 [SPARK-52053][PYTHON][TESTS] Enable test_job_cancellation and its parity test
    omit 586c440c1418 [SPARK-51884][SQL] Part 1.a Add outer scope attributes for SubqueryExpression
    omit a14d73e71576 [SPARK-52042][SQL] Fix an error of supported typed literals
    omit 98315521c358 [SPARK-52047][PYTHON] Raise PySparkValueError for unsupported plot kinds
    omit 1cec8567df1f [SPARK-51906][SQL] Dsv2 expressions in alter table add columns
    omit 6074fd9230b1 [SPARK-52049] Fix the bug that XML attributes can't be parsed as Variant
    omit 3bfe059db168 [SPARK-51983][PS][TESTS][FOLLOW-UP] Replace `assertEquals` with `assertEqual` for python 3.12+
    omit edc631211452 [SPARK-52013][CONNECT][ML] Remove `SparkConnectClient.ml_caches`
    omit d3c0f2f30b8f [SPARK-52033][SQL] Fix Generate node bug fix where the output of the child node can have multiple copies of the same Attribute
    omit a0a9ff0f388c [SPARK-51291][SS] Reclassify validation errors thrown from state store loading
    omit 688281ae13ec [SPARK-52040][PYTHON][SQL][CONNECT] ResolveLateralColumnAliasReference should retain the plan id
    omit e74c5c45e40d [SPARK-52006][SQL][CORE] Exclude CollectMetricsExec accumulator from Spark UI + event logs + metric heartbeats
    omit 271cd0d44f2e [SPARK-51981][SS] Add JobTags to queryStartedEvent
    omit 0fb544583f0c [SPARK-52036][SQL] Add SQLConfHelper trait to v2.FileScan
    omit bead4389cf32 [SPARK-52037][SQL] Order in LCA inner project lists should respect order in original project list
    omit f4f4b91d62af [SPARK-52039][CORE][SQL][TESTS] Remove unnecessary `with PrivateMethodTester` from `ClosureCleanerSuite2` and `InsertSuite`
    omit d48a9a3083d3 [SPARK-52038][BUILD] Upgrade icu4j to 77.1
    omit 1a010a1a1984 [SPARK-52035][ML] Decouple LinearRegressionTrainingSummary and LinearRegressionModel
    omit 584f4689ae30 [SPARK-52026][PS] Block pandas API on Spark on ANSI mode by default
    omit 8acb30d9e677 [SPARK-52017][SQL] Enable multiple self-references and self-references from a Subquery inside rCTEs
    omit f64f2d6d7084 [SPARK-51902][SQL] Enforce check constraint on table insertion
    omit 604bb34b26dc [SPARK-52034][SQL] Add common methods in SparkOperation trait for thriftserver operations
    omit f68452d48f03 [SPARK-52027][K8S][DOCS] Update `YuniKorn` docs with `1.6.3`
    omit bc013c031b6b [SPARK-51835][SQL][TESTS] Extend InMemoryTable to support table versions
    omit f643b64f4d47 [SPARK-52021][SQL] Separate operator and expression patterns
    omit 4574feb8b2e5 [SPARK-51849][SQL][FOLLOWUP] ApplyDefaultCollationToStringType.fetchDefaultCollation should be consistent with transform
    omit ca77925c86d6 [SPARK-51974][CONNECT][ML] Limit model size and per-session model cache size
    omit 1d0e82fb59af [SPARK-52019][SQL] Strip outer reference before creating a name in toPrettySQL
    omit d80e857a7930 [SPARK-51818][CONNECT] Move QueryExecution creation to AnalyzeHandler and don't Execute for AnalyzePlanRequests
    omit bb0b2d29d37e [SPARK-51821][CORE] Call interrupt() without holding uninterruptibleLock to avoid possible deadlock
    omit 9178c2c31fd7 [SPARK-51998][BUILD] Upgrade ORC to 2.1.2
    omit 39f6d25d8640 [SPARK-52022][PYTHON] Add additional check in default method for SparkThrowable.getQueryContext
    omit 9e7bfc54b2ce [MINOR][TESTS] Reduce the number of iterations in tuning tests
    omit 56be8b0204dc [SPARK-51983][PS] Prepare the test environment for pandas API on Spark with ANSI mode enabled
    omit b957d7f6f331 [SPARK-44856][PYTHON] Improve Python UDTF arrow serializer performance
    omit d5f735b54a4d [SPARK-51905][SQL] Disallow NOT ENFORCED CHECK constraint
    omit 633419a8c7a3 [SPARK-51989][PYTHON] Add missing Filter subclasses to __all__ list in datasource
    omit 30cf5b456123 [SPARK-52015][BUILD][TESTS] Update Oracle free version from 23.6 to 23.7
    omit 0eab1c0d0389 [SPARK-52007][SQL] Expression IDs shouldn't be present in grouping expressions when using grouping sets
    omit b02afd869714 [SPARK-51182][SQL] DataFrameWriter should throw dataPathNotSpecifiedError when path is not specified
    omit 6e89fa9e8945 Revert "[SPARK-51905][SQL] Disallow NOT ENFORCED CHECK constraint"
    omit 9f5ae883899a [SPARK-52010] Do not generate API docs for internal classes
    omit 61a6dee3ed4e [SPARK-51905][SQL] Disallow NOT ENFORCED CHECK constraint
    omit 3cae318d4a4d [SQL][MINOR][FOLLOW-UP] Fix warnings in DataSource/WriterV2Suites
    omit 4d4fdc0604fb [SPARK-50238][PYTHON][FOLLOW-UP] Fix Arrow UDF bug where Struct with metadata field was failing on the is_variant path
    omit 7fcb12684af3 [SPARK-51964][SQL] Correctly resolve attributes from hidden output in ORDER BY and HAVING on top of an Aggregate in single-pass Analyzer
    omit 894d82898673 [SPARK-51947] Spark connect model cache offloading
    omit e857f43cde5a [MINOR][PS][DOC] Update pandas API on Spark option doc
    omit 15732fcffad2 [SPARK-51950][BUILD] Upgrade `Parquet` to 1.15.2
    omit 98c44d0c1010 [SPARK-51985][CORE] Remove `Experimental` from `(Long|Double)AccumulatorSource`
    omit fd56ef68efda [SPARK-51979][SQL][TESTS] Add more SQL Query tests for SQL TVF return columns
    omit 31f7278bc00b [SPARK-51980][PYTHON][TESTS] Enable `--use-pep517` in `dev/run-pip-tests`
    omit 085bfc21ce0e [SPARK-51973][K8S][BUILD] Upgrade `kubernetes-client` to 7.2.0 for K8s 1.33
    omit 7019d5e63b72 [SPARK-51867][ML][FOLLOW-UP][ML] Use private to avoid exposing Data class
    omit a523d66755c0 [SPARK-51963][ML] Simplify IndexToString.transform
    omit 896b62d3780a [SPARK-51940][SS] Add interface for managing streaming checkpoint metadata
    omit db91acbde6a5 [SPARK-51958][CORE][SQL][TESTS] Replace hardcoded icu version with `c.ibm.icu.util.VersionInfo.ICU_VERSION` it tests
    omit 1c77a3ced6d8 [SPARK-51786][SQL] DSv2 expressions in column defaults in CREATE/REPLACE
    omit 3a989dbe8d93 [SPARK-51962][SQL][TESTS] Improve ORDER BY testing coverage for non-orderable sort expressions
    omit 13902f55687c [SPARK-51959][PYTHON] Make functions not import ParentDataFrame
    omit 1fe7f634240b [SPARK-50605][CONNECT][TESTS][FOLLOW-UP] Remove unused YarnConnectTest
    omit 6521297f9ce8 [SPARK-51946][SQL] Eagerly fail for creating hive-incompatible datasource table with 'col' a partition name
    omit fc1cb7867e55 [SPARK-51834][SQL] Support end-to-end table constraint management
    omit 0225903f6756 [SPARK-51428][SQL] Assign Aliases for collated expression trees deterministically
    omit aa6b5ed99ee3 [SPARK-51867][ML] Make scala model supporting save / load methods against local filesystem path
    omit 4445cd8c0c26 [SPARK-51956][K8S] Fix `KerberosConfDriverFeatureStep` to warn in case of failures
    omit 54eb1a2f863b [MINOR][PYTHON][TESTS] Clean up a group of connect tests
    omit ef336ad51ba6 [SPARK-51948][BUILD] Make changes of `mllib` module to trigger the test for the `connect` module
    omit 64a7e6de0ce2 [SPARK-51609][SQL] Optimize Recursive CTE execution for simple queries
    omit 6ad0bd448b6e [SPARK-51890][SQL] Add New Parameter `options` to Logical Plan `View`
    omit 946d2d74ce0f [SPARK-51941][SQL] Fix the precision problem for convertToCatalyst when the Input is BigDecimal between -1.0 and 1.0
    omit ad387ce02816 Revert "[SPARK-51867][ML] Make scala model supporting save / load methods against local filesystem path"
    omit 0614666bbc98 [SPARK-51944][BUILD] Upgrade Netty to 4.1.121.Final
    omit 6f9bf73c345d [SPARK-51867][ML] Make scala model supporting save / load methods against local filesystem path
    omit 86bf4c84805e [SPARK-51931][SQL] Add maxBytesPerOutputBatch to limit the number of bytes of Arrow output batch
    omit 84c972742fd4 [MINOR][DOCS] Minor update to example
    omit aea583640f4a [SPARK-51936][SQL] ReplaceTableAsSelect should overwrite the new table instead of append
    omit 02faae539e34 [SPARK-51802][PYTHON][DOCS] OSS PySpark User Guide Docs
    omit 291f4a3e3db9 [SPARK-51935][SQL] Fix lazy behavior of iterators in interpreted df.collect()
    omit 4e2fd0710375 [SPARK-51820][SQL][CONNECT] Address remaining issues for new `group`/`order` by ordinal approach
    omit 99270b296494 [SPARK-51901][SQL] Disallow generator functions in grouping sets
    omit c9433e21e0d1 [SPARK-51939][PYTHON][DOCS] Refine Python data source docstring by adding arrow batch examples
    omit 7fcae8cab68b [SPARK-51921][SS][PYTHON] Use long type for TTL duration in millisecond in transformWithState
    omit 48322c7ac541 [SPARK-51933][SS][DOCS] Document the new API `transformWithState` in PySpark
    omit fb2705c796ff [SPARK-51094][SQL][TESTS] Adjust some test parameters to work on s390x
    omit 6c37ee4943e7 [SPARK-51937][SQL][TESTS] Re-generate golden files of `postgreSQL/float4.sql` and `postgreSQL/int8.sql` for Java 21
    omit b0cb67996180 [SPARK-51932][CONNECT][TESTS] Enable SparkConnectProgressHandlerE2E in connect-only mode
    omit cd9af61ae11e [MINOR][DOCS] Add migration note for mapInPandas and mapInArrow validation
    omit d4513013ea70 [SPARK-51926][CORE][SQL] Add INVALID_CONF_VALUE.subXXX error class to conf errors
    omit b83a819b50ad [SPARK-51869][SS] Create classification for user errors within UDFs for Scala TransformWithState
    omit 1a9be7b0ab02 [SPARK-51930][BUILD] Upgrade datasketches-java to 6.2.0
    omit 439153819e3a [SPARK-51899][SQL] Implement error handling rules for spark.catalog.listTables()
    omit e32fb5efd337 [SPARK-51814][SS][PYTHON][FOLLOWUP][MINOR] Add missing type handling for PythonEvalType.toString
    omit aeff6796020a [SPARK-51880][ML][PYTHON][CONNECT] Fix ML cache object python client references
    omit 6f47783d3512 [SPARK-51827][SS][CONNECT] Support Spark Connect on transformWithState in PySpark
    omit 7292ff1ec763 [SPARK-51823][SS] Add config to not persist state store on executors
    omit 7832e917d0d5 [SPARK-51928][BUILD] Upgrade Apache `common-text` to 1.13.1
    omit 116c20ce0d2b [SPARK-51757][SQL] Fix LEAD/LAG Function Offset Exceeds Window Group Size
    omit 1f43d3763b73 [SPARK-51929][BUILD] Upgrade AWS SDK v2 to 2.29.52
    omit 4c9c41e0b7c4 Revert "[SPARK-51827][SS][CONNECT] Support Spark Connect on transformWithState in PySpark"
    omit f3efe1a4948b [SPARK-51924][BUILD] Upgrade jupiter-interface to 0.14.0 and Junit5 to 5.12.2
    omit 81ede347e3e2 [SPARK-51827][SS][CONNECT] Support Spark Connect on transformWithState in PySpark
    omit 528fe202ca93 [SPARK-51817][SPARK-49578][CONNECT] Re-introduce ansiConfig fields in messageParameters of CAST_INVALID_INPUT and CAST_OVERFLOW
    omit 3aa757000173 [SPARK-51898][PYTHON][CONNECT][TESTS][FOLLOW-UP] Fix inconsistent exception
    omit 30540df11bb9 [SPARK-51814][SS][PYTHON][FOLLOWUP] Sync missing Python function types
    omit 6cb9a05d1048 [SPARK-51923][BUILD] Upgrade Apache `commons-collections4` to 4.5.0
    omit 0d42e519f5ae [SPARK-51914][SQL] Add com.mysql.cj to spark.sql.hive.metastore.sharedPrefixes
    omit 276ca7ffb19b [SPARK-51913][SQL] JDBCTableCatalog#loadTable should throw no such table error correctly
    omit 08cf70240595 [SPARK-51925][CONNECT][TESTS] Fix the erroneous `assume` in the `ClassFinderSuite`
    omit b63497893649 [SPARK-51922][SS] Fix UTFDataFormatException thrown from StateStoreChangelogReaderFactory for v1
    omit 23785d33acab [SPARK-51904][SS] Removing async metadata purging for StateSchemaV3 and ignoring non-batch files when listing OperatorMetadata files
    omit c2305edd6b53 [SPARK-51900][SQL] Properly throw datatype mismatch in single-pass Analyzer
    omit 1c1d80fce693 [SPARK-51805][SQL] Get function with improper argument should throw proper exception instead of an internal one
    omit 676cac092540 [SPARK-51915][PYTHON][CONNECT][TESTS] Enable SparkConnectDataFrameDebug in connect-only mode
    omit e3d0c0fe660f [MINOR][PYTHON][DOCS] Add 4 missing functions to API references
    omit 80ed987c1029 [SPARK-51909][INFRA][FOLLOW-UP] Fix python lint
    omit 72d11ea32713 [SPARK-51908][K8S][TESTS] Update `setup-minikube` to v0.0.19
    omit 8c808aff7415 [SPARK-51896][CORE][SQL] Add Java Enum Support for TypedConfigBuilder
    omit 98ba05ecf5b3 [SPARK-51878][SQL] Improve fillDefaultValue by exec the foldable default expression
    omit 87cd48f86907 [SPARK-51909][INFRA] Add a scheduled workflow for PySpark Classic-only
    omit 2c0fc708b88f [SPARK-51891][SS] Squeeze the protocol of ListState GET / PUT / APPENDLIST for transformWithState in PySpark
    omit 83db3984761b [SPARK-51881][SQL] Make AvroOptions comparable
    omit bab6988cb4f9 [SPARK-51897][SQL] Fix compilation warnings related to "The auto insertion will be deprecated, please write `TimeType.apply` explicitly"
    omit 8da4904cfdb7 [SPARK-51898][PYTHON][CONNECT][TESTS] Refactor SparkConnectErrorTests to support connect-only mode
    omit d2a3f88f4359 [SPARK-51895][K8S][INFRA] Update K8s IT CI to use K8s 1.33
    omit bdc52322d176 [MINOR][PYTHON][TESTS] Further fix scheduled builder without grpc
    omit 5b7abdc265dd [SPARK-51877][PYTHON][CONNECT] Add functions 'chr', 'random' and 'uuid'
    omit 6eb966812b32 [SPARK-51889][PYTHON][SS] Fix a bug for MapState clear() in Python TWS
    omit 5a5bf04aca8a [SPARK-51874][CORE][SQL] Add TypedConfigBuilder for Scala Enumeration
    omit 516859f273b7 [SPARK-51119][SQL][FOLLOW-UP] Fix missing fallback case for parsing corrupt exists_default value
    omit 6df5cb777527 [SPARK-51439][SQL] Support SQL UDF with DEFAULT argument
    omit ce7f3352849a [SPARK-51865][CONNECT][TESTS] Print stacktrace message when `o.a.arrow.memory.BaseAllocator` fails to close
    omit d9816b709003 [SPARK-51873][ML] For OneVsRest algorithm, allow using save / load to replace cache
    omit 632e681f120f [SPARK-51820][SQL][FOLLOWUP] Fix `origin` for `UnresolvedOrdinal`
    omit 920634a0cdd4 [SPARK-51868][SQL] Move type coercion validation to a separate object
    omit a28977e25801 [SPARK-51711][ML][PYTHON][CONNECT] Propagates the active remote spark session to new threads to fix CrossValidator
    omit a9128a08c82d [SPARK-51876][PYTHON][TESTS] Add configuration for `log4j.configurationFile` in the PySpark submission args of `run_individual_python_test`
    omit 24511dd30e2c [SPARK-47791][SQL][FOLLOWUP] Avoid invalid JDBC decimal scale
    omit 124a17f1b994 [SPARK-50582][PYTHON][FOLLOW-UP] Add `quote` to functions `__all__`
    omit d7ce6ef55e5f [SPARK-51866][CONNECT][TESTS] Ensure `serializerAllocator/deserializerAllocator` are closed if `ArrowEncoderSuite#roundTripWithDifferentIOEncoders` fails to create `CloseableIterator`
    omit cdd52963281a [SPARK-51856][ML][CONNECT] Update model size API to count distributed DataFrame size
    omit 772c4cbd9f65 [SPARK-51822][SS] Throwing classified error when disallowed functions are called during StatefulProcessor.init()
    omit 58adb0174af3 [SPARK-51814][SS][PYTHON][FOLLLOW-UP] Uses `list(self)` instead of `StructType.fields` for old version compat
    omit c86f617b2f1a [SPARK-51849][SQL] Refactoring `ResolveDDLCommandStringTypes`
    omit 3a7d5d7416ea [SPARK-51862][ML][CONNECT][TESTS] Clean up ml cache before ReusedConnectTestCase and ReusedMixedTestCase
    omit 7904dd3cd612 [SPARK-51861][SQL][UI] Remove duplicated/unnecessary info of InMemoryRelation Plan Detail
    omit 1cf35f506b53 [SPARK-51774][CONNECT][FOLLOW-UP][TESTS] Skip ConnectErrorsTest if grpc is not available
    omit eab86494d0cd [SPARK-51814][SS][PYTHON][FOLLLOW-UP] Use RecordBatch.schema.names instead of column_names for old pyarrow compatibility
    omit 9bcf5595c35c [SPARK-51688][PYTHON][FOLLOW-UP] Use `socketPath.get` for logging instead of `socketHost.get`
    omit a6f842624e74 [SPARK-51860][CONNECT] Disable `spark.connect.grpc.debug.enabled` by default
    omit cafe74d49f21 [MINOR] rename the spark-connect-enabled tarball
    omit 99511599ebe9 [SPARK-51779][SS] Use virtual column families for stream-stream joins
    omit b4b1f7883a5f [SPARK-51838][PYTHON][TESTS][FOLLWO-UP] Ignore typing module in `test_wildcard_import`
    omit c77b163e4cb3 [SPARK-49488][SQL][FOLLOWUP] Do not push down extract expression if extracted field is second
    omit b557311d4394 [SPARK-51845][ML][CONNECT] Add proto messages `CleanCache` and `GetCacheInfo`
    omit 929042cfcfa5 [SPARK-51663][SQL][FOLLOWUP] change buildLeft and buildRight to function
    omit 7236d7dd5004 [SPARK-51833][SQL] Normalize relations in NormalizePlan
    omit 48ff0d3ba777 [SPARK-51838][PYTHON][TESTS][FOLLOW-UP] Skip `test_wildcard_import` in Python 3.10 and below
    omit f2e65e06526f Revert "[SPARK-51838][PYTHON][TESTS][FOLLWO-UP] Skip `test_wildcard_import` in low python versions"
    omit 0955193c2be5 [SPARK-51691][CORE] SerializationDebugger should swallow exception when try to find the reason of serialization problem
    omit a10f32da21b8 [SPARK-51838][PYTHON][TESTS][FOLLWO-UP] Skip `test_wildcard_import` in low python versions
    omit 8fef8ea6b687 [SPARK-51814][SS][PYTHON][FOLLOW-UP] Add pyspark.sql.tests.pandas.helper for testing pyspark-client
    omit e50fe86dda2e [SPARK-51844][PYTHON][TESTS] Add ReusedMixedTestCase for test env with both classic and connect
    omit cb325704174d [SPARK-51688][PYTHON][FOLLOW-UP] Log string instead of option instance
    omit ffc042868cd0 [SPARK-51843][PYTHON][ML][TESTS] Avoid per-test classic session start & stop
    omit 35b003b39cf8 [SPARK-51848][SQL] Fix parsing XML records with defined schema of array/structs/map of Variant
    omit ad2f611e40b7 [SPARK-51840][SQL] Restore Partition columns in HiveExternalCatalog#alterTable
    omit 2e16568ddc96 [SPARK-51553][SQL] Modify EXTRACT to support TIME data type
    omit 87b9866903ba [SPARK-51801][BUILD] Upgrade ORC Format to 1.1.0
    omit cf804610c15e Revert "[SPARK-51691][CORE][TESTS] SerializationDebugger should swallow exception when try to find the reason of serialization problem"
    omit ec430279b43d [SPARK-51814][SS][PYTHON] Introduce a new API `transformWithState` in PySpark
    omit e616b8404ff2 [SPARK-51836][PYTHON][CONNECT][TESTS][FOLLOW-UP] update `test_connect_classification`
    omit a6a251cffc11 [SPARK-51774][CONNECT][FOLLOW-UP][TESTS] Skip ConnectErrorsTest if grpc is not available
    omit 0ef86631e206 [SPARK-51838][PYTHON][TESTS] Add a test to check function wildcard import
    omit 40093c58be38 [SPARK-51836][PYTHON][CONNECT][TESTS] Avoid per-test-function connect session setup
    omit 23c77b7e701b [SPARK-51411][SS][DOCS] Add documentation for the transformWithState operator
    omit 0d3d3a47d3a5 [SPARK-51829][PYTHON][ML] Client side should update `client.thread_local.ml_caches` after deletion
    omit 8b33a832b2ba [SPARK-50967][SS][MINOR][FOLLOW-UP] Update `flatMapGroupsWithState` test conf
    omit 290502d1c9f8 [SPARK-51820][SQL] Move `UnresolvedOrdinal` construction before analysis to avoid issue with group by ordinal
    omit d7992b1ed7da [SPARK-51372][SQL][FOLLOW-UP] Retain the property map for DataSourceV2 TableInfo
    omit 257e1727efbb [MINOR][PYTHON] FIX change type hint of pyspark percentile functions to reflect tuple of variable amount of floats
    omit 335063fc8f88 [SPARK-51758][SS] Fix test case related to extra batch causing empty df due to watermark
    omit 45082b8ccd07 [SPARK-51803][CORE] Store external engine JDBC type in the spark schema in the…
    omit 9cea811bdf3a [SPARK-51832][BUILD] Use -q option to simplify maven version evaluation
    omit 564514899ca7 Revert "[SPARK-51758][SS][FOLLOWUP][TESTS] Fix flaky test around watermark due to additional batch causing empty df"
    omit b2954ca3fbb1 [SPARK-51826][K8S][DOCS] Update `YuniKorn` docs with `1.6.2`
    omit 6defd6956d80 [SPARK-51824][ML][CONNECT][TESTS] Force to clean up the ML cache after each test
    omit e9b504269f98 Revert "[SPARK-51824][ML][CONNECT][TESTS] Force to clean up the ML cache after each test"
    omit 216f05bed485 [SPARK-51414][SQL] Add the make_time() function
    omit 2b78cec410ca [SPARK-51758][SS][FOLLOWUP][TESTS] Fix flaky test around watermark due to additional batch causing empty df
    omit 3cebd7df1b90 [SPARK-51800][INFRA][FOLLOW-UP] Respect PYSPARK_UDS_MODE environment variable in SparkContext initialization
    omit 777044c173f2 [SPARK-51797][SQL] Add table name to TableConstraint and remove parser dependencies
    omit 19440a146a4c [SPARK-51824][ML][CONNECT][TESTS] Force to clean up the ML cache after each test
    omit 9ee1129b623a [SPARK-51739][PYTHON][FOLLOW-UP] Set spark.sql.execution.arrow.pyspark.validateSchema.enabled for 3.5 connect client build
    omit bfe95586b6a8 [SPARK-51816][SQL] Simplify `StatFunctions.multipleApproxQuantiles` with dataframe APIs
    omit 61e23effce4d [SPARK-51819][PYTHON] Update pyspark-errors test module to include missing tests
    omit cd47c3ceeeda [SPARK-50718][PYTHON][TESTS][FOLLOW-UP] Enable `pyspark.sql.tests.test_artifact`
    omit ed702c0db71a [SPARK-51812][SQL] Remove redundant parameters of some methods in `QueryExecution`
    omit 2ba156096e83 [SPARK-51811][BUILD] Upgrade `commons-io` to 2.19.0
    omit fd44ab6aaedc [SPARK-51806][BUILD] Upgrade `kryo-shaded` to 4.0.3
    omit beb71bb5a443 [SPARK-51813][SQL][CORE] Add a nonnullable DefaultCachedBatchKryoSerializer to avoid null propagating in DefaultCachedBatch serde
    omit 0a79406d25aa [SPARK-51773][SQL] Turn file formats into case classes to properly compare them
    omit e00e189392f9 [SPARK-51762][SQL] Fix Resolution of Views in Single-Pass Analyzer When Bridging is Enabled
    omit e14e60c8e2ed [SPARK-51771][SQL] Add DSv2 APIs for ALTER TABLE ADD/DROP CONSTRAINT
    omit a5510808b30a [SPARK-51768][SS][TESTS] Create Failure Injection Test for Streaming offset and commit log write failures
    omit 6e8b4b5a6116 [SPARK-51638][CORE] Fix fetching the remote disk stored RDD blocks via the external shuffle service
    omit b4faf063e375 [SPARK-50131][SQL] Add IN Subquery DataFrame API
    omit 203942c58318 [SPARK-51688][PYTHON][FOLLOW-UP] Implement UDS in Accumulators
    omit 23662621d08f [SPARK-51791][ML] `ImputerModel` stores coefficients with arrays instead of dataframe
    omit 9ec2670db9a6 [SPARK-51716][SQL] Support serializing Variant to XML
    omit 58eb0ad21f23 [SPARK-51794][INFRA] Install arm64 Python for MacOS daily test
    omit bb39e4ba8c09 [SPARK-51633][CORE][TESTS] Reset `Utils#customHostname` in the `finally` block of `ExecutorSuite#withExecutor`
    omit 54d3c2e945e8 [SPARK-51800][INFRA] Set up the CI for UDS in PySpark
    omit 39981866b679 [SPARK-51747][SQL][FOLLOW-UP] Data source cached plan conf and migration guide
    omit 467644ed5faf [SPARK-51790][SQL] Register UTF8String to KryoSerializer
    omit 898a0b46a0ab [SPARK-46640][FOLLOW-UP] Consider the whole expression tree when excluding subquery references
    omit d3b6dd13e9e9 [SPARK-51688][PYTHON] Use Unix Domain Socket between Python and JVM communication
    omit 53ce5c39c5bf [SPARK-51395][SQL][TESTS][FOLLOW-UP] Explicitly sets failOnError in Abs at tests
    omit bb1a63a8c361 [SPARK-51758][SS] Apply late record filtering based on watermark only if timeMode is passed as EventTime to the transformWithState operator
    omit 7087677cf185 [SPARK-51775][SQL] Normalize LogicalRelation and HiveTableRelation by NormalizePlan
    omit 5b19aa3abc8b [SPARK-51778][SQL][TESTS] Close SQL test gaps discovered during single-pass Analyzer implementation
    omit 8718eba1f6f7 Revert "[SPARK-47895][SQL] group by alias should be idempotent"
    omit c3af3dc0b70e [SPARK-51788][INFRA] Add a PySpark test that runs every 3 days using the Ubuntu Arm Runner
    omit 53081516b331 [SPARK-51780][SQL] Implement Describe Procedure
    omit 0f5c98d3f3bf [SPARK-51789][CORE] Respect spark.api.mode and spark.remote properly when parsing arguments in Spark Submission
    omit da67217590cd [SPARK-51513][SQL] Fix RewriteMergeIntoTable rule produces unresolved plan
    omit 1e717fe2b556 [SPARK-51776][SQL] Fix logging in single-pass Analyzer
    omit 919e60f84ca2 [SPARK-51739][PYTHON] Validate Arrow schema from mapInArrow & mapInPandas & DataSource
    omit 5102370dcf37 [SPARK-51774][CONNECT] Add GRPC Status code to Python Connect GRPC Exception
    omit ece4bc0759b8 [SPARK-51752][SQL] Enable rCTE referencing from within a CTE
    omit bc5ccad3733f [SPARK-51660][CORE] Gracefully handle when MDC is not supported
    omit 1fa05b8cb755 [SPARK-51777][SQL][CORE] Register sql.columnar.* classes to KryoSerializer
    omit b9dbf8be4685 [SPARK-51419][SQL][FOLLOWUP] Making hour function to accept any precision of TIME type
    omit adc42b4b99ac [SPARK-51420][SQL][FOLLOWUP] Support all valid TIME precisions in the minute function
    omit c577ae7202a7 [SPARK-51423][SQL] Add the current_time() function for TIME datatype
    omit 01c16af41518 [SPARK-51769][SQL] Add maxRecordsPerOutputBatch to limit the number of record of Arrow output batch
    omit 9bc8cd51b671 [SPARK-51358][SS] Introduce snapshot upload lag detection through StateStoreCoordinator
    omit 622fa35bb003 [SPARK-51537][CONNECT][CORE][FOLLOW-UP][TEST] Add test to ensure Spark plugins are not reloaded
    omit 104dc2b708c4 [SPARK-51503][SQL] Support Variant type in XML scan
    omit d4eea4c32d25 [SPARK-51723][SQL] Add DSv2 APIs for create/get table constraints
    omit cabbcabb4f48 [SPARK-51726][SQL] Use TableInfo for Stage CREATE/REPLACE/CREATE OR REPLACE table
    omit a33b9fc60cae [SPARK-51691][CORE][TESTS] SerializationDebugger should swallow exception when try to find the reason of serialization problem
    omit a0bc04437540 [SPARK-51761][INFRA] Add a daily test using the Ubuntu Arm Runner
    omit 6d731ee7665e [SPARK-51760][BUILD] Upgrade ASM to 9.8
    omit 6cdf54b341ee [SPARK-51630][CORE][TESTS] Remove `pids` size check from "SPARK-45907: Use ProcessHandle APIs to computeProcessTree in ProcfsMetricsGetter"
    omit 0e8cead01270 [SPARK-51699][BUILD] Upgrade to Apache parent pom 34
    omit d2a864f988c7 [SPARK-51747][SQL] Data source cached plan should respect options
    omit a9987a38d5fd [SPARK-51738][SQL][FOLLOWUP] Fix HashJoin to accept structurally-equal types
    omit c8cdeeca9236 [SPARK-51751][SQL] Fix multiple rCTEs for one WITH statement that reference each other
    omit c7ef21a49ac4 [SPARK-51754][PYTHON][DOCS][TESTS] Make `sampleBy` doctest deterministic
    omit 1472ceb3c787 [SPARK-51755][INFRA][PYTHON] Set up a scheduled builder for free-threaded Python 3.13
    omit 99b7d2a7441b [SPARK-51673][SQL] Apply default collation to alter view query
    omit 66926429450a [SPARK-51738][SQL] IN subquery with struct type
    omit 5602fbf93f9e [SPARK-51421][SQL] Get seconds of TIME datatype
    omit f4ce0ab8cee0 [SPARK-51512][CORE] Filter out null MapStatus when cleaning up shuffle data with ExternalShuffleService
    omit 0246a7cbac10 [SPARK-51732][SQL] Apply `rpad` on attributes with same `ExprId` if they need to be deduplicated
    omit c7024d35c45a [SPARK-51714][SS] Add Failure Ingestion test to test state store checkpoint format V2
    omit 3f16577c2199 [SPARK-51717][SS][ROCKSDB] Fix SST mismatch corruption that can happen for second snapshot created for a new query
    omit aa5a0a367bc2 [SPARK-51733][BUILD] `make-distribution.sh` supports sbt
    omit 53966ae9eba9 [SPARK-51296][SQL] Support collecting corrupt data in singleVariantColumn mode
    omit daf9fe3e84f5 [SPARK-49862][INFRA][PYTHON] Restore package installations for Python 3.13 daily builder
    omit 367d2069ab93 [MINOR][PYTHON][DOCS] Fix a typo in pyspark documentation
    omit cba9454f4273 [SPARK-51695][SQL] Introduce Parser Changes for Table Constraints (CHECK, UNIQUE, PK, FK)
    omit 9efddcf4a50b [SPARK-51611][SQL] New iteration of single-pass Analyzer functionality
    omit 187adb83fae9 [SPARK-51724][SS] RocksDB StateStore's lineage manager should be synchronized
    omit 92f5d38ac397 [SPARK-51711][ML][CONNECT] Memory based MLCache eviction policy
    omit 6581e8cccd79 [SPARK-51742][BUILD][TESTS] Upgrade `snowflake-jdbc` to 3.23.2
    omit 78bbd3948c69 [SPARK-51721][SQL] Change default value of ANALYZER_SINGLE_PASS_RESOLVER_RELATION_BRIDGING_ENABLED flag
    omit edebb1105be7 [SQL][MINOR] Fix warnings in DataSource/WriterV2Suites
    omit 554d67817e44 [SPARK-51712][SQL] Swallow non-fatal Throwables when resolving tables/views in spark.catalog.listTables()
    omit 54288f877ca0 [SPARK-51722][SQL] Remove "stop" origin from ParseException
    omit 738a50364f16 [SPARK-51395][SQL] Refine handling of default values in procedures
    omit 24e400045959 [SPARK-51496][SQL][FOLLOW-UP] Preserve the case of DataSourceV2Relation option keys
    omit 418cfd1f7801 [SPARK-51690][SS] Change the protocol of ListState.put()/get()/appendList() from Arrow to simple custom protocol
    omit 75d80c7795ca [SPARK-51079] Added constructor for ArrowSerializer for backward compatibility
    omit 24f600f7cc12 [SPARK-51725][SQL][TEST] Supplement tests for merge into
    omit ba7a537225a7 [SPARK-50953][FOLLOW-UP] Allow whitespace/tab in variantGet paths
    omit eae5ca789f61 [SPARK-51704][SQL] Eliminate unnecessary collect operation
    omit 9d3f937c555c [SPARK-51720][SQL] Add Cross Join as legal in recursion of Recursive CTE
    omit babb950a7fc7 [SPARK-51646][SQL] Fix propagating collation in views with default collation
    omit 8def486a0fc3 [SPARK-51350][SQL][FOLLOW-UP] Cleanup Show Procedure Test
    omit 3ab0f2de54af [SPARK-51675][SS][FOLLOW-UP] Clear internal maps on close to remove DB open dep in other places
    omit b0a7e2e01ccc [SPARK-51685][SS] Excessive Info logging from RocksDb operations causing too big executor stderr files
    omit 510edd763b91 [SPARK-51707][CONNECT][TESTS] Handling of `IllegalStateException` with `Memory leaked` message in the `afterAll` of `RemoteSparkSession`
    omit a222f78cc2ba [SPARK-51700][SS] Fix incorrect logging when no files are eligible for deletion in RocksDBFileManager
    omit a722b43c4ce1 [SPARK-51686][UI] Link the execution IDs of sub-executions for current execution if any
    omit 211208a6f1c3 [SPARK-51703][BUILD] Upgrade jetty to 11.0.25
    omit 295d37fad3b6 [SPARK-51701][PYTHON][TESTS] Move test objects to a separate file
    omit ad0bc93fe6ab [SPARK-51677][CONNECT][TESTS] Refactor `ClientE2ETestSuite/SparkSessionE2ESuite` to prevent memory leaks related to Arrow
    omit 7347cac4b723 [SPARK-51298][SQL] Support variant in CSV scan
    omit efcef707022b [SPARK-51698][INFRA] Ignore sql/api/src/main/gen/ in Git
    omit beb509fe6d79 [SPARK-51687][SQL] Pushdown filters with TIME values to parquet
    omit 5d71afc7ea0f [SPARK-51682][SS] State Store Checkpoint V2 should handle offset log ahead of commit log correctly
    omit 3ec2a9baf7b6 [SPARK-51681][CONNECT][BUILD] Remove redundant `build-helper-maven-plugin` configuration from connect related modules
    omit c202bb2c3799 [SPARK-51333][ML][PYTHON][CONNECT] Unwrap `InvocationTargetException` thrown in `MLUtils.loadOperator`
    omit 7cc72bd37804 [SPARK-51684][PYTHON][SS] Fix test failure in test_pandas_transform_with_state
    omit 2f1bce2a5c85 [SPARK-51441][SQL] Add DSv2 APIs for constraints
    omit 921eba838bf1 [SPARK-51680][SQL] Set the logical type for TIME in the parquet writer
    omit 6d606139d578 [SPARK-51668][SQL] Report metrics for failed writes to V2 data sources
    omit 88399847bd6b [SPARK-51655][SQL] Fix metric collection in UnionLoopExec and add test
    omit 4e5ed454fb29 [SPARK-51666][CORE] Fix sparkStageCompleted executorRunTime metric calculation
    omit c0fbc6ba7d2d [SPARK-51675][SS] Fix col family creation after opening local DB to avoid snapshot creation, if not necessary
    omit 13945c84731f [SPARK-51657] UTF8_BINARY default table collation shown by default in Desc As JSON (v1)
    omit 52bc072cdb1c [SPARK-51670][SQL] Refactor Intersect and Except to follow Union example to reuse in single-pass Analyzer
    omit c83b88a037ee [SPARK-51650][ML][CONNECT] Support delete ml cached objects in batch
    omit 45eb03c5119a [SPARK-51674][DOCS] Remove unnecessary Spark Connect doc link from Spark website
    omit d642b6bd174a [SPARK-51252][SS][TESTS] Make tests more consistent for StateStoreInstanceMetricSuite
    omit 72bd56357965 [SPARK-51537][CONNECT][CORE] construct the session-specific classloader based on the default session classloader on executor
    omit a760df7b8434 [SPARK-51667][SS][PYTHON] Disable Nagle's algorithm (via TCP_NODELAY = true) in TWS + PySpark for python <-> state server
    omit 02db87282dab [SPARK-47895][SQL] group by alias should be idempotent
    omit a73ca4766ce6 [SPARK-51669][SQL][TESTS] Generate random TIME values in tests
    omit 4128d0b9e1a4 [SPARK-50994][SQL][FOLLOWUP] Do not use RDD with tracked execution in QueryTest
    omit be8489f66f28 [SPARK-51663][SQL] Short circuit && operation for JoinSelectionHelper
    omit 87b8f2b3a7ab [SPARK-51664][SQL] Support the TIME data type in the Hash expression
    omit c8c8c9a22edc [SPARK-51661][SQL] Partitions discovery of TIME column values
    omit e4eded884472 [SPARK-51665][BUILD] Truncate the lists in dev/test-jars.txt and dev/test-classes.txt during release process
    omit f945d01c25ae [SPARK-51651][UI] Link the root execution id for current execution if any
    omit b1275a0190ac [SPARK-51648][UI] Move inlined javascript in ExecutionPage to spark-sql-viz.js
    omit 21fad7756b98 [SPARK-40353][SPARK-51599][PS][FOLLOW-UP] Fix failure in Python PS with old dependencies
    omit 73a3abae48ce [SPARK-51612][MINOR][FOLLOW-UP][SQL] Update Desc As JSON test to use withSQLConf
    omit c9733e0460a9 [SPARK-51645][SQL] Fix `CREATE OR REPLACE TABLE ... DEFAULT COLLATION ...` query
    omit 325fb95ab50d [SPARK-51662][SQL] Make OrcFileFormat comparable
    omit 825682d3c95e [SPARK-51191][SQL] Validate default values handling in DELETE, UPDATE, MERGE
    omit 28a756222800 [SPARK-51608][PYTHON] Log exception on Python runner termination
    omit 50afb735dbc9 [SPARK-51649][SQL] Dynamic writes/reads of TIME partitions
    omit 13eeb1087d61 [SPARK-51652][SQL] Refactor SetOperation computation out to reuse it in the single-pass Analyzer
    omit 0895d195f77c [SPARK-51614][SQL] Introduce ResolveUnresolvedHaving rule in the Analyzer
    omit c9fbcb1c74ce [SPARK-51616][SQL] Run CollationTypeCasts before ResolveAliases and ResolveAggregateFunctions
    omit be448f2cf196 [MINOR][BUILD][FOLLOWUP] Fix merge_spark_pr script for no jira case
    omit 4d7deabd6b8e [SPARK-51627][INFRA][FOLLOW-UP] Fix the workflow file name
    omit 1dfb04666ddc [MINOR][DOCS] Fix variable name typo in document
    omit 936d4f8b0d1f [SPARK-51615][SQL] Refactor ShowNamespaces to use RunnableCommand
    omit 38a195876bb7 [SPARK-51603][CONNECT][TESTS] Auto ignore tests that require starting `SparkConnectServer` when the directory `assembly/target/scala-2.13/jars/` not exist
    omit 0fdff1bb24a4 [SPARK-51606][CONNECT] Add SPARK_IDENT_STRING when stopping Spark Connect server
    omit 8c57500b6723 [SPARK-51632][INFRA] Passing Command Line Arguments of lint-js to eslint
    omit 73b21fcd19e2 [SPARK-51629][UI] Add a download link on the ExecutionPage for svg/dot/txt format plans
    omit edb28887cb3a [SPARK-50657][DOCS][FOLLOW-UP] Fix the recommended pyarrow version
    omit 762a744aff92 [SPARK-51062][PYTHON] Fix assertSchemaEqual to compare decimal precision and scale
    omit 15ff27147ff8 [SPARK-51619][PYTHON] Support UDT input / output in Arrow-optimized Python UDF
    omit 52dfa3d5e29f [SPARK-51566][PYTHON] Python UDF traceback improvement
    omit c8f50200bd19 [SPARK-51455][SQL][TESTS] Improve test coverage of the TIME data type
    omit cbcc298970fe [SPARK-51628][INFRA] Clean up the `assembly` module before maven testing in maven daily test
    omit 10ecd55f7111 [SPARK-51635][SQL] Merge PushProjectionThroughLimit and PushProjectionThroughOffset
    omit 9fe78e3d3349 [SPARK-51634][SQL] Support TIME in off-heap column vectors
    omit 937ff0919117 [SPARK-51631][PYTHON][INFRA] Refresh docker images for minimum versions
    omit 0528ac76144a [SPARK-51623][BUILD] Remove class files in source releases
    omit ef762a09bb42 [SPARK-51627][INFRA][PYTHON] Add a scheduled workflow for numpy 2.1.3
    omit 714884cb115f [SPARK-51625][SQL] Command in CTE relations should trigger inline
    omit 0f36edc2b985 [SPARK-51624][SQL] Propagate GetStructField metadata in CreateNamedStruct.dataType
    omit ab4bf8c879a4 [SPARK-51622][UI] Titling sections on ExecutionPage
    omit 09e726d3658f [SPARK-51605][CONNECT] Try to create the parent directory before touching the logFile
    omit 64785022e701 [MINOR][PYTHON][CONNECT][TESTS] Add `@timeout` to fail suspicious tests
    omit 4765c15a442e [SPARK-51573][SS] Fix Streaming State Checkpoint v2 checkpointInfo race condition
    omit 15eaa521629f [SPARK-51618][INFRA] Add a check to reject jars in CI
    omit 6ed724e9b4af [SPARK-51588][SQL][TESTS] Validate default values handling in micro-batch writes
    omit 0d2a06be7747 [SPARK-51617][BUILD] Explicitly commit/revert jar removals
    omit 46bd9ccecefd [SPARK-51575][PYTHON] Combine Python Data Source pushdown & plan read workers
    omit b829aeab4f74 [SPARK-51318][BUILD] Remove test jars in source releases
    omit 07c1dedae5cb [SPARK-51612][SQL] Display Spark confs set at view creation in Desc As Json
    omit 400b4d00ac17 [SPARK-51525][DOCS][SQL] Desc As JSON test for top-level default collation field
    omit 597cafc05d9a [SPARK-51610][SQL] Support the TIME data type in the parquet datasource
    omit b5f9a2880bb6 [SPARK-51600][CORE] Prepend classes of `sql/hive` and `sql/hive-thriftserver`  when `isTesting || isTestingSql` is true
    omit 57cf2360bcb2 [SPARK-51607][CONNECT][BUILD] Set `combine.self = "override"` to configuration of `maven-shade-plugin` in `connect` modules
    omit 49295b390d8c [SPARK-51574][PYTHON] Filter serialization for Python Data Source filter pushdown
    omit 4c5baa2abc68 [SPARK-51604][PYTHON][TESTS] Split `test_connect_session`
    omit 100d4a0ce5e3 [SPARK-51350][SQL] Implement Show Procedures
    omit 8a50b0f92db7 [SPARK-51419][SQL] Get hours of TIME datatype
    omit 2c492b22616c [SPARK-51372][SQL] Introduce TableInfo for table creations
    omit ced7f6b8ca7e [SPARK-51552][SQL] Disallow temporary variables in persisted views when under identifier
    omit a3bf597e9fa8 [SPARK-51118][TESTS][FOLLOWUP] Move test_udf_with_udt to BaseUDFTestsMixin
    omit 5b91bbf9289b [SPARK-51587][PYTHON][SS] Fix an issue where timestamp cannot be used in ListState when multiple state data is involved
    omit 94535c91f79c [MINOR][SQL] Desc As JSON test for top-level default collation field
    omit 5e2d6f771001 [SPARK-51404][SQL] Parse the `time(n)` type as `TimeType(n)`
    omit 9b51820d160f [SPARK-51594][SQL] Use empty schema when saving a view which is not Hive compatible
    omit 5a1fd0482e0e [SPARK-51593][SQL] Refactor `QueryExecutionMetering` instantiation
    omit 0c99d96efb8d [SPARK-51598][SQL][TEST] Improve the ExpressionsSchemaSuite
    omit a9c64862d879 [SPARK-51505][SQL] Log empty partition number metrics in AQE coalesce
    omit 5ab4b124bb03 [SPARK-51599][PS][CONNECT] Optimize `ps.read_excel` for large excel file
    omit 90050f51c1a7 [SPARK-51580][SQL] Throw proper user facing error message when lambda function is out of place in `HigherOrderFunction`
    omit b35fd35a28b6 [MINOR][DOCS] corrected spellings and typos
    omit 3b2dd85ef57b [SPARK-51252][SS] Add instance metrics for last uploaded snapshot version in HDFS State Stores
    omit 3155c4b8e0c0 [SPARK-51471][SS] RatePerMicroBatchStream - classify the ASSERT error when offset/timestamp in startOffset is larger than the endOffset
    omit 11d1409fc828 [SPARK-51586][SS] initialize input partitions independent of columnar support in continuous mode
    omit b2290444e9c1 [SPARK-51420][SQL][TESTS][FOLLOWUP] Re-generate `sql-expression-schema.md`
    omit 1ede242a5167 [SPARK-51589][SQL] Fix small bug failing to check for aggregate functions in |> SELECT
    omit dc7683445f2c [SPARK-51584][SQL] Add rule that pushes Project through Offset and Suite that tests it
    omit 084143b977a9 [SPARK-51576][SQL] Prohibit ANSI cast from `array<variant, containsNull = false>` to `array<_, containsNull = false>`
    omit 03b62c27862a [SPARK-51146][INFRA][FOLLOW-UP] Explicitly disable Spark Connect in server side scripts
    omit d363bf870669 Revert "[SPARK-51229][BUILD][CONNECT] Fix dependency:analyze goal on connect common"
    omit e4b6e9aa1ada [SPARK-42746][SQL][FOLLOWUP] Fixing potential flakiness in ListAgg golden files
    omit b8e0246f3ef4 [SPARK-48466][SQL][UI][FOLLOWUP] Handle nested empty relation in SparkPlanInfo
    omit fe44069e3a10 [SPARK-51551][ML][PYTHON][CONNECT] For tuning algorithm, allow using save / load to replace cache
    omit 977eefd41fa5 [MINOR][SQL][DOCS] Improve the SQL example for join strategy
    omit 92511b26c68b [SPARK-51591][PYTHON][CONNECT] Fix ThreadPoolExecutor failure in python 3.13 daily test
    omit c07e702b96ac [SPARK-51581][CORE][SQL] Use `nonEmpty`/`isEmpty` for empty check for explicit `Iterable`
    omit 39ad69aea4fd [SPARK-49082][SQL] Support widening Date to TimestampNTZ in Avro reader
    omit 8e9eada4b4ce [SPARK-51540][PS][DOCS] Best practice for distributed-sequence misalignment case
    omit ae1997919417 [SPARK-40353][PS][CONNECT] Fix index nullable mismatch in `ps.read_excel`
    omit 4e30f967e772 [SPARK-51118][PYTHON] Fix ExtractPythonUDFs to check the chained UDF input types for fallback
    omit 1c3eb7e9e0c7 [SPARK-51578][PYTHON][TESTS] Add a helper function to fail time outed tests
    omit 1ce53805e58a [SPARK-51590][SQL] Disable TIME in builtin file-based datasources
    omit bf5b6db06745 [MINOR][SQL][TESTS] Check file based V2 datasources on unsupported types
    omit f137f6ab5eb1 [SPARK-51420][SQL] Get minutes of TIME datatype
    omit 5f2d37e0b429 [SPARK-42746][FOLLOW-UP][SQL] Fix flaky listagg tests and refactor code
    omit 8469afd8f585 [SPARK-51569][SQL] Don't reinstantiate `InSubquery` in `InTypeCoercion` if there are no type changes
    omit 9dc5e94deb14 [SQL][MINOR][TEST] InMemoryTableCatalog should list procedures namespaces
    omit 59b0fca69d31 [SPARK-50953][FOLLOW-UP] Improve path parsing in variant_get
    omit 2822526d6651 [SPARK-51556][SQL] Add the `try_to_time` function
    omit 92877600b548 [SPARK-51558][BUILD][INFRA] Automatically set env `OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES` when testing on macOS
    omit bd581ec97987 [SPARK-51549][BUILD] Bump Parquet 1.15.1
    omit bfe63a3d3e53 [SPARK-51496][SQL] CaseInsensitiveStringMap comparison should ignore case
    omit 7dfb8fe3f158 Revert #48786 - "Revert "[SPARK-48273][SQL] Fix late rewrite of PlanWithUnresolvedIdentifier""
    omit aaa94c4e40df [SPARK-51544][SQL] Add only unique and necessary metadata columns
    omit f48751527251 [SPARK-51567][ML][CONNECT] Fix `DistributedLDAModel.vocabSize`
    omit 62c06692e47e [SPARK-47241][SQL][FOLLOWUP] Fix issue when laterally referencing a `Generator`
    omit 3ffb8009a3d6 [SPARK-51433][PYTHON][FOLLOWUP] Fix pyspark-client package file name
    omit c8c12e7c4303 [SPARK-51527][SQL] Make codegen log level configurable via SQLConf
    omit 8d260084b8a5 [SPARK-50416][CORE] A more portable terminal / pipe test needed for bin/load-spark-env.sh
    omit 674fd7a6f4d7 [SPARK-51565][SQL] Support SQL parameters in window frame clause
    omit b08cb2227bdb [SPARK-51271][PYTHON] Add filter pushdown API to Python Data Sources
    omit 9d90507d7865 [SPARK-51282][SPARK-51422][ML][FOLLOW-UP] Replace UDF with builtin functions
    omit 133b1b6ff9bd [SPARK-51473][ML][CONNECT] ML transformed dataframe keep a reference to the model
    omit cfe1b399b2fb [SPARK-51559][SQL] Make max broadcast table size configurable
    omit 94f3f3838929 [SPARK-50892][SQL] Add UnionLoopExec, physical operator for recursion, to perform execution of recursive queries
    omit f4493fd00ff7 [MINOR][SQL][TESTS] Ignore messages below error level from `HadoopFSUtils` for sql module tests
    omit 35e0e08acab1 [SPARK-51550][PYTHON][INFRA] Upgrade pyarrow to 19 for PySpark on macOS
    omit 25358ebb68f0 [SPARK-51340][ML][CONNECT] Model size estimation
    omit 37d191e4d19c [MINIOR][PYTHON] Specifying udf type in SCHEMA_MISMATCH_FOR_PANDAS_UDF error message
    omit ee3971fd071d [SPARK-51542][UI] Add a scroll-button for addressing top and bottom
    omit 2eafed55e017 [SPARK-51547][SQL] Assign name to the error condition: _LEGACY_ERROR_TEMP_2130
    omit fecfab4e2354 [SPARK-51543][BUILD] Upgrade jersey to 3.0.17
    omit a5dde0170c6c [SPARK-51546][TESTS] Fix npm vulnerabilities by `npm audit fix`
    omit 0670e4f7945a [SPARK-51541][SQL] Support the `TIME` data type in `Literal` methods
    omit 0de6f2995c78 [SPARK-51451][SQL] Fix ExtractGenerator to wait for UnresolvedStarWithColumns to be resolved
    omit fa9dab054222 [SPARK-51349][SQL][TESTS] Change precedence of null and "null" in sorting in QueryTest
    omit 76a423955a8b [SPARK-51456][SQL] Add the `to_time` function
    omit 513a080ba092 [SPARK-51525][SQL] Collation field for Desc As JSON StringType
    omit ecc659560597 [SPARK-51535][SQL] Don't compare hash codes directly in ProtobufCatalystDataConversionSuite
    omit 8550b44ccb14 [SPARK-51532][INFRA] Restore the installation of Python 3.11 in macOS daily test
    omit 646b84b948ce [SPARK-51522][SQL] Disable lazy union children test for single-pass Analyzer
    omit 4ef0dc0a571e [SPARK-51528][SQL] Don't compare hash codes directly in ProtobufCatalystDataConversionSuite
    omit a47e2cc955a4 [SPARK-51428][SQL] Reassign Aliases for collated expression trees deterministically
    omit 00296d241b76 [SPARK-51112][PYTHON][CONNECT][FOLLOW-UP] Explicitly set index=range(0) for empty table
    omit 6bb0f9a3c0b9 [MINOR][SQL] Re-generate `sql-expression-schema.md`
    omit e546c54d7da0 [SPARK-51280][CONNECT] Improve RESPONSE_ALREADY_RECEIVED error class
    omit 8135abdb9028 [MINOR][SQL] Improve the code for OracleDialect
    omit 9353e94e50f3 [SPARK-51517][SQL] Support the TIME data type in Hive results
    omit d7a98e9f0cc2 [SPARK-51487][PYTHON][INFRA] Refresh testing images for pyarrow 19
    omit f7495da2da8e [SPARK-51501][SQL] Disable ObjectHashAggregate for group by on collated columns
    omit 84b9848f9a69 [SPARK-51097][SS] Re-introduce RocksDB state store's last uploaded snapshot version instance metrics
    omit 2b12416eb142 [SPARK-50759][SQL] Deprecate a few legacy Catalog APIs
    omit f3a6f137a3e0 [MINOR][DOCS] Add Profile -Pdocker-integration-tests to sbt docker IT tests
    omit f253b374b073 [SPARK-51511][INFRA] Upgrade `apache-rat` in the `dev/check-license to` 0.16.1
    omit 63f629686494 [SPARK-51502][SQL][TESTS] Move collations test to collations package
    omit c66d2d946367 [SPARK-51509][CORE][UI] Make `Spark Master` Environment page support filters
    omit 7b1346b401d6 [SPARK-51397][SS] Fix maintenance pool shutdown handling issue causing long test times
    omit a9119b7c4911 [SPARK-51380][SQL] Add visitSQLFunction and visitAggregateFunction to improve the flexibility of V2ExpressionSQLBuilder
    omit 1c7b7d8de1c0 [SPARK-50763][SQL] Add Analyzer rule for resolving SQL table functions
    omit a8e92e4b1b06 [SPARK-51506][PYTHON][SS] Do not enforce users to implement close() in TransformWithStateInPandas
    omit 894814ea31d2 [SPARK-51497][SQL] Add the default time formatter
    omit 37b2a5a31644 [SPARK-51491][PYTHON] Simplify boxplot with subquery APIs
    omit f3b081066393 [SPARK-51438][SQL] Make CatalystDataToProtobuf and ProtobufDataToCatalyst properly comparable and hashable
    omit 71638c4f50d8 [SPARK-51443][SS] Fix singleVariantColumn in DSv2 and readStream
    omit 78941b725d9a [SPARK-51402][SQL][TESTS] Test TimeType in UDF
    omit 1be108eedb83 [SPARK-48922][SQL] Avoid redundant array transform of identical expression for map type
    omit 3a3fce4724e4 [SPARK-51494][BUILD] Upgrade to Apache parent pom 33
    omit 997e599738eb [SPARK-43221][CORE] Host local block fetching should use a block status of a block stored on disk
    omit 4b9b246aa3bf [SPARK-51466][SQL][HIVE] Eliminate Hive built-in UDFs initialization on Hive UDF evaluation
    omit d965217a69b5 [SPARK-51488][SQL] Support the TIME keyword as a data type
    omit ee37d3a74013 [SPARK-51484][SS] Remove unused function `private def newDFSFileName(String)` from `RocksDBFileManager`
    omit baf4b59dd7b9 [SPARK-51482][SQL] Support cast from string to time
    omit d87851f8a941 [MINOR][SQL] Reuse `dateTimeUtilsCls` in `Cast`
    omit 7d7941847def [MINOR][PYTHON] Generate with a newline at the end of error-conditions.json
    omit c2d27540e4ba [SPARK-51476][PYTHON][DOCS] Update document type conversion for Pandas UDFs (pyarrow 17.0.0, pandas 2.2.3, Python 3.11)
    omit c53959cd221f [MINOR][PYTHON] Reformat error classes
    omit c12997230c32 [SPARK-51474][SQL] Don't insert redundant ColumnarToRowExec for node supporting both columnar and row output
    omit 84f5fd99fef1 [SPARK-51467][UI] Make tables of the environment page filterable
    omit 9fc84c70091a [MINOR][BUILD] Fix merge_spark_pr script for no jira case
    omit 9ad4a6e112b1 [MINOR][SQL][TESTS] Remove duplicated plan node check in DataFrameSetOperationsSuite
    omit 0094f445b239 [SPARK-51468][SQL] Revert "From json/xml should not change collations in the given schema"
    omit a7b864fcdc7e [SPARK-51440][SS] classify the NPE when null topic field value is in kafka message data and there is no topic option
    omit 76a1413afb39 [SPARK-51462][SQL] Support typed literals of the TIME data type
    omit 0deb4cb09ff7 [SPARK-51409][SS] Add error classification in the changelog writer creation path
    omit 3a872b7ca11f [SPARK-51453][SQL] AssertTrue uses toPrettySQL instead of simpleString
    omit 5df372126d24 [SPARK-51446][SQL] Improve the codecNameMap for the compression codec
    omit 7ce5d3294325 [SPARK-51457][R][INFRA] Use R 4.4.3 in `windows` R GitHub Action job
    omit 9edb49c360b6 [SPARK-51452][UI] Improve Thread dump table search
    omit 890ee47a9a74 [SPARK-51454][SQL] Support cast from time to string
    omit ef4cd39a0313 [SPARK-51269][SQL] Simplify AvroCompressionCodec by removing defaultCompressionLevel
    omit f27464f269e4 [SPARK-51445][SQL][SS][CONNECT] Change the never changed `var` to `val`
    omit 2893999505a5 [SPARK-51447][SQL] Add `stringToTime` and `stringToTimeAnsi`
    omit f11bc758bafc [SPARK-51418][SQL] Fix DataSource PARTITON TABLE w/ Hive type incompatible partition columns
    omit 34c29cf9fb95 [SPARK-51365][SQL][TESTS] Add Envs to control the number of `SHUFFLE_EXCHANGE/RESULT_QUERY_STAGE` threads used in test cases related to `SharedSparkSession/TestHive`
    omit 0e3b7af4d260 [SPARK-51444][CORE] Remove the unreachable `if` branch from `TaskSchedulerImpl#statusUpdate`
    omit aab1934bfcf6 [SPARK-51442][SQL] Add time formatters
    omit f40bf4dd7166 [SPARK-51386][CORE][SQL] Assign name to error conditions _LEGACY_ERROR_TEMP_3300-3302
    omit 50e00b7b72a8 [SPARK-51425][CONNECT] Add client API to set custom `operation_id`
    omit a5b4d81f4d01 [MINOR][CORE] Remove redundant synchronized in ThreadUtils
    omit 6bf66c75e2b0 [SPARK-51288][DOCS] Add link for Scala API of Spark Connect
    omit 50e8d68d9214 [SPARK-51364][SQL][TESTS] Improve the integration tests for external data source by check filter pushed down
    omit 93675c6715b3 [MINOR][SQL] Slightly refactor and optimize illegaility check in Recursive CTE Subqueries
    omit 27c6b5bf82f1 [SPARK-51422][ML][PYTHON] Eliminate the JVM-Python data exchange in CrossValidator
    omit 940a62c3f36a [SPARK-49479][CORE] Cancel the Timer non-daemon thread on stopping the BarrierCoordinator
    omit a4acedcd9416 [SPARK-51432][SQL][PYTHON] Throw a proper exception when Arrow schemas are mismatched
    omit 2f331e0d2a4d [SPARK-47849][PYTHON][CONNECT] Change release script to release pyspark-client
    omit 1c14b06f8224 Revert "[SPARK-51396][SQL] RuntimeConfig.getOption shouldn't use exceptions for control flow"
    omit c0ec84e1b5b8 [SPARK-50511][PYTHON][FOLLOWUP] Avoid wrapping streaming Python data source error messages
    omit 314932259209 [SPARK-51341][CORE] Cancel time task with suitable way
    omit 772ac22f6ea1 [SPARK-51417][CONNECT] Give a second to wait for Spark Connect server to fully start
    omit 33d3e5bd4b76 [MINOR][DOCS] Fix small grammatical nit
    omit e4cc116791dc [SPARK-51416][CONNECT] Remove SPARK_CONNECT_MODE when starting Spark Connect server
    omit 24c51a232d97 [SPARK-51424][BUILD] Upgrade ORC to 2.1.1
    omit e821b774d659 [SPARK-51391][SQL][CONNECT] Fix `SparkConnectClient` to respect `SPARK_USER` and `user.name`
    omit 44e751f243a5 [SPARK-51321][SQL] Add rpad and lpad support for PostgresDialect and MsSQLServerDialect expression pushdown
    omit 8d2bf0bb0c53 [SPARK-51379][ML] Move treeAggregate's final aggregation from driver to executor
    omit 69d8ece27b4f [SPARK-51408][YARN][TESTS] AmIpFilterSuite#testProxyUpdate fails in some networks
    omit db06293dd100 [SPARK-51396][SQL] RuntimeConfig.getOption shouldn't use exceptions for control flow
    omit 7b39d2409dc6 [SPARK-49488][SQL][FOLLOWUP] Use correct MySQL datetime functions when pushing down EXTRACT
    omit 1054f0d97d8a [SPARK-51384][SQL] Support `java.time.LocalTime` as the external type of `TimeType`
    omit 792eef666588 [SPARK-51097][SS] Revert RocksDB instance metrics changes
    omit e17df6e452a2 [SPARK-51290][SQL] Enable filling default values in DSv2 writes
    omit 74293cc11200 [SPARK-51393][PYTHON] Fallback to regular Python UDF when Arrow is not found but Arrow-optimized Python UDFs enabled
    omit 46b285e81d30 [SPARK-51407][DOCS][FOLLOWUP] Fix `spark.connect.ml.backend.classes` description
    omit 22f1905c87fd [SPARK-51407][CONNECT][DOCS] Document missed `Spark Connect` configurations
    omit a30bdc3b4c53 [SPARK-51389][BUILD] Allow full customization of profiles in scalastyle
    omit cdba396d6e9b [SPARK-51401][SQL] Change `ExplainUtils.generateFieldString` to directly call `QueryPlan.generateFieldString`
    omit 60521214de1a [SPARK-49507][SQL] Fix the case issue after enabling metastorePartitionPruningFastFallback
    omit 6ba4771842a7 [SPARK-51371][SQL] Change toString to toPrettySQL when building Aliases in ResolveAggregateFunctions
    omit 4b4bdcfe1fe9 [SPARK-51388][SQL] Improve SQL fragment propagation in to_timestamp and UNION
    omit 26545d000bc8 [SPARK-51394][ML] Optimize out the additional shuffle in stats tests
    omit c005a37ec9a4 [SPARK-51385][SQL] Normalize out projection added in DeduplicateRelations for union child output deduplication
    omit a6600afbcba1 [SPARK-51357][SQL] Preserve plan change logging level for views
    omit 7fca7aeaf9a4 [SPARK-51277][PYTHON][FOLLOW-UP] Remove 0-arg check in Spark Connect side as well
    omit 12f29675735c [MINOR] Remove unused imports
    omit 4be3aaee7560 [SPARK-51390][INFRA] Add more dependencies in LICENSE-binary for Spark 4.0.0 release
    omit eb4855315049 Revert "[SPARK-48516][PYTHON][CONNECT] Turn on Arrow optimization for Python UDFs by default"
    omit 1098697a7980 Revert "[SPARK-48516][PYTHON][FOLLOW-UP] Add a note in migration guide about Arrow-optimized Python UDF enabled by default"
    omit 449d3cbb8005 Revert "[SPARK-51276][PYTHON] Enable spark.sql.execution.arrow.pyspark.enabled by default"
    omit 31cf3c375257 [SPARK-51383][PYTHON][CONNECT] Avoid making RPC calls if clients are already known as stopped
    omit f71291b17a92 [SPARK-51314][DOCS][PS] Add proper note for distributed-sequence about indeterministic case
    omit 9cfc628d8df3 [SPARK-50855][CONNECT][TESTS][FOLLOWUP] Refactor `TransformWithStateConnectSuite` to run `DROP TABLE IF EXISTS my_sink` in `beforeAll/afterEach`
    omit eb7144396286 [SPARK-51307][SQL] locationUri in CatalogStorageFormat shall be decoded for display
    omit 229be37478bd [SPARK-51072][CORE] CallerContext to set Hadoop cloud audit context
    omit fd1f7748397a [MINOR][CORE] Remove unused `private[executor] var httpUrlConnectionTimeoutMillis` from `ExecutorClassLoader`
    omit 8f0ac3b72f6e [SPARK-51367][BUILD] Upgrade slf4j to 2.0.17
    omit 3d949af2e59a [SPARK-51387][BUILD] Upgrade Netty to 4.1.119.Final
    omit 921ef95609cd [SPARK-51029][INFRA][FOLLOWUP] Remove LICENSE and NOTICE for unused/duplicated hive deps
    omit 8f2c066cb1b7 [SPARK-50615][FOLLOWUP][SQL] Avoid dropping metadata in the push rule
    omit 14fff2a7a953 [SPARK-51342][SQL] Add `TimeType`
    omit c2f2be68dd09 [SPARK-51373][SS] Removing extra copy for column family prefix from 'ReplyChangelog'
    omit 57564803b5f5 [SPARK-51368][SS][PYTHON][TEST] Disable Schema Evolution Suite in Connect TransformWithState for Python Suite
    omit 62f7f14edc53 [SPARK-51378][CORE] Apply JsonProtocol's accumulableExcludeList to ExecutorMetricsUpdate and TaskEndReason
    omit 0ce096dc8feb [SPARK-45641][DOCS][FOLLOWUP] Add `Started At` description for web-ui.md
    omit 3820ae0f86b9 [SPARK-50855][SS][CONNECT] Spark Connect Support for TransformWithState In Scala
    omit 4f97dcbebc66 [SPARK-51381][SQL][CONNECT] Show `Session ID` in `Spark Connect Session` page
    omit d5a17546f9e0 [SPARK-51375][SQL][CONNECT] Suppress `SparkConnect(Execution|Session)Manager.periodicMaintenance` log messages
    omit cf1c3bf54580 [SPARK-51362][SQL] Change toJSON to use NextIterator API to eliminate adjacent record dependency
    omit 86c2140b62e4 [SPARK-51363][SQL] Desc As JSON` clustering column names
    omit 4b2ae4864104 [SPARK-50578][PYTHON][SS][FOLLOW-UP] Add the Python new package into setup.py
    omit 8299a1458b27 [SPARK-51355][DOCS] Add a brief explanation for Spark Connect at PySpark Overview page
    omit 61859161778b [SPARK-51374][CORE] Switch to Using java.util.Map in Logging APIs
    omit 7653b4003330 [SPARK-51229][BUILD][CONNECT] Fix dependency:analyze goal on connect common
    omit 1ad7f31baf98 [SPARK-51370] Code clarity improvements for INSERT INTO parsing codes in AstBuilder.scala
    omit b4896c0f3e83 [SPARK-51369][SQL] Fix FOR cursor variable to work with mixed case names
    omit ec44e29b4d55 Revert "[SPARK-51353][INFRA][BUILD] Retry dyn/closer.lua for mvn before falling back to archive.a.o"
    omit 2ea5621a6053 [SPARK-49489][SQL][HIVE] HMS client respects `hive.thrift.client.maxmessage.size`
    omit e66e20ac3a4d Revert "[SPARK-51309][BUILD] Upgrade rocksdbjni to 9.10.0"
    omit ec03f1e35cc1 [SPARK-51353][INFRA][BUILD] Retry dyn/closer.lua for mvn before falling back to archive.a.o
    omit 4df5abc33e2f [SPARK-51354][K8S][TEST][DOC] Fix sbt K8s integration test arg javaImageTag
    omit d88db28910d0 [MINOR][TESTS][CONNECT] Fix the teardown function of test_connect_function.py
    omit a33f232dc4ef [MINOR][DOCS] Fix code comments for Executor
    omit 50a328ba9857 [SPARK-49960][SQL] Custom ExpressionEncoder support and TransformingEncoder fixes
    omit 496fe7a50919 [SPARK-51351][SS] Do not materialize the output in Python worker for TWS
    omit 5b4567165400 [SPARK-51316][PYTHON][FOLLOW-UP] Revert unrelated changes and mark mapInPandas/mapInArrow batched in byte size
    omit 208a7ee6771d [SPARK-49756][SQL][FOLLOWUP] Use correct pgsql datetime fields when pushing down EXTRACT
    omit ddd0af615cd7 [SPARK-51339][BUILD] Remove `IllegalImportsChecker` for `s.c.Seq/IndexedSeq` from `scalastyle-config.xml`
    omit 3478e6b89f99 [SPARK-51270][SQL] Support UUID type in Variant
    omit 88addf461344 [SPARK-51337][SQL] Add maxRows to CTERelationDef and CTERelationRef
    omit 71422d1903d2 [SPARK-51303][SQL][TESTS] Extend `ORDER BY` testing coverage
    omit b05f18b6424a [SPARK-51278][FOLLOWUP][DOCS] Update JSON format from documentation
    omit 89e05a88deb9 [SPARK-51326][CONNECT] Remove LazyExpression proto message
    omit a3671e55839c [SPARK-51281][SQL] DataFrameWriterV2 should respect the path option
    omit 412da42ab2e9 [SPARK-51310][SQL] Resolve the type of default string producing expressions
    omit 07e6a0610151 [SPARK-50994][CORE] Perform RDD conversion under tracked execution
    omit e61a885e217e [SPARK-51322][SQL] Better error message for streaming subquery expression
    omit b47b0040b32d [SPARK-51333][ML][PYTHON][CONNECT] Unwrap `InvocationTargetException` thrown by `invokeMethod`
    omit 70d532a796e2 [SPARK-51302][CONNECT] Spark Connect supports JDBC should use the DataFrameReader API
    omit 3ba50cdeb159 [SPARK-51323][PYTHON] Duplicate "total" on Py SQL metrics
    omit 53fc76313234 [SPARK-51316][PYTHON] Allow Arrow batches in bytes instead of number of rows
    omit bbd4b9671032 [SPARK-51329][ML][PYTHON] Add `numFeatures` for clustering models
    omit 2fca41c3ca28 [SPARK-50792][SQL][TESTS][FOLLOWUP] Test case should reuse the exists table
    omit 727167acc30c [SPARK-51206][PYTHON][CONNECT] Move Arrow conversion helpers out of Spark Connect
    omit 989844621f55 [SPARK-51324][SQL] Fix nested FOR statement throwing error if empty result
    omit 04af02fd6c6e [SPARK-51273][SQL] Spark Connect Call Procedure runs the procedure twice
    omit 9efaa91deb50 [SPARK-51277][PYTHON] Implement 0-arg implementation in Arrow-optimized Python UDF
    omit ff7b4a423e6d [SPARK-51315][SQL] Enabling object level collations by default
    omit 971b9a40c42a [SPARK-51309][BUILD] Upgrade rocksdbjni to 9.10.0
    omit 87fd092b19be [SPARK-51282][ML][PYTHON][CONNECT] Optimize OneVsRestModel transform by eliminating the JVM-Python data exchange
    omit 7d7a05b7fb1c [SPARK-51265][SQL] IncrementalExecution should set the command execution code correctly
    omit 381d5bc65e4d [SPARK-51312][SQL] Fix createDataFrame from RDD[Row]
    omit 14c13788b587 [SPARK-51221][CONNECT][TESTS] Use unresolvable host name in SparkConnectClientSuite
    omit 85f05e38b703 [SPARK-51313][PYTHON] Fix timestamp format for PySparkLogger
    omit 8e8eccbf5118 [SPARK-51317][PYTHON] Require pandas as well for Arrow-optimized Python UDF
    omit 93d10c25d5b8 [SPARK-50856][SS][PYTHON][CONNECT] Spark Connect Support for TransformWithStateInPandas In Python
    omit 4f5ee806597c [SPARK-51311][BUILD] Promote bcprov-jdk18on to compile scope
    omit 0184c5bf6670 [SPARK-50785][SQL] Refactor FOR statement to utilize local variables properly
    omit 7feb911c814b [SPARK-50795][SQL][FOLLOWUP] Set isParsing to false for the timestamp formatter in DESCRIBE AS JSON
    omit 352d1ed84db2 [SPARK-51306][TESTS] Fix test errors caused by improper DROP TABLE/VIEW in describe.sql
    omit 529e887b794e [SPARK-50914][PYTHON][CONNECT] Match GRPC dependencies for Python-only master scheduled job
    omit d4b4cfc79491 [SPARK-51099][PYTHON][FOLLOWUP] Avoid logging when selector.select returns 0 without waiting the configured timeout
    omit ea15a953e818 [SPARK-50692][SQL] Add the LPAD and RPAD pushdown support for H2
    omit 9f637b5ca0f6 [SPARK-51305][SQL][CONNECT] Improve `SparkConnectPlanExecution.createObservedMetricsResponse`
    omit 9de3b7c677f1 [SPARK-51156][CONNECT][FOLLOWUP] Remove unused `private val AUTH_TOKEN_ON_INSECURE_CONN_ERROR_MSG` from `SparkConnectClient`
    omit 60bcc71af299 [SPARK-51095][CORE][SQL] Include caller context for hdfs audit logs for calls from driver
    omit 48fc0fba41cb [SPARK-49912] Refactor simple CASE statement to evaluate the case variable only once
    omit 7e5bf7262d76 [SPARK-51078][SPARK-50963][ML][PYTHON][CONNECT][TESTS][FOLLOW-UP] Add back tests for default value
    omit 968542f21a06 [SPARK-51304][DOCS][PYTHON] Use `getCondition` instead of `getErrorClass` in contribution guide
    omit ee8e10f077cd [SPARK-50015][PYTHON][FOLLOW-UP] Update _minimum_grpc_version in setup.py for pyspark-client
    omit a084d64dbedb [SPARK-50098][PYTHON][FOLLOW-UP] Update _minimum_googleapis_common_protos_version in setup.py for pyspark-client
    omit 67a337e202af [SPARK-51278][PYTHON] Use appropriate structure of JSON format for `PySparkLogger`
    omit 3515b207c41d [SPARK-51300][PS][DOCS] Fix broken link for `ps.sql`
    omit c6097c7fbc8a [MINOR][DOCS] Clarify spark.remote and spark.master in pyspark-connect and pyspark-client at install.rst
    omit f1d78dc2df26 [SPARK-51297][DOCS] Fixed the scope of the query option in sql-data-sources-jdbc.md
    omit b04e9c491893 [SPARK-51294][SQL][CONNECT][TESTS] Improve the readability by split the variable of jars and configs for SparkConnectServerUtils
    omit 0b5b0d56a431 [SPARK-51293][CORE][SQL][SS][MLLIB][TESTS] Cleanup unused private functions from test suites
    omit 30279686b63a [SPARK-51292][SQL] Remove unnecessary inheritance from PlanTestBase, ExpressionEvalHelper and PlanTest
    omit 9a1f921ae19d [SPARK-51258][SQL][FOLLOWUP] Remove unnecessary inheritance from SQLConfHelper
    omit 7e9547c63293 [SPARK-51156][CONNECT] Static token authentication support in Spark Connect
    omit 30f4f4e56e73 [SPARK-51258][SQL] Remove unnecessary inheritance from SQLConfHelper
    omit 666f45d9e5ef [SPARK-51279][CONNECT] Avoid constant sleep for waiting Spark Connect server in Scala
    omit 9ac566dea896 [SPARK-51276][PYTHON] Enable spark.sql.execution.arrow.pyspark.enabled by default
    omit eb4a28b97398 [SPARK-51274][PYTHON] PySparkLogger should respect the expected keyword arguments
    omit e1842c7f3023 [SPARK-51263][CORE][SQL][TESTS] Clean up unnecessary `invokePrivate` method calls in test code
    omit 140a69b4c213 [SPARK-51283][SQL][TESTS] Add test cases for LZ4 and SNAPPY for text
    omit b70b90461681 [SPARK-51284][SQL] Fix SQL Script execution for empty result
    omit 3d76e0bbc30f [SPARK-51275][PYTHON][ML][CONNECT] Session propagation in python readwrite
    omit 46e12a4bebfa [SPARK-51267][CONNECT] Match local Spark Connect server logic between Python and Scala
    omit 45900c434c9a [SPARK-48516][PYTHON][FOLLOW-UP] Add a note in migration guide about Arrow-optimized Python UDF enabled by default
    omit 4ffc398d4307 [SPARK-51119][SQL][FOLLOW-UP] Add fallback to ResolveDefaultColumnsUtil existenceDefaultValues
    omit f37be893d018 [SPARK-51222][SQL][FOLLOW-UP] Optimize ReplaceCurrentLike
    omit 1d72acc7873a [SPARK-50864][TESTS] Disable slow tests
    omit 42ab97a3e6e7 [SPARK-51249][SS] Fixing the NoPrefixKeyStateEncoder and Avro encoding to use the correct number of version bytes
    omit d6ca11e93535 [SPARK-51092][SS] Skip the v1 FlatMapGroupsWithState tests with timeout on big endian platforms
    omit da1854e0cb38 [SPARK-51097][SS] Adding state store instance metrics for last uploaded snapshot version in RocksDB
    omit bbb9c2c1878e [SPARK-47208][DOCS][FOLLOWUP] Replace `spark.driver.minMemoryOverhead` with `spark.executor.minMemoryOverhead`
    omit 62f0d29b5fee [SPARK-51266][CORE] Remove the unused definition of `private[spark] object TaskDetailsClassNames`
    omit fb17856a22be [SPARK-48530][SQL] Support for local variables in SQL Scripting
    omit a867a9e47357 [SPARK-51259][SQL][FOLLOWUP] Improve performance of natural and using join keys computation
    omit a661f9f305d9 [SPARK-51259][SQL] Refactor natural and using join keys computation
    omit f447c43af4a1 [SPARK-51219][SQL][TESTS][FOLLOWUP] ShowTablesExec` remove `ArrayImplicits`
    omit 35442bae7a85 [SPARK-51150][ML][FOLLOW-UP] Pass session in `OneVsRestParams.saveImpl`
    omit 6603a4ef75eb [SPARK-51254][PYTHON][CONNECT] Disallow --master with Spark Connect URL
    omit 2beb7ed9b2de [SPARK-51260][SQL] Move V2ExpressionBuilder and PushableExpression to Catalyst module
    omit 948840b6effd [MINOR][PYTHON][DOCS] Refine the docstring of `VariantVal`
    omit cecafac01664 [SPARK-51257][SQL][TESTS] Add order-by-alias.sql
    omit 26febf763c1b [SPARK-51247][SQL] Move SubstituteExecuteImmediate to 'resolution' batch and prepare it for SQL Scripting local variables
    omit 5759882268dc [SPARK-50655][SS] Move virtual col family related mapping into db layer instead of encoder
    omit 53c326b3080c [SPARK-51255][INFRA] Install dependencies related to docs in release image
    omit 2c628997a1cd [SPARK-51083][CORE] Modify JavaUtils to not swallow InterruptedExceptions
    omit 0af25b836549 [SPARK-51239][INFRA] Upgrade Github Action image for `TPCDSQueryBenchmark` from 20.04 to latest
    omit b32d3f779bbe [SPARK-51240][BUILD] Upgrade commons-codec to 1.18.0
    omit a44b6139d914 [SPARK-51202][ML][PYTHON] Pass the session in meta algorithm python writers
    omit aa12070c1321 [MINOR][DOCS] Add missing backticks in `Upgrading from PySpark 3.5 to 4.0`
    omit dc3fb50d587d [SPARK-51176][PYTHON][CONNECT] Meet consistency for unexpected errors PySpark Connect <> Classic
    omit 489ba0d24975 [SPARK-51242][CONENCT][PYTHON] Improve Column performance when DQC is disabled
    omit 4134e9f208bb [SPARK-51178][CONNECT][PYTHON] Raise proper PySpark error instead of `SparkConnectGrpcException`
    omit ead7d58ab0b6 [SPARK-48114][SQL] Move subquery validation out of CheckAnalysis
    omit 500bf78e77ca [SPARK-51246][SQL] Make InTypeCoercion produce resolved Casts
    omit a8b694f4d3dc [SPARK-50767][SQL] Remove codegen of `from_json`
    omit 6dbd12a81c22 [SPARK-51241][SQL][TESTS] Add test cases with ignore nulls for ANY_VALUE
    omit aa37f8981187 [SPARK-51219][SQL] Fix `ShowTablesExec.isTempView` to work with non-`V2SessionCatalog` catalogs
    omit ef0685a0dea4 [SPARK-51152][PYTHON][SQL][DOCS] Add usage examples for the get_json_object function
    omit 2c76dff9536d [SPARK-51234][PYTHON][DOCS] Document an import change in `from pyspark.sql.functions import *`
    omit aeea73867858 [SPARK-51085][SQL] Restore SQLContext Companion
    omit bd2b4782ccf8 [SPARK-50849][CONNECT] Add example project to demonstrate Spark Connect Server Libraries
    omit 479e0b34c4c9 [SPARK-51192][CONNECT] Expose `processWithoutResponseObserverForTesting` in `SparkConnectPlanner`
    omit 17b943106b71 [SPARK-51237][SS] Add API details for new transformWithState helper APIs as needed
    omit 8a75e12f96a1 [SPARK-51228][SQL] Introduce subquery normalization to NormalizePlan
    omit e1f7851bdee8 [SPARK-50692][SQL][FOLLOWUP] Add comments for LPAD and RPAD
    omit 8423d74a76f4 [SPARK-51232][PYTHON][DOCS] Remove PySpark 3.3 and older logic from `binder/postBuild`
    omit 0002cdd6ea1f [SPARK-42746][SQL][FOLLOWUP] Correct the comments for SupportsOrderingWithinGroup and Mode
    omit 0c30983e663d [SPARK-51218][SQL] Avoid map/flatMap in NondeterministicExpressionCollection
    omit 4c37a7a1af6b [SPARK-51238][K8S][INFRA][DOCS] Upgrade Volcano to 1.11.0
    omit b63b90e4c55d [SPARK-51183][SQL] Link to Parquet spec in Variant docs
    omit ec9086284de6 [SPARK-51235][K8S][DOCS] Update `YuniKorn` docs with `1.6.1`
    omit ad3f13ef9735 [SPARK-51231][BUILD] Add `--enable-native-access=ALL-UNNAMED` to `.mvn/jvm.config` and `jvmArgs` to suppress Maven warnings
    omit d75a7d6fbe4a [SPARK-51217][ML][CONNECT] ML model helper constructor clean up
    omit b3dac8814a61 [SPARK-51215][ML][PYTHON][CONNECT] Add a helper function to invoke helper model attr
    omit e5a0ee9ce37d [SPARK-51233][BUILD] Update `pom.xml` to use `https` for `licenses` element
    omit f798e7ab0128 [SPARK-51212][PYTHON] Add a separated PySpark package for Spark Connect by default
    omit 3f4ba72786b3 [SPARK-51227][PYTHON][CONNECT] Fix PySpark Connect `_minimum_grpc_version` to 1.67.0
    omit b8d395e1a16a [SPARK-50098][PYTHON][CONNECT][FOLLOWUP] Fix PySpark Connect `_minimum_googleapis_common_protos_version` to 1.65.0
    omit 0bcff44a1784 [SPARK-51222][SQL] Optimize ReplaceCurrentLike
    omit c36916f7a112 [SPARK-51225][BUILD] Upgrade several Maven plugins to the latest versions
    omit b2cbdaa4fd77 [SPARK-51226][BUILD] Upgrade `extra-enforcer-rules` to support Java 24
    omit f601eb700cd7 [SPARK-51146][SQL][FOLLOWUP] Respect system env `SPARK_CONNECT_MODE` in places that access the api mode config
    omit 09b93bd657aa [SPARK-51214][ML][PYTHON][CONNECT] Don't eagerly remove the cached models for `fit_transform`
    omit d6ad7798a9f7 [SPARK-51216][BUILD] Remove the useless `bigtop-dist` profile and the related outdated files
    omit 47edf4b54e2e [SPARK-51213][SQL] Keep Expression class info when resolving hint parameters
    omit 7992a2fd519c [SPARK-51186][PYTHON] Add `StreamingPythonRunnerInitializationException` to PySpark base exception
    omit e8402557cc7c [SPARK-51209][CORE][FOLLOWUP] Use `user.name` system property first as a fallback
    omit 83ee2a072227 [SPARK-51119][SQL][FOLLOW-UP] ColumnDefinition.toV1Column should preserve EXISTS_DEFAULT resolution
    omit 55c65d982460 [SPARK-51204][BUILD] Upgrade `sbt-assembly` to 2.3.1
    omit c76ed499eaf1 [SPARK-51201][SQL] Make Partitioning Hints support byte and short values
    omit 2e0c02e158f9 [SPARK-51177][PYTHON][CONNECT] Add `InvalidCommandInput` to Spark Connect Python client
    omit 78b175462836 [SPARK-51210][CORE] Add `--enable-native-access=ALL-UNNAMED` to Java options for Java 24+
    omit ffd58bca521a [SPARK-51209][CORE] Improve `getCurrentUserName` to handle Java 24+
    omit e3972078b40e [SPARK-51208][SQL] `ColumnDefinition.toV1Column` should preserve `EXISTS_DEFAULT` resolution
    omit 49f9ecf31046 [SPARK-51205][BUILD][TESTS] Upgrade `bytebuddy` to 1.17.0 to support Java 25
    omit 0d6bdce5b2fa [SPARK-51113][SQL] Fix correctness with UNION/EXCEPT/INTERSECT inside a view or EXECUTE IMMEDIATE
    omit e92e12a0d188 [SPARK-51067][SQL] Revert session level collation for DML queries and apply object level collation for DDL queries
    omit 18ff7b3274ff [SPARK-51181][SQL] Enforce determinism when pulling out non deterministic expressions from logical plan
    omit 80fb9d04dc95 [SPARK-51200][BUILD] Add SparkR deprecation info to `README.md` and `make-distribution.sh` help
    omit 920486dcc099 [SPARK-51198][CORE][DOCS] Revise `defaultMinPartitions` function description
    omit 3d4f6b40eb8d [SPARK-51194][BUILD] Upgrade `scalafmt` to 3.8.6
    omit ae2d2e8411f9 [SPARK-51195][BUILD][K8S] Upgrade `kubernetes-client` to 7.1.0
    omit 0a102327d527 [SPARK-51193][CORE] Upgrade Netty to 4.1.118.Final and netty-tcnative to 2.0.70.Final
    omit 8e39b7f662cb [SPARK-51197][ML][PYTHON][CONNECT][TESTS] Unit test clean up
    omit 74d88b61646f [SPARK-51185][CORE] Revert simplifications to PartitionedFileUtil API to reduce memory requirements
    omit 97372e0381fc [SPARK-51189][CORE] Promote `JobFailed` to `DeveloperApi`
    omit b3e64ad4b155 [SPARK-51059][ML][CONNECT][DOCS] Document how ALLOWED_ATTRIBUTES works
    omit 3bc516e210ad [SPARK-51190][ML][PYTHON][CONNECT] Fix TreeEnsembleModel.treeWeights
    omit d669575524ab [SPARK-51188][BUILD] Upgrade Arrow to 18.2.0
    omit f7497425cd6d [SPARK-50960][PYTHON][CONNECT] Add `InvalidPlanInput` to Spark Connect Python client
    omit dc5682955a2e [MINOR][SQL][TESTS] Remove redundant space at `PropagateEmptyRelationSuite` class definition
    omit 340bc002b937 [SPARK-51184][CORE] Remove `TaskState.LOST` logic from `TaskSchedulerImpl`
    omit 6440bdec8af6 Revert "[SPARK-51113][SQL] Fix correctness with UNION/EXCEPT/INTERSECT inside a view or EXECUTE IMMEDIATE"
    omit d0e07ac92473 [SPARK-51113][SQL] Fix correctness with UNION/EXCEPT/INTERSECT inside a view or EXECUTE IMMEDIATE
    omit 03868a8e0cc9 [SPARK-51163][BUILD] Exclude duplicated jars from connect-repl
    omit 7aad4d06f53b [SPARK-51160][SQL] Refactor literal function resolution
    omit 207390b29beb [SPARK-51008][SQL] Add ResultStage for AQE
    omit cb2732dfdbec [SPARK-51146][INFRA][FOLLOWUP] Use awk to update release scripts
    omit 42ecabf42077 [SPARK-51175][CORE] Make `Master` show elapsed time when removing drivers
    omit 8bacf995f297 [SPARK-51141][ML][CONNECT][DOCS] Document the Support of ML on Connect
    omit 749eae404a50 [SPARK-51171][BUILD] Upgrade `checkstyle` to 10.21.2
    omit d6fe024ac469 [MINOR][INFRA] List pip installation before test in python macos test and python connect test
    omit 00f1057a2c40 [SPARK-50582][SQL][PYTHON] Add quote builtin function
    omit 16784fac9dfc [SPARK-50854][SS] Make path fully qualified before passing it to FileStreamSink
    omit 2cd53651908e [SPARK-42746][SQL][FOLLOWUP] Improve the golden files by print the hex string of binary
    omit 274dc5e05846 [SPARK-51173][TESTS] Add `configName` Scalastyle rule
    omit decb677b8e1e [SPARK-51172][SS] Rename to spark.sql.optimizer.pruneFiltersCanPruneStreamingSubplan
    omit f52a6790f787 [SPARK-51114][SQL] Refactor PullOutNondeterministic rule
    omit 00b2bc49fcf7 [SPARK-51136][CORE] Set `CallerContext` for History Server
    omit 9793587af6c6 [SPARK-51170][PYTHON][CONNECT][TESTS] Separate local and local-cluster in pure Python build in master
    omit e2c81ade5c89 [SPARK-51164][CORE][TESTS][FOLLOWUP] Add hadoop.caller.context.enabled for scalatest-maven-plugin
    omit 6f52d5a594f8 [SPARK-51127][PYTHON][FOLLOWUP] Update version for the configs
    omit 6a4e0438582a [SPARK-51165][CORE] Enable `spark.master.rest.enabled` by default
    omit b21428f6e14a [SPARK-51159][INFRA] Pin `plotly<6.0.0` and `torch<2.6.0` in release docker image
    omit d07b56002346 [SPARK-51164][CORE][TESTS] Fix `CallerContext` test by enabling `hadoop.caller.context.enabled`
    omit c9720afd4f4e [SPARK-50557][CONNECT][SQL] Support RuntimeConfig.contains(..) in Scala SQL Interface
    omit c66cc0ea7dfa [SPARK-50940][SPARK-50941][ML][PYTHON][CONNECT][FOLLOW-UP] Directly reuse the CrossValidatorModelWriter and TrainValidationSplitModelWriter
    omit ad8222a89d4b [SPARK-51158][YARN][TESTS] No longer restrict the test cases related to `connect` in the `YarnClusterSuite` to only run on GitHub Actions
    omit ba3e2713c8ff [SPARK-45891][SQL][FOLLOWUP] Disable `spark.sql.variant.allowReadingShredded` by default
    omit 533d9c397f9e [SPARK-51146][INFRA] Publish a new Spark distribution with Spark Connect enabled (extra tarball)
    omit 1ba759feb784 [SPARK-51157][SQL] Add missing @varargs Scala annotation for Scala function APIs
    omit 513532966302 [SPARK-51119][SQL][FOLLOW-UP] Readers on executors resolving EXISTS_DEFAULT should not call catalogs
    omit e7709f0f60e5 [SPARK-51155][CORE] Make `SparkContext` show total runtime after stopping
    omit d3be2a702084 [SPARK-51154][BUILD][TESTS] Remove unused `jopt` test dependency
    omit 641629786909 [SPARK-51153][BUILD] Remove unused `javax.activation:activation` dependency
    omit 394458fc1fa8 [SPARK-51147][SS] Refactor streaming related classes to a dedicated streaming directory
    omit 6a668cde87c3 [SPARK-51150][ML] Explicitly pass the session in meta algorithm writers
    omit e7ceb5b7e119 [MINOR][DOCS] Fix incorrect description of constraint on spark.sql.adaptive.coalescePartitions.minPartitionSize
    omit cea79dc1918b [SPARK-51132][ML][BUILD] Upgrade `JPMML` to 1.7.1
    omit 54959ab0e561 [SPARK-51148][BUILD] Upgrade `zstd-jni` to 1.5.6-10
    omit 6a71f76f8aac [SPARK-51142][ML][CONNECT] ML protobufs clean up
    omit bdb7704b2b19 [SPARK-51146][INFRA] Publish a new Spark distribution with Spark Connect enabled (step 1)
    omit 937deccc5093 [SPARK-51119][SQL] Readers on executors resolving EXISTS_DEFAULT should not call catalogs
    omit 9d88020f246f [SPARK-51112][CONNECT] Avoid using pyarrow's `to_pandas` on an empty table
    omit c91fdd4a145d Revert "[SPARK-51138][PYTHON][CONNECT][TESTS] Skip pyspark.sql.tests.connect.test_parity_frame_plot_plotly.FramePlotPlotlyParityTests.test_area_plot"
    omit 22d2eb33f01d [SPARK-51143][PYTHON] Pin `plotly<6.0.0` and `torch<2.6.0`
    omit 8071868d1fc9 [SPARK-51140][ML] Sort the params before saving
    omit 165e9d0c2a8b [SPARK-51106][CORE][TESTS] Use `try-with-resources` to ensure resources are closed after testing
    omit cd9c50992593 [SPARK-51023][CORE] log remote address on RPC exception
    omit 56c78792a69b [SPARK-51145][BUILD][TESTS] Upgrade `mysql-connector-j`, `MySQL` docker image version to 9.2.0 and `MariaDB` docker image version to 11.4.5
    omit 066abfa665a3 [SPARK-50753][PYTHON][DOCS] Add pyspark plotting to API documentation
    omit 222dd81fb87c [SPARK-51073][SQL] Remove `Unstable` from `SparkSessionExtensionsProvider` trait
    omit e823afad625e [SPARK-50917][EXAMPLES] Add Pi Scala example to work both for Connect and Classic
    omit 59dd406ffab6 [SPARK-48516][PYTHON][CONNECT] Turn on Arrow optimization for Python UDFs by default
    omit dd153307cb97 [SPARK-50953][PYTHON][CONNECT] Add support for non-literal paths in VariantGet
    omit 9f866475fca4 [SPARK-50881][PYTHON] Use cached schema where possible in conenct dataframe.py
    omit 34e6e447510e [SPARK-51139][ML][CONNECT] Refine error class `MLAttributeNotAllowedException`
    omit 2fc278d39851 [SPARK-51133][BUILD] Upgrade Apache `commons-pool2` to 2.12.1
    omit c7edcaed290e [SPARK-51109][SQL] CTE in subquery expression as grouping column
    omit 59a6ca5bdd45 [SPARK-48239][INFRA][FOLLOWUP] Update the release docker image to follow what we use in Github Action jobs
    omit d0848aaaeda2 [SPARK-51135][SQL] Fix ViewResolverSuite for ANSI modes
    omit 736737e6c161 [SPARK-51138][PYTHON][CONNECT][TESTS] Skip pyspark.sql.tests.connect.test_parity_frame_plot_plotly.FramePlotPlotlyParityTests.test_area_plot
    omit 301b666a1fcb [SPARK-51065][SQL] Disallowing non-nullable schema when Avro encoding is used for TransformWithState
    omit ba7849e55115 [SPARK-51130][YARN][TESTS] Run the test cases related to `connect` in the `YarnClusterSuite` on Github Actions only
    omit af924205817f [SPARK-51048][CORE] Support stop java spark context with exit code
    omit a5e905d138b2 [SPARK-51127][PYTHON] Kill the Python worker on idle timeout
    omit 553149732aef [SPARK-50945][ML][PYTHON][CONNECT] Support Summarizer and SummaryBuilder on Connect
    omit 099a59c9f3cf [SPARK-51080][ML][PYTHON][CONNECT] Fix save/load for `PowerIterationClustering`
    omit 3ef58c38980b [SPARK-51103][PYTHON][DOCS] Add DataFrame conversion to table argument to API documentation
    omit df89c8ed40b0 [SPARK-51087][PYTHON][CONNECT] Raise a warning when memory-profiler is not installed for memory profiling
    omit 717027d86b30 [SPARK-51129][DOC] Fix code tab switching in Spark Website
    omit ad13a88dad7e [SPARK-48353][SQL][FOLLOWUP] Enable ANSI for SQL Scripting execution suites
    omit 4305bb1a2759 [SPARK-51131][SQL] Throw exception when SQL Script is found inside EXECUTE IMMEDIATE command
    omit bafd007a0b41 [SPARK-50075][FOLLOWUP][DOCS] Add table-valued function API docs
    omit f5f7c365d519 [SPARK-51093][SQL][TESTS] Fix minor endianness issues in tests
    omit a01ae99d5e90 [SPARK-49531][PYTHON][CONNECT][INFRA][FOLLOW-UP] Match pandas version of Python Connect build with development build
    omit 20bd4aae6e8b [SPARK-51128][DOC] Self host docsearch.min.css.map in Spark website
    omit 616baa874e5d [SPARK-51108][INFRA] Install Python packages for `yarn` module in `maven_test.yml`
    omit 7a1dcb3b40c4 [SPARK-51126][PS][DOCS] Optimize the memory usage in kde examples
    omit 935c2b0022ea [SPARK-51101][ML][PYTHON][CONNECT][TESTS] Add doctest for `pyspark.ml.connect.functions`
    omit f4b729de2e71 [SPARK-51107][CORE] Refactor CommandBuilderUtils#join to reuse the lines about strings join and reduce the redundant lines
    omit 8d18df320b71 [SPARK-51099][PYTHON] Add logs when the Python worker looks stuck
    omit 5a925c6f1d1b [SPARK-51104][DOC][FOLLOWUP] Self-host docsearch.min.css in Spark website
    omit e89b19f0d162 [SPARK-51104][DOC] Self-host JavaScript and CSS in Spark website
    omit a79ba486766b [SPARK-51042][SQL] Read and write the month and days fields of intervals with one call in Unsafe* classes
    omit 7416ae95532c [SPARK-49698][CONNECT][SQL] Add ClassicOnly annotation for classic only methods
    omit b968ce1d3ac1 [SPARK-43415][CONNECT][SQL] Implement `KVGDS.agg` with custom `mapValues` function
    omit aefaa6694567 [SPARK-49371][SPARK-51086][SQL][CONNECT][DOCS] Fix various doc SQL issues
    omit cfa8dbd651d9 [SPARK-51057][SS] Remove scala option based variant API for value state
    omit 84e924eb34d7 [SPARK-51010][SQL] Fix AlterColumnSpec not reporting resolved status correctly
    omit 88485a386a7a [SPARK-51050][SQL][TESTS] Add group by alias tests to the group-by-alias.sql
    omit d36eddad3e9f [SPARK-51105][ML][PYTHON][CONNECT][TESTS] Add parity test for ml functions
    omit 9dbbb5bd63ba [SPARK-51096][SQL][TESTS] Splitting TransformWithStateSuite into UnsafeRow and Avro encoding suites
    omit 96bd8f13f128 [SPARK-50605][CONNECT] Support SQL API mode for easier migration to Spark Connect
    omit 93e9f6e4e4a4 [SPARK-51084][SQL] Assign appropriate error class for `negativeScaleNotAllowedError`
    omit 0b7c4f15b301 [SPARK-50922][ML][PYTHON][CONNECT][FOLLOW-UP] Import pyspark.core module later for Spark Classic only
    omit 13eaf41aaade [SPARK-50799][PYTHON] Refine the docstring of rlike, length, octet_length, bit_length, and transform
    omit e6d44e449e1e [SPARK-51100][ML][PYTHON][CONNECT] Replace transformer wrappers with helper model attribute relations
    omit ad71780c5b41 [SPARK-51098][DOCS] Link exceptAll and subtract in Python DataFrame docs
    omit e2ef5a482604 [SPARK-51079][PYTHON] Support large variable types in pandas UDF, createDataFrame and toPandas with Arrow
    omit 940eb3666dfc [SPARK-51074][SS][CONNECT] Set timeout for communicating init configs in streaming foreachBatch
    omit f784b3be7542 [SPARK-49942][SQL] Rename `errorClass` to `condition` in `classifyException()`
    omit 4fd750c3f41c [SPARK-51091][ML][PYTHON][CONNECT] Fix the default params of `StopWordsRemover`
    omit 6a1d38219428 [SPARK-50954][ML][PYTHON][CONNECT] Support client side model path overwrite for meta algorithms
    omit c7f6e37e5192 [SPARK-51089][ML][PYTHON][CONNECT] Support `VectorIndexerModel.categoryMaps` on connect
    omit ec587936e881 [SPARK-51082][SQL][TESTS] Move all group by alias related tests to group-by-alias.sql
    omit 8b092beaa0be [SPARK-50975][ML][PYTHON][CONNECT] Support `CountVectorizerModel.from_vocabulary` on connect
    omit 6ed7733265c2 [SPARK-50958][ML][PYTHON][CONNECT] Support `Word2VecModel.findSynonymsArray` on connect
    omit 3675568a34d4 [SPARK-51090][DOCS][INFRA] Upgrade jekyll from 4.3.3 to 4.4.1
    omit 1fa293f681c4 [SPARK-51081][SQL] Refactor `Join.output` for single-pass resolver
    omit 4b08932a4fbb [SPARK-50949][ML][PYTHON][CONNECT][FOLLOW-UP] Fix uid issue and empty array args
    omit 5af4c764aa8b [SPARK-50930][ML][PYTHON][CONNECT] Support `PowerIterationClustering` on Connect
    omit 4138efd47139 [SPARK-50944][ML][PYTHON][CONNECT] Support `KolmogorovSmirnovTest` on Connect
    omit c4e200895ecc [SPARK-50949][ML][PYTHON][CONNECT] Introduce a helper model to support `StringIndexModel.from_labels_xxx`
    omit ac2f3a4dc814 [SPARK-51076][PYTHON][CONNECT] Arrow Python UDF fallback for UDT input and output types
    omit 198435ed01a5 [SPARK-51078][PYTHOM][ML][TESTS][FOLLOW-UP] Separate language selection test for StopWordsRemover
    omit 1fc9d7d92ec6 [SPARK-50982][SQL] Support more SQL/DataFrame read path functionality in single-pass Analyzer
    omit 16d007f05e27 [SPARK-50393][PYTHON][CONNECT] Introduce common TableArg for Spark Classic and Spark Connect
    omit 80cdc1562b66 [SPARK-51064][SQL] Enable `spark.sql.sources.v2.bucketing.enabled` by default
    omit 4953a9cfabec [SPARK-49308][CONNECT] Support UserDefinedAggregateFunction in Spark Connect Scala Client
    omit ebcec7c02f34 [SPARK-50104][CONNECT] Support SparkSession.executeCommand in Connect
    omit e0a7db2d2a7d [SPARK-51078][SPARK-50963][ML][PYTHON][CONNECT] Fix the py4j call in `StopWordsRemover`
    omit 505c644b40f4 [SPARK-51055][SS][CONNECT] Streaming foreachBatch should call init logic inside a try
    omit 572f57a0a41f [SPARK-50985][SS] Classify Kafka Timestamp Offsets mismatch error instead of assert and throw error for missing server in KafkaTokenProvider
    omit 0e25ea88ea42 [SPARK-50943][ML][PYTHON][CONNECT] Support `Correlation` on Connect
    omit 62534426a50d Revert "[SPARK-50605][CONNECT] Support SQL API mode for easier migration to Spark Connect"
    omit cee4e819b078 [SPARK-51078][CONNECT][TESTS][PYTHON] Skip unsupported StopWordsRemover test in Spark Connect
    omit 1bc4d15bbcd6 [SPARK-50605][CONNECT] Support SQL API mode for easier migration to Spark Connect
    omit 1c018d160ead Revert "[SPARK-46090][SQL] Support plan fragment level SQL configs in AQE"
    omit 0f163a5c2779 [SPARK-48353][SQL] Introduction of Exception Handling mechanism in SQL Scripting
    omit 9d6fb581726f [SPARK-51034][SQL] Reformat Describe As JSON statistics dict for parse-ability
    omit 2581ca17eafc [SPARK-51058][PYTHON] Avoid using jvm.SparkSession
    omit 400e2b29caf4 [SPARK-50541][SQL][FOLLOW-UP] Describe As JSON test case for non-existent table
    omit e037cb119a28 [SPARK-50133][PYTHON][CONNECT] Support DataFrame conversion to table argument in Spark Connect Python Client
    omit 5f23bc226c3c [SPARK-51070][SQL] Use `scala.collection.Set` instead of Set in ValidateExternalType
    omit 6f5ee5c65fef [SPARK-51043][SS][CONNECT] Fine grained user logging for Spark Connect foreachBatch
    omit ac8f8b08941b [SPARK-51061][CORE] Hide `Jetty` info in REST Submission API
    omit d29648d05fd9 [SPARK-50942][ML][PYTHON][CONNECT] Support `ChiSquareTest` on Connect
    omit bff86f11375c [SPARK-51060][ML][PYTHON][CONNECT][FOLLOW-UP] Fix the `uid` of `Bucketizer` fitted by `QuantileDiscretizer`
    omit 3e8a0c169207 [SPARK-51066][ML][PYTHON][CONNECT][TESTS] Enable more parity tests
    omit 5008c3d89421 [SPARK-50969][ML][CONNECT][PYTHON] Fix `GaussianMixtureModel.gaussians` on connect
    omit 3ac219843905 [SPARK-51063][BUILD][TESTS] Upgrade `bouncycastle` to 1.80
    omit f03f45e3a7f1 [SPARK-51011][CORE] Add logging for whether a task is going to be interrupted when killed
    omit 8fc6a200eb53 [SPARK-50843][ML][PYTHON][CONNECT][FOLLOW-UP] Optimize the RPC for `TreeEnsembleModel.trees`
    omit f840abbe593c [SPARK-50899][ML][PYTHON][CONNECT] Support PrefixSpan on connect
    omit 9ad8d226a8e0 [SPARK-51060][ML][PYTHON][CONNECT] Support `QuantileDiscretizer` on Conenct
    omit 736752ec1ea0 [SPARK-50974][ML][PYTHON][CONNECT] Add support foldCol for CrossValidator on connect
    omit f2d65eeaa2b7 [SPARK-51015][ML][PYTHON][CONNECT] Support RFormulaModel.toString on Connect
    omit ba18f502efbf [SPARK-50843][ML][PYTHON][CONNECT] Support return a new model from existing one
    omit e1c8b4d3fdee [SPARK-51056][PYTHON] Avoid unnecessary object creation in SparkSession.getActiveSession
    omit 5cd25fc8878e [SPARK-51047][SS][TESTS] Add tests to verify scan ordering for non-zero start ordinals as well as non-ascending ordinals
    omit 23704d5b25b4 [SPARK-51044][SS][TESTS] Add ordering related tests for list state
    omit cfc3f1f479e2 [SPARK-50967][SS] Add option to skip emitting initial state keys within the FMGWS operator
    omit bf442c5f1914 [SPARK-50265][CONNECT] Support spark.udf.registerJavaUdf in Connect
    omit 39433d4f6732 [SPARK-51054][BUILD] Upgrade Netty to 4.1.117.Final
    omit a9d194d0020b [SPARK-51053][SQL] Update `CookieSigner` to use `SHA-512`
    omit 4a33e965846a [MINOR][SQL] Remove signature from Hive thriftserver DEBUG log
    omit b62c3f4f15b0 [SPARK-51049][CORE] Increase S3A Vector IO threshold for range merge
    omit 7eb12d7799e4 [SPARK-51045][TESTS] Regenerate benchmark results after upgrading to Scala 2.13.16
    omit ece14704cc08 [SPARK-49428][SQL] Move Connect Scala Client from Connector to SQL
    omit ecf6851a97c0 [SPARK-51041][BUILD] Add `hive-llap-client` and `hive-llap-common` as test dependency of `hive-thriftserver`
    omit 994b4b13db9d [SPARK-51021] Add log throttler
    omit 19fbeaa04abe [SPARK-51030][CORE][TESTS] Add a check before `Utils.deleteRecursively(tempDir)` to ensure `tempDir` won't be cleaned up by the ShutdownHook in `afterEach` of `LocalRootDirsTest`
    omit 7243de6fe716 [SPARK-51039][BUILD] Fix `hive-llap-common` dependency to use `hive.llap.scope` in root `pom.xml`
    omit 9e1bfa13fd15 [SPARK-51038][INFRA] Add `branch-4.0` to daily `Publish Snapshot` GitHub Action job
    omit 585f43477be3 [SPARK-50807][BUILD] Upgrade Scala to 2.13.16
    omit a1b0f256c04e [SPARK-51036][BUILD][TESTS] Upgrade `snowflake-jdbc` to 3.22.0
    omit 5cce6e9affc2 [SPARK-51033][CORE][TESTS] Fix `CryptoStreamUtilsSuite` to use `sameElements` for array comparison
    omit 339b036594d9 [SPARK-51029][BUILD] Remove `hive-llap-common` compile dependency
    omit 5b796c396625 [SPARK-50663][SQL] Fix grammar for IF ELSE statement (ELSE IF -> ELSEIF)
    omit 6bbfa2dad8c7 [SPARK-50979][CONNECT] Remove .expr/.typedExpr implicits
    omit 840f74a18691 [SPARK-51022][SQL] Remove unused `tableSampleClause` in `build()` method of `MsSqlServerSQLQueryBuilder` and `DB2SQLQueryBuilder`
    omit 42898c62b4c1 [SPARK-51027][SQL] Prevent `HiveClient.runSqlHive` invocation in non-testing environment
    omit 3badfd2c41d3 [SPARK-51026][CORE][TESTS] Remove the unnecessary test file `StagePageSuite.scala`
    omit d66667f56d74 [SPARK-50906][SQL][SS] Add nullability check for if inputs of to_avro align with schema
    omit 8dbf1dd82bfb [SPARK-50909][PYTHON][FOLLOWUP] Disable segfault tests in pypy
    omit af9f4ef1e741 [SPARK-49961][SQL] Correct transform type signature for both Scala and Java
    omit c7e8d63f9c18 [SPARK-51025][K8S] Install `libssl-dev` in K8s Dockerfile
    omit cb38cd2ead7e [SPARK-51024][BUILD] Upgrade `wildfly-openssl` to 2.2.5.Final
    omit 373fd4aee5c6 [MINOR][DOCS] Fix typos in `docs/sql-data-sources-jdbc.md`
    omit 8a6329870a84 [SPARK-50965][SQL][TESTS] Making sure that multiple parameterized queries work on SparkConnect
    omit 468ae8aa2938 [SPARK-50666][SQL][FOLLOWUP] Revise JDBC hint support and remove a typo in comment
    omit 80f70a5dcf87 [SPARK-51001][SQL] Refine arrayEquals
    omit 98957380be98 [SPARK-51012][SQL] Remove SparkStrategy from Connect Shims
    omit dd51f0e7592d [SPARK-50918][ML][PYTHON][CONNECT] Refactor read/write for Pipeline
    omit b49ef2ada0fc [SPARK-50922][ML][PYTHON][CONNECT] Support OneVsRest on Connect
    omit 9a450190e810 [SPARK-51014][ML][PYTHON][CONNECT] Support RFormula on connect
    omit 04f862aaf94d [SPARK-51017][INFRA] Add Daily PyPy3.10 GitHub Action Job for `branch-4.0`
    omit aa24a9a235b1 [SPARK-50812][ML][PYTHON][CONNECT] Add support PolynomialExpansion
    omit e891627bf9e2 [SPARK-51006][BUILD] Upgrade `gcs-connector` to `2.2.26`
    omit 0c283a86653a Revert "[SPARK-50922][ML][PYTHON][CONNECT] Support OneVsRest on Connect"
    omit 4e5a813119e8 [SPARK-51009][INFRA] Add Daily Maven Java 21 CI for `branch-4.0`
    omit 22bac2eeb590 [SPARK-50922][ML][PYTHON][CONNECT] Support OneVsRest on Connect
    omit a452e1bc2189 [SPARK-50977][CORE] Enhance availability of logic performing aggregation of accumulator results
    omit d259132156e2 [SPARK-50858][PYTHON] Add configuration to hide Python UDF stack trace
    omit 704c901986b4 [SPARK-50938][ML][PYTHON][CONNECT] Support VectorSizeHint and VectorSlicer on Connect
    omit 9d8f39b2007b [SPARK-51005][ML][PYTHON][CONNECT] Support VectorIndexer and ElementwiseProduct on Connect
    omit fef1b2375c30 [SPARK-50666][SQL] Support hint for reading in JDBC data source
    omit b5deb8dc25b8 [SPARK-51004][ML][PYTHON][CONNECT] Add supports for IndexString
    omit b0e18ba71c98 [SPARK-51003][ML][PYTHON][CONNECT] Support LSH models on Connect
    omit 0e4242d4b2e1 [SPARK-50923][SPARK-50927][ML][PYTHON][CONNECT] Support FMClassifier and FMRegressor on Connect
    omit 9d0e88894514 [SPARK-50941][ML][PYTHON][CONNECT] add supports for TrainValidationSplit
    omit 3ba76bff5369 [SPARK-50924][SPARK-50926][ML][PYTHON][CONNECT] Support AFTSurvivalRegression and IsotonicRegression on Connect
    omit 6baddd08866d [SPARK-50921][ML][PYTHON][CONNECT] Support MultilayerPerceptronClassifier on Connect
    omit 771d81ab48a6 [SPARK-50558][SQL] Introduce simpleString for ExpressionSet
    omit 66c29209eee6 [SPARK-50995][ML][PYTHON][CONNECT] Support `clusterCenters` for KMeans and BisectingKMeans
    omit b6b00e87b00b [SPARK-50929][ML][PYTHON][CONNECT] Support `LDA` on Connect
    omit 394eebd8da4f [SPARK-51000][K8S][DOCS] Drop K8s v1.29 Support
    omit 2b70117304a7 [SPARK-50998][K8S][DOCS] Fix `spark.kubernetes.configMap.maxSize` default value in docs
    omit 77096a29e4fe [SPARK-50997][DOCS] Remove `\t` character in `docs`
    omit 9da1cd02951b [SPARK-50996][K8S] Increase `spark.kubernetes.allocation.batch.size` to 10
    omit e7821c82e28d [SPARK-50993][SQL] Move nullDataSourceOption from QueryCompilationErrors into QueryExecutionErrors
    omit 7c12ff62ac03 [SPARK-50988][ML][PYTHON][CONNECT] Fix uid inconsistencies for estimator and model
    omit 43785de6e773 [SPARK-50925][ML][PYTHON][CONNECT] Support GeneralizedLinearRegression on Connect
    omit d0b1b0b22e8d [SPARK-50990][SQL] Refactor `UpCast` resolution out of the `Analyzer`
    omit 762599cc64df [SPARK-50940][ML][PYTHON][CONNECT] Adds support CrossValidator/CrossValidatorModel on connect
    omit e0437e0021ef [SPARK-50920][ML][PYTHON][CONNECT] Support NaiveBayes on Connect
    omit acdda8a3eb15 [SPARK-50989][ML][PYTHON][CONNECT] Support NGram, Normalizer and Interaction on connect
    omit 4af6be6f4ae6 [SPARK-50937][ML][PYTHON][CONNECT] Support `Imputer` on Connect
    omit ff6b1a9fb705 [SPARK-49639][SQL] Remove the ANSI config suggestion in INVALID_INTERVAL_WITH_MICROSECONDS_ADDITION
    omit a45eb2b5ba2d [SPARK-50956][ML][TESTS][FOLLOW-UP] Skip flaky EvaluationTestsOnConnect
    omit aa9a10470aa4 [SPARK-50936][ML][PYTHON][CONNECT] Support HashingTF, IDF and FeatureHasher on connect
    omit 8d2b57d1a1f8 [SPARK-50812][ML][PYTHON][CONNECT] Support TargetEncoderModel on Connect
    omit 21f051246101 [SPARK-50987][DOCS] Make `spark-connect-overview.md`s version strings up-to-date
    omit 60dd497a19b7 [SPARK-50978][PYTHON][SS] Fix naming convention for mapStateClient
    omit f5549d7c34ca [SPARK-50984][SQL][TESTS] Make `ExpressionImplUtilsSuite` robust by matching JDK msgs via regex
    omit 44966c94b686 [SPARK-50976][ML][PYTHON] Fix the save/load of `TargetEncoder`
    omit 5db31aec33c5 [SPARK-49700][CONNECT][SQL] Unified Scala Interface for Connect and Classic
    omit a03c4cbd37f0 [SPARK-50973][SQL][AVRO] Cleanup deprecated api usage related to `avro.Schema#toString(boolean)`
    omit 7e3de64b2ac6 [SPARK-50792][SQL][FOLLOWUP] Improve the push down information for binary
    omit 4021d911ec5d [SPARK-50838][SQL] Performs additional checks inside recursive CTEs to throw an error if forbidden case is encountered
    omit 5c1f7c248e13 [SPARK-50934][ML][PYTHON][CONNECT] Support CountVectorizer and OneHotEncoder on Connect
    omit 39e9b3bec72b [SPARK-50932][ML][PYTHON][CONNECT] Support Bucketizer on Connect
    omit c662441de3a4 [SPARK-50970][PYTHON][SS] Unify naming convention for stateful processor APIs
    omit 5d6bcb9362db [SPARK-50851][ML][CONNECT][FOLLOW-UP] Simplify the message by applying the UDT
    omit e887f054b5a5 [SPARK-50933][ML][PYTHON][CONNECT] Support Feature Selectors on Connect
    omit 4e3b8318cac1 [SPARK-50853][CORE] Close temp shuffle file writable channel
    omit 1569c8a33aa6 [SPARK-50971][CORE] Combine the two retrievals of `taskDescription` in `CoarseGrainedExecutorBackend#statusUpdate`
    omit a8a6078e9975 [SPARK-50972][BUILD] Upgrade Guava to 33.4.0
    omit 6955bd59f63b [SPARK-50931][ML][PYTHON][CONNECT] Support Binarizer on connect
    omit 3377962bc918 [SPARK-50963][ML][PYTHON][TESTS][FOLLOW-UP] Enable a parity test
    omit 560dd5e02384 [SPARK-50928][ML][PYTHON][CONNECT] Support GaussianMixture on Connect
    omit 311a4e04e3c0 [SPARK-50884][ML][PYTHON][CONNECT] Support isLargerBetter in Evaluator
    omit 8f66aefaab56 [SPARK-50908][SS][PYTHON] Disable flaky TTL test in `test_pandas_transform_with_state.py`
    omit 2b657febab25 [SPARK-50798][SQL][FOLLOWUP] Normalize expressions with random seed
    omit be77e1eb049f [SPARK-50770][SS] Removing package scope for transformWithState operator APIs
    omit 7c316f7f8e75 [SPARK-50968][PYTHON] Fix the usage of `Column.__new__`
    omit 2d74c3d9885a [SPARK-50935][ML][PYTHON][CONNECT] Support DCT on connect
    omit 5c26684193ec [SPARK-50919][ML][PYTHON][CONNECT] Support LinearSVC on connect
    omit 42b15c9eb7d7 [SPARK-50963][ML][PYTHON][CONNECT] Support Tokenizers, SQLTransform and StopWordsRemover on Connect
    omit fe29f83127f8 [SPARK-50962][SQL] Avoid StringIndexOutOfBoundsException in AttributeNameParser
    omit 1a4923773219 [SPARK-50939][ML][PYTHON][CONNECT] Support Word2Vec on Connect
    omit 83961bc59a41 [MINOR][SQL][FOLLOWUP] Fix a `mode` test in DataFrameSuite
    omit 74cf4d82edd7 [SPARK-50959][ML][PYTHON] Swallow the exception of JavaWrapper.__del__
    omit 6e4240f591a4 [SPARK-50947][PYTHON][SQL][CONNECT] Assign appropriate error class and SparkException for duplicated artifacts
    omit 620f55262cab [SPARK-50957][PROTOBUF] Make protobuf.utils.SchemaConverters private
    omit 65adc68a2c7c [SPARK-50918][ML][PYTHON][CONNECT] Support Pipeline on connect
    omit 263e46a1bff7 [SPARK-50812][ML][PYTHON][CONNECT][TESTS] refactor some tests
    omit 9f5a94c33250 [MINOR][SQL] Rename the internal var DataFrameWriter.mode to DataFrameWriter.curmode
    omit 0721d9c24170 [SPARK-50956][PYTHON][TESTS] Skip flaky `EvaluationTestsOnConnect.test_multiclass_classifier_evaluator`
    omit 5e936f29e486 [SPARK-50826][ML][CONNECT] Refactor the way handling `ALLOWED_ATTRIBUTES`
    omit e02ff1c0bcad [SPARK-50091][SQL] Handle case of aggregates in left-hand operand of IN-subquery
    omit 4b352823720a [SPARK-50883][SQL] Support altering multiple columns in the same command
    omit 917a9a10d541 [SPARK-50909][PYTHON] Setup faulthandler in PythonPlannerRunners
    omit 62a305b424ab [SPARK-50815][FOLLOW-UP] Handle Variant-related edge-case for createDataFrame
    omit 3bbb12cbd5e6 Revert "[SPARK-50118][CONNET] Reset isolated state cache when tasks are running"
    omit 44a334f520a0 [MINOR][SQL] Avoid importing java.util at DataFrameWriter
    omit e66503cf4d0f [SPARK-50952][BUILD] Include `jjwt`-related libraries and provide `jjwt-provided` profile
    omit af2768d027a5 [SPARK-50951][BUILD][TESTS] Update Oracle free version from 23.5 to 23.6
    omit 54a59b7f3ceb [SPARK-50946][CORE][TESTS] Add version check for Java 17.0.14 to make `UtilsSuite` test pass
    omit 6bde48c3f891 [SPARK-49646][SQL] add spark config for fixing subquery decorrelation for union/set operations when parentOuterReferences has references not covered in collectedChildOuterReferences
    omit 665884619481 [SPARK-50948][ML][PYTHON][CONNECT] Add support for StringIndexer/PCA on Connect
    omit 454463bd12f1 [SPARK-50879][ML][PYTHON][CONNECT][FOLLOW-UP] Support RobustScaler on Connect
    omit 8611d0fa05b4 [SPARK-50915][PYTHON][CONNECT] Add `getCondition` and deprecate `getErrorClass` in `PySparkException`
    omit 8313320ab013 [SPARK-50820][SQL] DSv2: Conditional nullification of metadata columns in DML
    omit 48aa25acb28a [SPARK-50913][PYTHON][TESTS] Skip flaky EvaluationTestsOnConnect.test_binary_classifier_evaluator
    omit 09cb059c8dde [SPARK-50798][SQL][FOLLOWUP] Further improvements to `NormalizePlan`
    omit 42838a94153d [SPARK-50912][PYTHON][TESTS] Skip FrameTakeAdvParityTests.test_take_adv because of OOM for now
    omit 3450184d332c [SPARK-50901][ML][PYTHON][CONNECT] Support Transformer `VectorAssembler`
    omit fab0cca80eb4 [SPARK-50895][SQL] Create common interface for expressions which produce default string type
    omit 001e2443d9cc [SPARK-50905][SQL][TESTS] Rename `Customer*` to `Custom*` in `SparkSessionExtensionSuite`
    omit dabd21249f1e [SPARK-50904][SQL] Fix collation expression walker query execution
    omit 98f276730d1d [SPARK-50902][CORE][K8S][TESTS] Add `CRC32C` test cases
    omit e88d8bc90510 [SPARK-50880][SQL] Add a new visitBinaryComparison method to V2ExpressionSQLBuilder
    omit 6d66f26c3e5d [SPARK-50879][ML][PYTHON][CONNECT] Support feature scalers on Connect
    omit 3ba74bf9b509 [SPARK-50898][ML][PYTHON][CONNECT] Support `FPGrowth` on connect
    omit ce073960422a [SPARK-50751][SQL] Assign appropriate error condition for `_LEGACY_ERROR_TEMP_1305`: `UNSUPPORTED_TABLE_CHANGE_IN_JDBC_CATALOG`
    omit 48acded7f1b9 [SPARK-50718][PYTHON] Support `addArtifact(s)` for PySpark
    omit 4c663168f4c6 [SPARK-50897][ML][CONNECT] Avoiding instance creation in ServiceLoader
    omit 5534b91dee6b [SPARK-50875][SQL] Add RTRIM collations to TVF
    omit 24abb0fbb1db [SPARK-50870][SQL] Add the timezone when casting to timestamp in V2ScanRelationPushDown
    omit 1cdb9181f91b [SPARK-50894][SQL] Postgres driver version bump to 42.7.5
    omit cbc8892e83d1 [SPARK-50900][ML][CONNECT] Add VectorUDT and MatrixUDT to ProtoDataTypes
    omit 79347e08e2e7 [SPARK-50844][ML][CONNECT] Make model be loaded by ServiceLoader when loading
    omit 335046236b16 [SPARK-50896][INFRA] Add Daily Maven CI to branch-4.0
    omit 07aa4ff08aad [SPARK-50735][CONNECT] Failure in ExecuteResponseObserver results in infinite reattaching requests
    omit efeb1e01ce42 [SPARK-50893][CONNECT] Mark UDT.DataType optional
    omit 78e4b79a1595 [SPARK-50878][ML][PYTHON][CONNECT] Support ALS on Connect
    omit 4443078ab8fa [SPARK-50877][ML][PYTHON][CONNECT] Support KMeans & BisectingKMeans on Connect
    omit 29c77ddac93a [SPARK-50847][SQL] Deny ApplyCharTypePadding from applying on specific In expressions
    omit 4874b6e2c34a [SPARK-50891][BUILD] Remove the explicit dependency on `Guava` from `plugins.sbt`
    omit cf5934ad56e3 [MINOR][DOCS] Fix miss semicolon on list file sql example
    omit 1c4cfcb0277d [SPARK-50876][ML][PYTHON][CONNECT] Support Tree Regressors on Connect
    omit 1e378b34676b [SPARK-50890][PYTHON][TESTS][CONNECT] Skip test_take in Spark Connect only build
    omit c7df014aab14 [SPARK-49273][CONNECT][SQL] Origin support for Spark Connect Scala client
    omit 95ec08ee64e1 [SPARK-50862][BUILD] Upgrade rocksdbjni to 9.8.4
    omit 404bc2e59610 [SPARK-50882][SQL][TESTS] Skip `TPCDSCollationQueryTestSuite.q22-v2.7` test in GitHub Action CI
    omit 66dd7ddc4be3 [SPARK-50739][SQL][FOLLOW] Simplify ResolveRecursiveCTESuite with dsl
    omit 6b491bc35f12 [SPARK-50861][BUILD] Upgrade dropwizard metrics to 4.2.30
    omit c3f6c58b8903 [SPARK-50872][SQL][UI] Makes ToPrettyString expression not affect UI presentation
    omit b104db93034f [SPARK-50885][BUILD] Change to use `SbtPomKeys` to obtain `io.grpc.version` in `SparkBuild.scala`
    omit 98d9968b9a74 [SPARK-50808][CORE] Fix issue in writeAll with mixed types not getting written properly
    omit 8b12bbb69c0d [SPARK-50868][BUILD] Upgrade `scalafmt` to 3.8.5
    omit 93e56c26acec [SPARK-50874][ML][PYTHON][CONNECT] Support `LinearRegression` on connect
    omit ef363b636b2f [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet
    omit 205e382d5643 [SPARK-50652][SS] Add checks to RocksDB V2 backward compatibility
    omit 3ef529be73e0 [SPARK-49014][BUILD][DOCS][FOLLOWUP] Synchronize the Avro version in `SparkBuild.scala/docs/comments to` 1.12.0
    omit 4e47023eaa01 [SPARK-50871][BUILD] Upgrade `scala-parallel-collections` to 1.2.0
    omit 3b114af47ace [SPARK-50739][SQL] Recursive CTE. Analyzer changes to unravel and resolve the recursion components
    omit 2721a50234b0 [SPARK-50851][ML][CONNECT][PYTHON] Express ML params with `proto.Expression.Literal`
    omit 10dd350acbb1 [MINOR][CORE] Fix compilation warnings in `UnsafeInMemorySorter.java`
    omit 3b0ac4539170 [SPARK-50866][INFRA] Limit GHA job execution time to up to 2 hours
    omit 3a120384f060 [SPARK-50867][PYTHON][TESTS] Move `search_jar` and `SPARK_HOME` to `sqlutils.py`
    omit c81fa332099c [SPARK-50863][PYTHON][ML][TESTS] Skip slow pytorch tests
    omit 371ae0359b89 [SPARK-50263][CONNECT] Replace `System.currentTimeMillis` with `System.nanoTime`
    omit 3892db7bf05a [SPARK-50618][SS][SQL] Make DataFrameReader and DataStreamReader leverage the analyzer more
    omit 5fa6dbca7739 [SPARK-50860][INFRA] Clarify Exclusion of Doc-Only Updates from User-Facing Changes
    omit baf13fbd4afe [SPARK-50859][BUILD] Upgrade AWS SDK v2 to 2.25.53
    omit 41aa0b5ebf44 [SPARK-50714][SQL][SS] Enable schema evolution for TransformWithState when Avro encoding is used
    omit 11d758f48141 [SPARK-50852][INFRA] Add Java 21 Daily GitHub Action job for `branch-4.0`
    omit e2cff2264d3f [SPARK-50829][SQL] Add flag to disable session collation by default
    omit ffb31565e5af [SPARK-50633][INFRA][FOLLOWUP] Let CODECOV_TOKEN transfer to build_and_test.yml
    omit 523ff15b2891 [SPARK-46934][SQL][FOLLOWUP] Handle Special Characters for HMS
    omit 96adcc442112 [SPARK-50831][SQL] Enable trimming collation by default
    omit 86804f6da5fc [SPARK-50827][ML][CONNECT] Support plugin
    omit 8bbec5df6e7e [SPARK-50795][SQL] Store timestamp as `long` type in `describe` LinkedHashMap
    omit 6ebed5b27814 [SPARK-50830][SQL] Return single-pass result as the dual run analysis result
    omit d12bb23e4992 [SPARK-50825][ML][PYTHON][CONNECT] Support Tree Classifiers in ML Connect
    omit 97241eb0cfc9 [SPARK-49907][ML][CONNECT][FOLLOW-UP] Move pyspark.ml.remote to pyspark.ml.connect
    omit 813591faddb6 [SPARK-50792][SQL] Format binary data as a binary literal in JDBC
    omit 9b323340749d [SPARK-50768][SQL][CORE][FOLLOW-UP] Apply TaskContext.createResourceUninterruptibly() to risky resource creations
    omit 90801c2b9740 [SPARK-50811] Support enabling JVM profiler on driver
    omit 0ca963337637 [SPARK-50654][SS][TESTS][FOLLOWUP] Fix compile issue of `RocksDBStateStoreCheckpointFormatV2Suite`
    omit b6685e36354a [SPARK-50654][SS] CommitMetadata should set stateUniqueIds to None in V1
    omit 0c58c003c2f9 [SPARK-49886][SS] Query level failure test for rocksDB format V2
    omit c3671a592ddd [SPARK-50804][SQL] to_protobuf() should not throw MatchError
    omit dff46cfeabfb [SPARK-49907][ML][CONNECT][FOLLOW-UP] Add ml connect related packages into PyPI packages
    omit 255d923d0efa [SPARK-50756][CORE] Use error class for exceptions in SparkConf.validateSettings
    omit bd56285e7fb2 [SPARK-50817][SS] Fix RocksDB Error Reporting
    omit 4bd37aa6f22a [SPARK-50819] Refactor Spark profiler module
    omit 9841ae2c254e [SPARK-50836][INFRA] Add non-ANSI SQL Daily CI for `branch-4.0`
    omit 608690cba1ec [SPARK-50833][INFRA] Update `buf` GitHub Action job to compare against branch-4.0
    omit cb4032c66ae4 [SPARK-50835][INFRA][FOLLOWUP] Use Python 3.11 in `branch-4.0` `Python-only` Daily CI
    omit aefc3538bc7a [SPARK-50835][INFRA] Add `Python-only` Daily CI to branch-4.0
    omit e1ccc168e062 [SPARK-50834][INFRA] Add Daily Github Action CI to `branch-4.0`
    omit 971464505714 [SPARK-50822][BUILD] Setting version to 4.1.0-SNAPSHOT
    omit 1c97b1babbbe [SPARK-50522][SQL] Support for indeterminate collation
    omit 6a7d42b78d37 [SPARK-50810][BUILD][FOLLOWUP] Fix code change detection for profiler module
    omit 973db2114bf6 [SPARK-50828][PYTHON][ML][CONNECT] Deprecate `pyspark.ml.connect`
    omit 39bb2d84ce22 [SPARK-50815][PYTHON][SQL] Fix bug where passing null Variants in createDataFrame causes it to fail and add Variant support in createDataFrame in Spark Connect
    omit 21a37a79df97 [SPARK-50814][DOCS] Remove unused SQL error pages
    omit 47d831ea58dc [SPARK-50810][BUILD] Enable SBT CI for profiler module
    omit 15173157c33e [SPARK-49907][ML][CONNECT][FOLLOWUP] Remove unused `expressions.proto` import
    omit 7e1248ab770b [SPARK-50824][PYTHON] Avoid importing optional Python packages for checking
    omit 0593ac6d345a [SPARK-50541][SQL][FOLLOWUP] Migrate DESC TABLE AS JSON to v2 command
    omit d79286546057 [SPARK-50821][PYTHON] Upgrade Py4J from 0.10.9.8 to 0.10.9.9
    omit d2ecfd73a3d7 [SPARK-50823][PYTHON] Upgrade cloudpickle from 3.1.0 to 3.1.1
    omit 6f3b778e1a12 [SPARK-50768][CORE] Introduce TaskContext.createResourceUninterruptibly to avoid stream leak by task interruption
    omit f223b8da9e23 [SPARK-50818][PYTHON] Replace `has_numpy` with `have_numpy`
    omit f0842d049ded [SPARK-50813][SQL] Allow only unqualified label names inside SQL Scripts
    omit 488c36247168 [SPARK-50788][TESTS] Add Benchmark for Large-Row Dataframe
    omit 1fd836271b6f [SPARK-50403][SQL] Fix parameterized `EXECUTE IMMEDIATE`
    omit 2d498d51ab83 Revert "[SPARK-50804][SQL] to_protobuf() should not throw MatchError"
    omit bfaa814f3026 Revert "[SPARK-46934][SQL][FOLLOWUP] Read/write roundtrip for struct type with special characters with HMS - a backward compatible approach"
    omit 216b53304613 [SPARK-50795][SQL] Display all DESCRIBE AS JSON dates in ISO-8601 format and types as `dataType.simpleString`
    omit fafe43c87d8b [SPARK-49907][ML][CONNECT] Support spark.ml on Connect
    omit f96417fe0d37 [SPARK-47081][CONNECT][FOLLOW-UP] Respect spark.connect.progress.reportInterval over timeout
    omit f355293f8bc8 [SPARK-50801][SQL] Improve `PlanLogger.logPlanResolution` so it shows just unresolved and resolved plans
    omit e4cbb91eb272 [SPARK-50804][SQL] to_protobuf() should not throw MatchError
    omit bba6839d8714 [SPARK-50762][SQL] Add Analyzer rule for resolving SQL scalar UDFs
    omit 3c7f5e25b70c [SPARK-50790][PYTHON] Implement parse json in pyspark
    omit 94a31bd9c144 [SPARK-48809][PYTHON][DOCS] Reimplemented `spark version drop down` of the `PySpark doc site` and fix bug
    omit b89082b4190d [SPARK-50774][SQL] Centralize collation names in one place
    omit ebe39f6adc8b [MINOR][DOCS] Fix the examples of createDataFrame
    omit e945a9031d4e [SPARK-50783] Canonicalize JVM profiler results file name and layout on DFS
    omit 0e51dba27800 [SPARK-50633][FOLLOWUP] Set `CODECOV_TOKEN` with environment variables
    omit e62697fc585a [SPARK-50805][CORE] Move method `nameForAppAndAttempt` to `o.a.s.u.Utils`
    omit 8858a4c9628f [SPARK-47081][CONNECT][FOLLOW-UP] Respect `spark.connect.progress.reportInterval` over timeout
    omit 22731393069a [SPARK-50601][SQL] Support withColumns / withColumnsRenamed in subqueries
    omit 0a8798f378ca [SPARK-48745][INFRA][PYTHON][TESTS][FOLLOWUP] use `conda-incubator/setup-miniconda` action
    omit c5529d31513d [SPARK-50773][CORE] Disable structured logging by default
    omit 313e824931fd [SPARK-50392][PYTHON][FOLLOWUP] Move `import`s into methods to fix `connect-only` builds
    omit c1e5f53cbe0f [SPARK-50791][SQL] Fix NPE in State Store error handling
    omit 3569e768e657 [SPARK-50789][CONNECT] The inputs for typed aggregations should be analyzed
    omit 5ef0eb5f4bb9 [SPARK-50734][SQL][FOLLOWUP] Replace `ScalaObjectMapper` with `ClassTagExtensions` to fix compilation warning
    omit aaff18b98044 [SPARK-50798][SQL] Improve `NormalizePlan`
    omit d7545d0fc096 [SPARK-50624][SQL] Add TimestampNTZType to ColumnarRow/MutableColumnarRow
    omit 3f864558aaff [SPARK-50782][SQL] Replace the use of reflection in `CodeGenerator.updateAndGetCompilationStats` with direct calls to `CodeAttribute#code`
    omit 4cecde625df1 [SPARK-50797][SQL][TESTS] Move `HiveCharVarcharTestSuite` from `o/a/s/sql` to `o/a/s/sql/hive`
    omit e284b2ce1ec2 [SPARK-50525][SQL][TESTS][FOLLOWUP] Fix `DataFrameSuite.repartition by MapType` test assumption
    omit 898f7afc5dd7 [SPARK-50707][SQL][TESTS][FOLLOWUP] Fix `CharVarcharTestSuite` test case assumption
    omit d66166a5f024 [SPARK-50796][BUILD] Upgrade `protobuf-java` to 4.29.3
    omit 13e9e4cfeb3f [SPARK-48745][INFRA][PYTHON][TESTS][FOLLOWUP] Fix the `pyspark-error` testing environment in GA
    omit 14a3389151df [SPARK-50794][BUILD] Upgrade Ivy to 2.5.3
    omit f0f850dc21ec [SPARK-50787][DOCS] Fix typos and add missing semicolons in sql examples
    omit 9d4b7a5dd444 [SPARK-50784][TESTS] Fix `lint-scala` not to ignore `scalastyle` errors
    omit 3a84dfc776ae [SPARK-50786][SQL] Remove `removeWhitespace helper` for DESCRIBE TABLE
    omit be834ee7290f [SPARK-50776][K8S][TESTS][FOLLOWUP] Fix code style
    omit f9cb80a5947b [SPARK-50616][SQL] Add File Extension Option to CSV DataSource Writer
    omit d3848ced1aff [SPARK-50728][BUILD] Update ORC to 2.1.0
    omit 62a23dd55a69 [SPARK-50780][SQL] Use `overrideStdFeatures` instead of `setFeatureMask` in  `JacksonParser`
    omit 9547a476df1f [SPARK-50776][K8S][TESTS] Fix test assertions on executor service account
    omit 7bc8e99cde42 [SPARK-50707][SQL] Enable casting to/from char/varchar
    omit 3c831ebf7cd8 [MINOR][DOCS] Add new dataframe methods to API references
    omit 8668c1739570 [MINOR][DOCS] Fix miss semicolon on describe sql example
    omit 793ad7c87b46 [SPARK-50633][FOLLOWUP] Turn on options `verbose` for `codecov/codecov-action`
    omit 784586700d9a [SPARK-50781][SQL] Cache `QueryPlan.expressions`
    omit 868549e7ac63 [SPARK-50560][SQL] Improve type coercion and boundary checking for RANDSTR SQL function
    omit 8b209d188f6c [SPARK-50124][SQL][FOLLOWUP] InsertSortForLimitAndOffset should propagate missing ordering columns
    omit a022bf0a348f [SPARK-50634][INFRA] Upgrade `codecov/codecov-action` to v5
    omit 1208d3cf6483 [SPARK-50633][INFRA] Fix `codecov/codecov-action@v4` daily scheduling failure
    omit 6320b5cdbb1d [SPARK-50779][SQL] Adding feature flag for object level collations
    omit fe3d3ae6172b [SPARK-50719][PYTHON] Support `interruptOperation` for PySpark
    omit a4f2870b0855 [SPARK-50525][SQL] Define InsertMapSortInRepartitionExpressions Optimizer Rule
    omit 68305acc01fc [SPARK-50600][CONNECT][SQL] Set analyzed on analysis failure
    omit a8bec11af941 [SPARK-49565][SQL] Improve auto-generated expression aliases with pipe SQL operators
    omit e638f6d20f25 [SPARK-50778][PYTHON] Add metadataColumn to PySpark DataFrame
    omit 48a81194351e [SPARK-50777][CORE] Remove redundant no-op `init/destroy` methods from `Filter` classes
    omit 66162360e773 [SPARK-48344][SQL] Enhance SQL Script Execution: Replace NOOP with COLLECT for Result DataFrames
    omit a22991700fab [SPARK-49883][SS][TESTS][FOLLOWUP] RocksDB Fault Tolerance Test
    omit f8e8fcc835d7 [SPARK-50705][SQL] Make `QueryPlan` lock-free
    omit 0123a5ecbe6d [SPARK-50764][PYTHON] Refine the docstring of xpath related methods
    omit f0fb8bf0f81a [SPARK-50749][SQL] Fix ordering bug in CommutativeExpression.gatherCommutative method
    omit ee868b9a827e [SPARK-50561][SQL] Improve type coercion and boundary checking for UNIFORM SQL function
    omit 22735bedc497 [SPARK-50772][SQL] Retain table aliases after SET, EXTEND, DROP operators
    omit b84dc909a885 [SPARK-50769][SQL] Fix ClassCastException in HistogramNumeric
    omit cb093e68f312 [SPARK-50685][PYTHON][FOLLOW-UP] Improve Py4J performance by leveraging getattr
    omit bbb8aca0b510 [SPARK-50754][BUILD] Upgrade commons-codec to 1.17.2
    omit dabf22795969 [SPARK-50766][TESTS] Regenerate benchmark results
    omit ceb7583ed394 Revert "[SPARK-50756][SQL] Use error class for exceptions in SparkConf.validateSettings"
    omit afb4f822470a [SPARK-50752][PYTHON][SQL] Introduce configs for tuning Python UDF without Arrow
    omit 507de06137f1 [SPARK-50083][SQL] Integrate `_LEGACY_ERROR_TEMP_1231` into `PARTITIONS_NOT_FOUND`
    omit eb02144cffb3 [SPARK-50760][SQL][TESTS] Add negative test cases to `table-aliases.sql` in line with `HiveResolutionSuite`
    omit e0c9ddcec9ae [SPARK-50743][SQL] Normalize CTERelationDef and CTERelationRef IDs
    omit c5a9bc79071d [SPARK-50761][INFRA] Make `pyspark-pandas` can be tested alone
    omit 8bd7789872b4 [SPARK-50722][SQL] Detect self-contained WITH nodes
    omit 96da1abc0f3a [MINOR] Remove TODO comment about `ExecutorAdded` message
    omit 0b443f4a5041 [SPARK-50755][SQL] Pretty plan display for InsertIntoHiveTable
    omit cfb2e4054f77 [SPARK-50756][SQL] Use error class for exceptions in SparkConf.validateSettings
    omit ab5461cf54b0 [SPARK-50076][FOLLOW-UP] Fix multiple log key names
    omit be0ae1388cd8 [SPARK-50541][SQL][TESTS][FOLLOWUP] Use `SPARK_VERSION` instead of hard-coded version strings
    omit 194aa18821c0 [SPARK-49868][DOC][FOLLOWUP] Update docs for executor failure tracking configrations
    omit bba8cf48d14f [SPARK-50734][SQL] Add catalog API for creating and registering SQL UDFs
    omit 204c67298117 [MINOR][TESTS] Fix comments on `runIfTestsEnabled` methods
    omit 36d23eff4b4c [SPARK-50541] Describe Table As JSON
    omit 22cbb9694ca5 [SPARK-50746][SQL] Replace Either with VariantPathSegment
    omit b3182e5521b9 [SPARK-50744][SQL] Add a test case for view/CTE name resolution precedence
    omit 0467aca97120 [SPARK-49783][YARN] Fix resource leak of yarn allocator
    omit e68b98a35cfe [MINOR][PYTHON][TESTS] Restore daily coverage build
    omit 5be9587a3ae5 [SPARK-48730][SQL] Implement CreateSQLFunctionCommand for SQL Scalar and Table Functions
    omit e24e7b4640b5 [SPARK-50392][PYTHON] DataFrame conversion to table argument in Spark Classic
    omit 9820108dbc1e [MINOR][DOCS] Use SBT slash syntax in `ExternalAppendOnlyUnsafeRowArrayBenchmark` doc
    omit 1419f38bd50e [SPARK-39903][SQL][TESTS] Reenable TPCDS q72 of `TPCDSQuerySuite` in GitHub Actions
    omit 4aa389fdb16a [SPARK-50745][SQL][TESTS] Use `eventually` in `BufferHolderSparkSubmitSuite`
    omit 32e3df7fea18 [SPARK-50742][CORE] Remove `spark.hadoop.fs.s3a.connection.establish.timeout` setting
    omit 91bc50930dc9 [SPARK-50741][BUILD] Upgrade `zstd-jni` to 1.5.6-9
    omit 2f0340701f5c [SPARK-50243][SQL][TESTS][FOLLOWUP] Remove `ARTIFACTS_SESSION_ISOLATION_ALWAYS_APPLY_CLASSLOADER` setting from `AdaptiveQueryExecSuite`
    omit d44fd2b57110 [SPARK-50694][SQL] Support renames in subqueries
    omit fbb4502070da [SPARK-50716][CORE][FOLLOWUP] Fix the scenario in `JavaUtils#deleteRecursivelyUsingJavaIO` where `BasicFileAttributes` cannot be read
    omit 47825e502a22 [SPARK-50736][INFRA] Enable testing module `pyspark-logger`
    omit 0d3f4cddf0f5 [SPARK-50708][CONNECT] Delete Artifact resources on GC of `ArtifactManager` instance
    omit 51501dfaa7ae [SPARK-50596][PYTHON][FOLLOW-UP] Pin Py4J version in dev/requirements.txt
    omit a1a2251ed7e4 [SPARK-50738][PYTHON] Upgrade black to 23.12.1
    omit b3d2f28962db [SPARK-50671][BUILD] Error compiling spark-protobuf module using user-defined protoc
    omit 459483acb45e [SPARK-50357][PYTHON] Support Interrupt(Tag|All) APIs for PySpark
    omit 56284bfb1e89 [SPARK-48459][FOLLOWUP][MINOR] Cleanup unused global variable
    omit 54b4c26504c3 [SPARK-50732][SQL][TESTS] Mark `DataFrameAggregateSuite` and its derived classes as `SlowSQLTest`
    omit 2cc45c2d0273 [SPARK-50733][K8S][INFRA] Update K8s IT CI to use K8s 1.32
    omit 7ebd39500086 [SPARK-50729][SQL][TESTS] Remove `DebugFilesystem` from `TPCDS(Collation)?QueryTestSuite`
    omit aed5d110c2aa [SPARK-50731][SQL][TESTS] Mark `*TransformWithStateSuite` as `SlowSQLTest`
    omit 1793a20dbdcb [SPARK-50700][SQL] `spark.sql.catalog.spark_catalog` supports `builtin` magic value
    omit 3ecb2908dea6 [SPARK-50614][SQL] Revert API change for shredding
    omit b613b228892d [SPARK-50703][PYTHON] Refine the docstring of regexp_replace, regexp_substr, and regexp_instr
    omit 24c9247208c2 [SPARK-50730][SQL][TESTS] Skip TPC-DS collation benchmark in branch-3.5
    omit 91f3fdd25852 [SPARK-50578][PYTHON][SS][FOLLOWUP] Remove unnecessary if block
    omit 0d0fa86a0311 [SPARK-50578][PYTHON][SS][FOLLOWUP] Change to use `Thread.interrupt` instead of `Thread.stop` to interrupt the execution of TransformWithStateInPandasPythonPreInitRunner#daemonThread
    omit 98dc76364ffa [SPARK-50716][CORE] Fix the cleanup logic for symbolic links in `JavaUtils.deleteRecursivelyUsingJavaIO` method
    omit dccb129ea045 [SPARK-50493][SPARK-37687][BUILD] Migrate kubernetes-client from `6.x` to `7.x`
    omit a59941a8ff53 [SPARK-50678][BUILD] Clean up outdated style check rules related to Guava
    omit 580b3c0158e4 [SPARK-50636][SQL] Extending CTESubstitution.scala to make it aware of recursion
    omit b210f422b007 [SPARK-49087][SQL][CONNECT] Distinguish UnresolvedFunction calling internal functions
    omit 2acc969a4891 [SPARK-50717][PS][DOCS] Update doc and add examples for `from_pandas`
    omit a3a8d1ca668a [SPARK-50715][PYTHON][CONNECT] `SparkSession.Builder` sets the configs in batch
    omit fa0c9954aa4d [SPARK-50614][SQL] Fix shredding debug message
    omit 032e53180d37 [SPARK-50311][TESTS][FOLLOWUP] Fix test name properly
    omit 9087d84b5127 [SPARK-50710][CONNECT] Add support for optional client reconnection to sessions after release
    omit c374c00432bd [SPARK-50712][INFRA][PS][TESTS] Add a daily build for Pandas API on Spark with old dependencies
    omit 3c0824d92068 [SPARK-50711][PS][DOCS] Upgrade the minimum version of Pandas in PS to 2.2.0
    omit 5c634845f084 [SPARK-50614][SQL] Add Variant shredding support for Parquet
    omit 492fcd83258b [SPARK-50683][SQL] Inline the common expression in With if used once
    omit 721a41794e11 [SPARK-50702][PYTHON] Refine the docstring of regexp_count, regexp_extract and regexp_extract_all
    omit 3f333a0820a9 [SPARK-50642][CONNECT][SS] Fix the state schema for FlatMapGroupsWithState in spark connect when there is no initial state
    omit 1c79b54ac934 [SPARK-49491][SQL] Replace AnyRefMap with HashMap
    omit 5334494b00fc [SPARK-50648][CORE] Cleanup zombie tasks in non-running stages when the job is cancelled
    omit e1fb18d7a3d4 [SPARK-50692] Add RPAD pushdown support
    omit 8a09817bf948 [SPARK-50701][PYTHON] Make plotting require the minimum plotly version
    omit 5ef556bc64e4 [SPARK-50706][PYTHON][TESTS] Skip test_value_state_ttl_expiration in Coverage build
    omit 6099de7f5995 [SPARK-50697][SQL] Enable tail-recursion wherever possible
    omit c4145db09ffe [SPARK-50689][SQL][FOLLOWUP] Enforce deterministic ordering in LCA aggregate lists
    omit fd8230b84975 [SPARK-49229][CONNECT] Deduplicate Scala UDF handling in the SparkConnectPlanner
    omit 4c39d6fa648a [SPARK-50699][PYTHON] Parse and generate DDL string with a specified session
    omit 43a9b88991b2 [SPARK-50693][CONNECT] The inputs for TypedScalaUdf should be analyzed
    omit 97ee25a7af96 [SPARK-50691][SQL][FOLLOWUP] Use UnsafeProjection for LocalRelation rows instead of ComparableLocalRelation
    omit 7d5aaaad61f2 [SPARK-50688][SQL] Eliminate ambiguity for rowTag missing in xml write path
    omit 73a2ebb901e5 [SPARK-50696][PYTHON] Optimize Py4J call for DDL parse method
    omit 31cb8113f12c [SPARK-50632][BUILD] Upgrade tink to 1.16.0
    omit 89fb67f7e880 [SPARK-50676][SQL] Remove unused `private lazy val mapValueContainsNull` from `ElementAt`
    omit c1e51f225635 [SPARK-50690][SQL] Fix discrepancy in DESCRIBE TABLE view query output columns quoting
    omit 51b011f64273 [SPARK-50661][CONNECT][SS] Fix Spark Connect Scala foreachBatch impl. to support Dataset[T].
    omit af53ee455e21 [SPARK-50689][SQL] Enforce deterministic ordering in LCA project lists
    omit 939129ec01af [SPARK-50674][PYTHON] Fix check for ‘terminate’ method existence in UDTF evaluation
    omit 003be89e2af8 [SPARK-50687][PYTHON] Optimize the logic to get stack traces for DataFrameQueryContext
    omit 2d320aaa420f [SPARK-50685][PYTHON] Improve Py4J performance by leveraging getattr
    omit 9297c5d6e90a [SPARK-50684][PYTHON] Improve Py4J performance in DataFrameQueryContext
    omit b309db092cbb [SPARK-50682][SQL] Inner Alias should be canonicalized
    omit a0539bf440fb [SPARK-50310][CONNECT][PYTHON][FOLLOW-UP] Delay is_debugging_enabled call after modules are initialized
    omit 1a1fcd61322b Revert "[SPARK-50310][CONNECT][PYTHON] Call `with_origin_to_class` when the `Column` initializing"
    omit 59fb88733433 [MINOR][PYTHON] Leverage functools.cached_property in `DataFrame`
    omit f72ff1b096ed [SPARK-50681][PYTHON][CONNECT] Cache the parsed schema for MapInXXX and ApplyInXXX
    omit 2372bc029bcf [SPARK-50679][SQL] Duplicated common expressions in different With should be projected only once
    omit b8a8e0d526e0 [MINOR][PYTHON] Leverage functools.cached_property in `SparkSession`
    omit 789702b62988 [MINOR][PYTHON] Leverage functools.cached_property in `ImageSchema`
    omit 783b3e351d51 [SPARK-50622][SS][MINOR] RocksDB Refactor
    omit 20e4508d38f6 [SPARK-50573][SS] Adding State Schema ID to State Rows to schema evolution
    omit 928655beb9bf [SPARK-50677][BUILD] Upgrade jupiter-interface to 0.13.3 and Junit5 to 5.11.4
    omit f3426b7b45dc [SPARK-50669][SQL] Change the signature of TimestampAdd expression
    omit 94f9bb0b61a4 [SPARK-50673][ML] Avoid traversing model coefficients twice in `Word2VecModel` constructor
    omit c92021091502 [SPARK-50578][PYTHON][SS] Add support for new version of state metadata for TransformWithStateInPandas
    omit aac494e74c60 [SPARK-50134][SPARK-50130][SQL][CONNECT] Support DataFrame API for SCALAR and EXISTS subqueries in Spark Connect
    omit d3022e99ce08 [SPARK-50672][PYTHON][TESTS] Make `openpyxl` optional in PySpark Tests
    omit 92948e73713f [SPARK-50675][SQL] Table and view level collations support
    omit 7a4114cdf994 [SPARK-50644][FOLLOWUP][SQL] Fix scalar cast in the shredded reader
    omit ac91a7d3eac9 [SPARK-50608][SQL][DOCS] Fix malformed configuration page caused by unclosed tags
    omit 38c6ef456ed6 [SPARK-50529][SQL] Change char/varchar behavior under the `spark.sql.preserveCharVarcharTypeInfo` config
    omit 5c075c3e577f [SPARK-50667][PYTHON][TESTS] Make `jinja2` optional in PySpark Tests
    omit 2475b35a2ac7 [SPARK-50665][SQL] Substitute LocalRelation with ComparableLocalRelation in NormalizePlan
    omit a483dfd86018 [SPARK-50650][SQL] Improve logging in single-pass Analyzer
    omit be2da526b482 [SPARK-49632][SQL][FOLLOW-UP] Fix suggestion for `to_date` function
    omit 4ad7f3d1f3e5 [SPARK-50647][INFRA] Add a daily build for PySpark with old dependencies
    omit 9c9bdab64a4c [SPARK-50657][PYTHON] Upgrade the minimum version of `pyarrow` to 11.0.0
    omit ef4be07fdad9 [SPARK-50220][PYTHON] Support listagg in PySpark
    omit 112276d93c0e [SPARK-50360][SS][FOLLOWUP][TESTS] Changelog reader test improvements
    omit 6eb556fa9921 [SPARK-50409][SQL] Fix set statement to ignore `;` at the end of `SET;`, `SET -v;` and `SET key;`
    omit 241454253a73 [SPARK-50590][INFRA][FOLLOW-UP] Further skip unnecessary image build by the job modules
    omit 062bc4c25844 [SPARK-50659][SQL] Refactor Union output computation out to reuse it in the single-pass Analyzer
    omit 495e24830ff2 [MINOR][SQL][DOCS] Fix spacing with SQL configuration documentation
    omit f9e117e01672 [SPARK-50659][SQL] Move Union-related errors to QueryCompilationErrors
    omit 2c1c4d2614ae [SPARK-50644][SQL] Read variant struct in Parquet reader
    omit 202b42e57fde [SPARK-50649] Fix inconsistencies with casting between different collations
    omit 50d49ee2df5c [MINOR][INFRA] Skip step `List Python packages` when `PYTHON_TO_TEST` is not specified
    omit debae7173e87 [SPARK-50651][SQL][DOCS] Add note about octal representation for characters
    omit a30a3fd0e74d [SPARK-49530][PYTHON] Support pie subplots in pyspark plotting
    omit 7cd5c4a1d1eb [SPARK-50641][SQL] Move `GetJsonObjectEvaluator` to `JsonExpressionEvalUtils`
    omit 876450c41300 [SPARK-50646][PYTHON][DOCS] Document explicit style of pyspark plotting
    omit 08675b169270 [SPARK-50630][SQL] Fix GROUP BY ordinal support for pipe SQL AGGREGATE operators
    omit ab95c4e05fa9 [SPARK-50527][INFRA][FOLLOW-UP] Python 3.12 image clean up
    omit f8fd398439d0 [SPARK-50645][INFRA] Make more daily builds able to manually trigger
    omit f6f6dae726e5 [SPARK-50590][INFRA] Skip uncessary image build and push
    omit 5ac42e2fe122 [SPARK-50534][SPARK-50535][TEST][CONNECT] Fix sporadic test failures
    omit 827d2a016c54 [SPARK-50638][SQL] Refactor the view resolution into the separate file to reuse it in the single-pass Analyzer
    omit 482a27c82fba [SPARK-50637][SQL] Fix code style for the single-pass Analyzer
    omit a2e3188b4997 [SPARK-50640][CORE][TESTS] Update `ChecksumBenchmark` by removing `PureJavaCrc32C` and setting `Adler32` as a baseline
    omit bccdf1ffd467 [SPARK-50483][SPARK-50545][DOC][FOLLOWUP] Mention behavior changes in migration guide
    omit 629fe3f42173 [SPARK-50370][SQL] Codegen Support for `json_tuple`
    omit 78592a070c27 [SPARK-50615][SQL] Push variant into scan
    omit 939f3dfc0551 [SPARK-50621][PYTHON] Upgrade Cloudpickle to 3.1.0
    omit d499400e8cd8 [SPARK-50243][SQL][CONNECT] Cached classloader for ArtifactManager
    omit 09d93ece2cf7 [SPARK-50602][SQL] Fix transpose to show a proper error message when invalid index columns are specified
    omit f9aaeb4c1742 [SPARK-50586][BUILD] Use CommonJS format for ESLint configuration file
    omit f593cda97ed2 [SPARK-50612][SQL] Normalize ordering in the project list of an inner Project
    omit f4c88ca597ba [SPARK-50558][SQL] Add configurable logging limits for the number of elements in InSet and In
    omit 3a61eeff340a [SPARK-50619][SQL] Refactor VariantGet.cast to pack the cast arguments
    omit 7f6d554a4933 [SPARK-49436][CONNECT][SQL] Common interface for SQLContext
    omit 2cb66e6a86ee [SPARK-49632][SQL] Remove the ANSI config suggestion in CANNOT_PARSE_TIMESTAMP
    omit 2b1369b4abed [SPARK-50499][PYTHON] Expose metrics from BasePythonRunner
    omit 332efb2eabb2 [SPARK-49636][SQL] Remove the ANSI config suggestion in INVALID_ARRAY_INDEX and INVALID_ARRAY_INDEX_IN_ELEMENT_AT
    omit 788fa5ad1fe8 [SPARK-50609][INFRA] Eliminate warnings in docker image building
    omit 0dd90d99be2c [SPARK-50611][SQL] Improve HybridAnalyzer so it throws ExplicitlyUnsupportedResolverFeature in specific cases
    omit 0e1fa6facd4d [SPARK-49661][SQL] Removing redudant checks of collation in binary collation hashing expression
    omit 5ef99bda9863 [SPARK-50606][CONNECT] Fix NPE on uninitiated SessionHolder
    omit 76672cbf366c [SPARK-50472][SQL] Introduce initial implementation of the single-pass Analyzer
    omit 9c9bbf6a6d60 [SPARK-50604][SQL][TESTS] Extract MariaDBDatabaseOnDocker and upgrade `Mariadb` docker image version
    omit cb84939e02ff [SPARK-50596][PYTHON] Upgrade Py4J from 0.10.9.7 to 0.10.9.8
    omit 62d49b35ab24 [SPARK-50504][SQL] Enable SQL pipe syntax by default
    omit 229118ca7a12 [SPARK-50599][SQL] Create the DataEncoder trait that allows for Avro and UnsafeRow encoding
    omit 2b41131d7fa6 [MINOR][SS] Minor update to watermark propagation comments
    omit accde839ead8 [SPARK-50598][SQL] An initial, no-op PR which adds new parameters to already existing classes UnresolvedWith, CTERelationRef and CTERelationDef to enable later implementation of recursive CTEs
    omit 63c7ca4df297 [SPARK-50597][SQL] Refactor batch construction in Optimizer.scala and SparkOptimizer.scala
    omit 79026ad70380 [SPARK-50588][BUILD] `build-docs` skips building R docs on host when `SKIP_RDOC=1`
    omit 4c0f7b581243 [SPARK-50594][BUILD][INFRA] Align the gRPC-related Python package version and pin protobuf version in `page.yml` and `docs/Dockerfile`
    omit 1c3d580eaa00 [SPARK-50592][INFRA] Make 3.5 daily build able to manually trigger
    omit 576caec1da85 [SPARK-50580][BUILD] Upgrade log4j2 to 2.24.3
    omit 868c58730e56 [SPARK-50583][INFRA][FOLLOW-UP] Fix 3.5 daily build
    omit 44ab3492455e [SPARK-50581][SQL] fix support for UDAF in Dataset.observe()
    omit 0faf9d5da06d [SPARK-50579][SQL] Fix `truncatedString`
    omit d293ba66a0b1 [SPARK-50511][PYTHON] Avoid wrapping Python data source error messages
    omit 3f8e3959108d [SPARK-50583][INFRA] Apply Python 3.11 image in PR build
    omit 15dcd21323fa [SPARK-50360][SS][FOLLOWUP][MINOR] Make readVersion lazy value
    omit ef37f9a8e202 [SPARK-50568][SS][TESTS] Fix `KafkaMicroBatchSourceSuite` to cover `KafkaOffsetReaderConsumer`
    omit a6f82c64e3e4 [SPARK-49461][SS][TESTS][FOLLOWUP] Move related resource files to correct place
    omit 279d5eefef32 [SPARK-50564][PYTHON] Upgrade `protobuf` Python package to 5.29.1
    omit e7a2e4b8f4cf [MINOR][DOCS] Modify a link in running-on-yarn.md
    omit 7d7711de86c9 [SPARK-50576][BUILD] Upgrade `common-text` to `1.13.0`
    omit 3df17696258f [SPARK-50574][DOCS] Upgrade `rexml` to 3.3.9
    omit 769e569b7916 [SPARK-49954][SQL][FOLLOWUP] Move states to SchemaOfJsonEvaluator
    omit d2965ae99aa7 [SPARK-49839][SQL] SPJ: Skip shuffles if possible for sorts
    omit 976192af22ba [SPARK-50559][SQL] Store Except, Intersect and Union's outputs as lazy vals
    omit 2b9eb08a019f [SPARK-50515][CORE] Add read-only interface to `SparkConf`
    omit f8de6c7b74b8 [SPARK-50565][SS][TESTS] Add transformWithState correctness test
    omit 3db46bf1fb48 [SPARK-50526][SS] Add store encoding format conf into offset log and block non supported stateful operators from using avro
    omit 5538d8536e9d [SPARK-50540][PYTHON][SS] Fix string schema for StatefulProcessorHandle
    omit 3e7b614a2da6 [SPARK_50076] Fix logkeys
    omit c1a9fc1026de [SPARK-45891][SQL][TESTS] Make VariantSuite test agnostic of the stableDerivedColumnAlias config
    omit ebd6b7c5aacd [SPARK-50546][SQL] Add subquery cast support to collation type coercion
    omit 42ce6044e15b [SPARK-50338][CORE] Make LazyTry exceptions less verbose
    omit f5740d0b94e5 [SPARK-50571][INFRA] Apply Python 3.11 image in RocksDB as UI Backend daily build
    omit fbc061d0c856 [SPARK-48898][SQL] Set nullability correctly in the Variant schema
    omit 3362ec8c3563 [SPARK-50567][INFRA] Apply Python 3.11 image in No-ANSI daily build
    omit d1db5104d080 [SPARK-50563][BUILD] Upgrade `protobuf-java` to 4.29.1
    omit 885915f54537 [SPARK-50566][MINOR][SS] Fix code style violation for RocksDB file
    omit 819bac990314 [SPARK-50157][SQL] Using SQLConf provided by SparkSession first
    omit 98cef08cae07 [SPARK-50562][INFRA] Apply Python 3.11 image in Java 21 daily build
    omit 429402b7c3d8 [SPARK-49883][SS] State Store Checkpoint Structure V2 Integration with RocksDB and RocksDBFileManager
    omit df08177de2fd [SPARK-48416][SQL] Support nested correlated With expression
    omit f979bc8d37d2 [SPARK-50554][INFRA] Add a separate docker file for Python 3.11 daily coverage build
    omit b5d195a5cd1b [SPARK-48344][SQL] Add Frames and Scopes to support Exception Handlers and Local Variables
    omit e4be5e67fefc [SPARK-50552][INFRA] Add a separate docker file for PyPy 3.10 daily build
    omit 032623f62c07 [SPARK-50553][CONNECT] Throw `InvalidPlanInput` for invalid plan message
    omit d84b2d4565c5 [SPARK-50428][SS][PYTHON] Support TransformWithStateInPandas in batch queries
    omit 1f6cb60e3c78 [SPARK-50544][PYTHON][CONNECT] Implement `StructType.toDDL`
    omit bfc5c22cb934 [SPARK-50517][PYTHON][TESTS][FOLLOW-UP] Add refactored package into pure Python test
    omit d464e85b0a3c [SPARK-50549][DOCS] Use `rouge` `4.5.x` by removing the upper bound `3.x`
    omit 48efe3f57f9f [SPARK-50134][SPARK-50132][SQL][CONNECT][PYTHON] Support DataFrame API for Lateral Join in Spark Connect
    omit b2c8b3069ef4 Revert "[SPARK-48898][SQL] Set nullability correctly in the Variant schema"
    omit 3bb9a72c8691 [SPARK-50545][CORE][SQL] `AccessControlException` should be thrown even if `ignoreCorruptFiles` is enabled
    omit 9394b35ed7bb [SPARK-48898][SQL] Set nullability correctly in the Variant schema
    omit 2fa72d251f6f [SPARK-50542][INFRA] Add a separate docker file for python 3.13 daily build
    omit d268e0c4ac64 [SPARK-46934][SQL][FOLLOWUP] Read/write roundtrip for struct type with special characters with HMS - a backward compatible approach
    omit 2f5728f3ea75 [SPARK-50543][BUILD] Fix  log printing in `dev/check-protos.py`
    omit 0f27d73b6c8b [SPARK-49565][SQL] Add SQL pipe syntax for the FROM operator
    omit 58c77bab9317 [SPARK-49349][SQL][FOLLOWUP] Rename isContainsUnsupportedLCA function
    omit 6cdc96f7399e [SPARK-50517][PYTHON][TESTS][FOLLOW-UP] Add refactored package into pure Python test
    omit f903efb9e49a [SPARK-50537][CONNECT][PYTHON] Fix compression option being overwritten in `df.write.parquet`
    omit 2093bae154e0 [SPARK-50536][CORE] Log downloaded archive file sizes in `SparkContext` and `Executor`
    omit 18f0e231335a [SPARK-45891][SQL] Rebuild variant binary from shredded data
    omit e70f8abacec4 [SPARK-50491][SQL] Fix bug where empty BEGIN END blocks throw an error
    omit 559fda794d1d [SPARK-50530][SQL] Fix bad implicit string type context calculation
    omit faef3fa9b2c3 [SPARK-50527][INFRA] Add a separate docker file for python 3.12 daily build
    omit 12967fe02e8a [SPARK-49349][SQL] Improve error message for LCA with Generate
    omit bac386dcfc6b [SPARK-50528][CONNECT] Move `InvalidCommandInput` to common module
    omit 03c57995beb1 [SPARK-50513][SS][SQL] Split EncoderImplicits from SQLImplicits and provide helper object within StatefulProcessor to access underlying SQL Encoder related implicit functions
    omit 02ebf128247b [SPARK-50460][PYTHON][CONNECT] Generalize and simplify Connect exception handling
    omit 5f34af975f35 [SPARK-50477][INFRA][FOLLOW-UP] Python 3.9 testing image clean up
    omit a89fcfca29f6 [SPARK-50507][PYTHON][TESTS][FOLLOW-UP] Add refactored package into pure Python test
    omit 156cf1608273 [SPARK-50524][SQL] Lower `RowBasedKeyValueBatch.spill` warning message to debug level
    omit 2cb818350de6 [SPARK-50480][SQL] Extend CharType and VarcharType from StringType
    omit 0830a190b993 [SPARK-50329][SQL] fix InSet$toString
    omit 290b4b31bae2 [SPARK-49461][SS][TESTS][FOLLOWUP] Add compatibility test for new commit log
    omit 74eeceda08e0 [SPARK-50523][SQL] Fix casts on complex types in collation coercion
    omit 32431cf35975 [SPARK-50063][SQL][CONNECT] Add support for Variant in the Spark Connect Scala client
    omit 88c38138f155 [SPARK-50503][SQL] Prohibit partitioning by Variant data
    omit 88102d35d1f8 [SPARK-50497][SQL] Fail queries with proper message if MultiAlias contains non-generator function
    omit 85d92d7c3a6a [SPARK-50517][PYTHON][TESTS] Group arrow function related tests
    omit 6c2e87aae659 [SPARK-50507][PYTHON][TESTS][FOLLOW-UP] Add refactored package into pure Python test
    omit bb17665955ad [SPARK-49249][SQL][FOLLOWUP] Rename `spark.sql.artifact.isolation.(always.apply.classloader -> alwaysApplyClassloader)`
    omit fff679307f56 [SPARK-50516][SS][MINOR] Fix the init state related test to use StreamManualClock
    omit 2fea84e1334c [SPARK-48426][DOCS][FOLLOWUP] Add `Operators` page to `sql-ref.md`
    omit c1267d697490 [SPARK-50512][SQL][DOCS] Fix `CREATE TABLE` syntax in `sql-pipe-syntax.md`
    omit 28766d4120f4 [SPARK-50514][DOCS] Add `IDENTIFIER clause` page to `menu-sql.yaml`
    omit be0780b8f931 [SPARK-50350][SQL] Avro: add new function `schema_of_avro` (`scala` side)
    omit 934a3878f733 [SPARK-50483][CORE][SQL] BlockMissingException should be thrown even if ignoreCorruptFiles is enabled
    omit 1d6932c30b73 [SPARK-50478][SQL] Fix StringType matching
    omit ede9cfc92f0f [SPARK-50494][INFRA] Add a separate docker file for python 3.10 daily build
    omit c149942f0f96 [SPARK-50449][SQL] Fix SQL Scripting grammar allowing empty bodies for loops, IF and CASE
    omit 851f5f2ff905 [SPARK-50492][SS] Fix java.util.NoSuchElementException when event time column is dropped after dropDuplicatesWithinWatermark
    omit b1c118f48ea4 [SPARK-50507][PYTHON][TESTS] Group pandas function related tests
    omit ecd1911a59b2 [SPARK-50506][DOCS] Codify Spark Standalone documentation consistently
    omit d67ca731f19c [SPARK-50132][SQL][PYTHON] Add DataFrame API for Lateral Joins
    omit a435b2e634bd [SPARK-50489][SQL][PYTHON][FOLLOW-UP] Add applyInArrow in `DeduplicateRelations#collectConflictPlans`
    omit fc69194fc032 [SPARK-50310][CONNECT][PYTHON] Call `with_origin_to_class` when the `Column` initializing
    omit 21451fb312fe [SPARK-50505][DOCS] Fix `spark.storage.replication.proactive` default value documentation
    omit 6add9c89855f [SPARK-50501][BUILD] Update cross-spawn to surpress a warning in lint
    omit ee8db4ef5d00 [SPARK-50343][SPARK-50344][SQL] Add SQL pipe syntax for the DROP and AS operators
    omit faa2b0472693 [SPARK-50421][CORE] Fix executor related memory config incorrect when multiple resource profiles worked
    omit 36285956ed2b [SPARK-50498][PYTHON] Avoid unnecessary py4j call in `listFunctions`
    omit 05703908935b [SPARK-50495][K8S][INFRA][DOCS] Upgrade Volcano to 1.10.0
    omit af4f37c94ce4 [SPARK-50339][SPARK-50360][SS] Enable changelog to store lineage information
    omit 7278bc70b126 [SPARK-50489][SQL][PYTHON] Fix self-join after `applyInArrow`
    omit fe904e6973b7 [SPARK-49709][CONNECT][SQL] Support ConfigEntry in the RuntimeConfig interface
    omit 424839745a94 [SPARK-49695][SQL] Postgres fix xor push-down
    omit f1eecd357861 [SPARK-50485][SQL] Unwrap SparkThrowable in (Unchecked)ExecutionException thrown by tableRelationCache
    omit 812a9ad3ea92 [MINOR] Fix some typos
    omit 10e0b619092b [SPARK-49670][SQL] Enable trim collation for all passthrough expressions
    omit 74c3757513c9 [MINOR][DOCS] Add a migration guide for encode/decode unmappable characters
    omit 3d063a01d7c2 [SPARK-50487][DOCS] Update broken jira link
    omit 45da6f66d155 [SPARK-50477][INFRA] Add a separate docker file for python 3.9 daily build
    omit 5fc6b713874a [SPARK-50405][SQL] Handle collation type coercion of complex data types properly
    omit 784a97b3ee8c [MINOR][INFRA] Update labeler
    omit 0e45e2179962 [SPARK-50486][PYTHON][DOCS] Refine the docstring for string functions - part 2
    omit 13315eeec07e [SPARK-50481][CORE] Improve `SortShuffleManager.unregisterShuffle` to skip checksum file logic if checksum is disabled
    omit ecc33d26e8fa [SPARK-50482][CORE] Deprecated no-op `spark.shuffle.spill` config
    omit d0dbc6c5e5c4 [SPARK-50470][SQL] Block usage of collations for map keys
    omit 7b974ca75896 [SPARK-50463][SQL] Fix `ConstantColumnVector` with Columnar to Row conversion
    omit 6cd13344bffd [SPARK-50425][BUILD] Bump Apache Parquet to 1.15.0
    omit d427aa3c6f47 [SPARK-50473][SQL] Simplify classic Column handling
    omit 98e94af58f16 [SPARK-48898][SQL] Fix Variant shredding bug
    omit f572ad1a722a [SPARK-50474][BUILD] Upgrade checkstyle to 10.20.2
    omit 38ab95bed4eb [SPARK-50467][PYTHON] Add `__all__` for builtin functions
    omit 4abaab3ffeba [SPARK-50430][CORE][FOLLOW-UP] Keep the logic of manual putting key and values in Properties
    omit 2041519cd9d9 [SPARK-47993][DOCS][FOLLOWUP] Update RDD programing guide to Python 3.9+
    omit 75397c9207b7 [MINOR][INFRA] Add a space in the JIRA user assignment message
    omit 33f248ce2364 [SPARK-50458][CORE][SQL] Proper error handling for unsupported file system when reading files
    omit 758b5c9ba941 [SPARK-50466][PYTHON] Refine the docstring for string functions - part 1
    omit 31abad98972b [SPARK-50471][PYTHON] Support Arrow-based Python Data Source Writer
    omit d0e2c06e874a [SPARK-50372][CONNECT][SQL] Make all DF execution path collect observed metrics
    omit 6c84f158469a [SPARK-50468][BUILD] Upgrade Guava to 33.3.1-jre
    omit be12eb732b0b [SPARK-50465][PYTHON][TESTS] Use assertDataFrameEqual in pyspark.sql.tests.test_group and test_readwriter
    omit 98350095d33c [SPARK-50452][BUILD] Upgrade jackson to 2.18.2
    omit ae4625c9c375 [SPARK-50433][DOCS][TESTS] Fix confguring log4j2 guide docs for Spark on YARN and UT
    omit f382cdf89536 [SPARK-50432][BUILD] Remove workaround for THRIFT-4805
    omit 171e2ce7d117 [SPARK-50373] Prohibit Variant from set operations
    omit 05728e4ff64e [SPARK-49294][UI] Add width attribute for shuffle-write-time checkbox
    omit e7071c0237da [SPARK-50435][PYTHON][TESTS] Use assertDataFrameEqual in pyspark.sql.tests.test_functions
    omit dc73342db941 [SPARK-50436][PYTHON][TESTS] Use assertDataFrameEqual in pyspark.sql.tests.test_udf
    omit 7d46fdb5cea2 [SPARK-48148][FOLLOWUP] Fix JSON parser feature flag
    omit b45e3c01217a [MINOR] Use putAll to populate Properties
    omit faf74adad0f3 [SPARK-50032][SQL] Allow use of fully qualified collation name
    omit 3fab712f69f0 [SPARK-50441][SQL] Fix parametrized identifiers not working when referencing CTEs
    omit 4b97e11b53c2 [SPARK-42746][SQL] Implement LISTAGG function
    omit d5b534d18c06 [SPARK-50295][DOCS][FOLLOWUP] Document `build-docs` in `docs/README.md`
    omit 547661002fb8 [SPARK-50311][PYTHON][FOLLOWUP] Remove @remote_only from supported APIs
    omit cd687ff7d95f [SPARK-50310][PYTHON] Add a flag to disable DataFrameQueryContext for PySpark
    omit aad357583c8c [SPARK-50453][BUILD] Upgrade Netty to 4.1.115
    omit 376bd4aec8de [SPARK-50333][SQL][FOLLOWUP] Codegen Support for `CsvToStructs`(`from_csv`) - remove Invoke
    omit 0c16e93dce06 [SPARK-50446][PYTHON] Concurrent level in Arrow-optimized Python UDF
    omit 3791de906f82 [SPARK-50444][BUILD] Upgrade dropwizard metrics to 4.2.29
    omit b45045e56f81 [SPARK-49992][SQL] Default collation resolution for DDL and DML queries
    omit aaf859053c7a [SPARK-50448][SQL][TESTS] Extract postgres image common conf as PostgresDatabaseOnDocker
    omit 2c2c0e055127 [SPARK-48356][SQL] Support for FOR statement
    omit 4eed18455332 [SPARK-50439][BUILD] Upgrade `Arrow` to 18.1.0
    omit 13c1da7aa91d [SPARK-50311][PYTHON] Support `(add|remove|get|clear)Tag(s)` APIs for PySpark
    omit b9bff4b145e5 [SPARK-50295][INFRA] Add a script to build docs with image
    omit e6252d6c6143 [SPARK-50194][SS][PYTHON] Integration of New Timer API and Initial State API with Timer
    omit bb994d14966e [SPARK-50440][SQL] Refactor AttributeSeq.resolveCandidates
    omit 2a61dd48a45b [SPARK-50445][PYTHON][DOCS] Refine docString of generator Functions
    omit 0c31f5a807e7 [SPARK-50443][SS] Fixing Maven build errors introduced by Guava cache in RocksDBStateStoreProvider
    omit 68be1da5ba8e [SPARK-49566][SQL] Add SQL pipe syntax for the SET operator
    omit ce57a3d13154 [SPARK-50067][SPARK-50066][SPARK-49954][SQL][FOLLOWUP] Make returnNullable as false
    omit 405644fc4ace [SPARK-50130][SQL][FOLLOWUP] Simplify the implementation of col.outer()
    omit 59a7ad46588f [SPARK-50437][SS] Reduce overhead of creating deserializers in TransformWithStateExec
    omit 6697725f1715 Revert "[MINOR][TESTS] Use SparkContext.setLogLevel in QuietTest"
    omit 1a502d32ef5a [SPARK-50285] Metrics for commits to StagedTable instances
    omit 6edcf435661e [SPARK-49966][SQL][FOLLOWUP] Codegen Support for `JsonToStructs`(`from_json`) - remove Invoke
    omit 0138019b5497 [SPARK-50426][PYTHON] Avoid static Python data source lookup when using builtin or Java data sources
    omit e03319fd9219 [SPARK-49676][SS][PYTHON] Add Support for Chaining of Operators in transformWithStateInPandas API
    omit 7614819884ca [SPARK-50430][CORE] Use the standard Properties.clone instead of manual clone
    omit 30cba12a51fc [SPARK-48344][SQL] Add SQL Scripting Execution Framework
    omit e55511c98e14 [SPARK-50388][PYTHON][TESTS][FOLLOW-UP] Move `have_flameprof` to `pyspark.testing.utils`
    omit 5425d453d244 [SPARK-50394][PYTHON][INFRA][FOLLOW-UP] Reduce parallelism further in Pure Python library builds
    omit 4ec9ebf243d0 [SPARK-50427][CONNECT][PYTHON] Expose configure_logging as a public API
    omit 02bfce6056ec [SPARK-50302][SS] Ensure secondary index sizes equal primary index sizes for TransformWithState stateful variables with TTL
    omit 7cbfc2cad078 [SPARK-50422][SQL] Make `Parameterized SQL queries` of `SparkSession.sql` API GA
    omit f7a264375890 [SPARK-50423][PYTHON][DOCS] Refine the docstring for misc and hash functions
    omit 9bab16a854d8 [MINOR][TESTS] Use SparkContext.setLogLevel in QuietTest
    omit 9054aa287601 [SPARK-50406][PYTHON][TESTS] Improve pyspark.sql.tests.test_udtf
    omit afb5d6f23abf [SPARK-50410][CONNECT] Refactor the `sql` function in `SparkSession` to eliminate duplicate code
    omit 69d433bcfd5a [SPARK-50387][SS] Update condition for timer expiry and relevant test
    omit 331d0bf30092 [SPARK-50017][SS] Support Avro encoding for TransformWithState operator
    omit f7122137006e [SPARK-50415][BUILD] Upgrade `zstd-jni` to 1.5.6-8
    omit 6667ba27cbad [SPARK-50412][BUILD] Rename `ojdbc11.version` to `ojdbc17.version`
    omit 214fd53af7fb [SPARK-45151][UI][FOLLOWUP] Add missing space before date value
    omit 5dc65a5416b5 [SPARK-50287][SQL] Merge options of table and relation when creating WriteBuilder in FileTable
    omit 976f8875edd4 [SPARK-50286][SQL] Correctly propagate SQL options to WriteBuilder
    omit da4bcb732faa [MINOR][PYTHON] Remove some unused third-party library imports
    omit 7b4922ea90d1 [SPARK-50408][PYTHON][DOCS] Refine the docstring for datetime functions - part 4
    omit 658879e0aeef [SPARK-50382][CONNECT] Add documentation for general information on application development with/extending Spark Connect
    omit bf893624161c [SPARK-50397][CORE] Remove deprecated `--ip` and `-i` arguments from `Master/Worker`
    omit 4f8d1f575e99 [SPARK-50394][PYTHON][INFRA][FOLLOW-UP] Reduce parallelism further in Pure Python library builds
    omit d9757b5653b9 [SPARK-50402][SQL][TESTS] Upgrade PostgreSQL docker image to 17.2
    omit 8887d53117ee [SPARK-50395][SQL] Fix malformed URI syntax in Windows
    omit 3cdebbeb8814 [SPARK-50400][BUILD] Upgrade log4j2 to 2.24.2
    omit d356b17f1dca [SPARK-50380][SQL][TESTS][FOLLOWUP] Enable ANSI for conditional branches with error expression test
    omit 656ece1adb0f [SPARK-50081][SQL] Codegen Support for `XPath*`(by Invoke & RuntimeReplaceable)
    omit 779a526a015d [SPARK-50396][PYTHON][DOCS] Refine the docstring for datetime functions - part 3
    omit 7ae2d81e9bc3 [MINOR][DOCS] Fix miss semicolon on insert example sql
    omit 5bdaa72801d6 [SPARK-50398][CORE] Use ExitCode `0` for `--help` usage in Spark scripts
    omit d5da49d56d7d Revert [SPARK-50230][SQL] Added logic to support reading unknown collation name as utf8_binary
    omit ad8250562cee [SPARK-50324][PYTHON][CONNECT] Make `createDataFrame` trigger `Config` RPC at most once
    omit 77055b812559 [SPARK-50385][CORE] Use class name prefix for REST Submission API thread pool
    omit f9d2f4226366 [SPARK-50394][PYTHON][INFRA] Reduce parallelism in Pure Python library builds
    omit 5e076efd7510 [SPARK-50390][PYTHON][DOCS] Refine the docstring for datetime functions - part 2
    omit de1c77b9da01 Revert "[SPARK-50291][PYTHON] Standardize verifySchema parameter of createDataFrame in Spark Classic"
    omit 8d7e3d4af618 Revert "[SPARK-50298][PYTHON][CONNECT] Implement verifySchema parameter of createDataFrame in Spark Connect"
    omit f5bb11cfb3ef [SPARK-50301][SS] Make TransformWithState metrics reflect their intuitive meanings
    omit ea222a3c72dd [SPARK-49566][SQL] Add SQL pipe syntax for the EXTEND operator
    omit 0fbd34c45b58 [SPARK-50377][SQL] Allow to evaluate foldable RuntimeReplaceable
    omit d9781d99ae87 [SPARK-50389][SQL][TESTS] Remove`sql-tests/results/try_arithmetic.sql.out.java21`
    omit 985f98873d51 [SPARK-48248][PYTHON][TESTS][FOLLOW-UP] Avoid duplicated test name for `test_infer_array_element_type_with_struct`
    omit d3119fac0a09 [SPARK-50378][SS] Add custom metric for tracking spent for proc initial state in transformWithState
    omit 190c50451333 [SPARK-50388][PYTHON][TESTS] Further centralize import checks
    omit 5d1f585eba69 [SPARK-50383][CORE] Support `Virtual Threads` in REST Submission API
    omit 6881ec0ebfab [SPARK-50366][SQL] Isolate user-defined tags on thread level for SparkSession in Classic
    omit d8a6075dd617 [SPARK-50380][SQL] ReorderAssociativeOperator should respect the contract in ConstantFolding
    omit 2d09ef22f8ad [SPARK-50381][CORE] Support `spark.master.rest.maxThreads`
    omit 2e1c3dc8004b [SPARK-50087] Robust handling of boolean expressions in CASE WHEN for MsSqlServer and future connectors
    omit 136c7221fa6a [SPARK-50334][SQL] Extract common logic for reading the descriptor of PB file
    omit 229b1b8654bb [SPARK-50375][BUILD] Upgrade `commons-io` to 2.18.0
    omit f2de888fc8a5 [MINOR][DOCS] Remove wrong and ambiguous default statement in datetime rebase mode settings
    omit cbb16b9633af [MINOR][DOCS] Fix miss semicolon on create table example sql
    omit fbf255ef7b01 [SPARK-50379][SQL] Fix DayTimeIntevalType handling in WindowExecBase
    omit b05ef451061f [SPARK-50175][SQL] Change collation precedence calculation
    omit 0f1e410a94d3 [SPARK-50016][SQL] Assign appropriate error condition for `_LEGACY_ERROR_TEMP_2067`: `UNSUPPORTED_PARTITION_TRANSFORM`
    omit ee21e6b07a0d [SPARK-50113][CONNECT][PYTHON][TESTS] Add `@remote_only` to check the APIs that only supported with Spark Connect
    omit 95faa0283e18 [SPARK-49490][SQL] Add benchmarks for initCap
    omit 3bc374d945ff [SPARK-50333][SQL] Codegen Support for `CsvToStructs` (by Invoke & RuntimeReplaceable)
    omit a40919912f5c [SPARK-50376][PYTHON][ML][TESTS] Centralize the dependency check in ML tests
    omit ad46db4ef671 [SPARK-50130][SQL][FOLLOWUP] Make Encoder generation lazy
    omit 30d0b011ef87 [SPARK-50364][SQL] Implement serialization for LocalDateTime type in Row.jsonValue
    omit 6ee53da5f356 [SPARK-50258][SQL] Fix output column order changed issue after AQE optimization
    omit 81a56df543f6 [SPARK-50362][PYTHON][ML] Skip `CrossValidatorTests` if `torch/torcheval` not installed
    omit 533b8cab827b [SPARK-50363][PYTHON][DOCS] Refine the docstring for datetime functions - part 1
    omit 23f276f64d6d [SPARK-50353][SQL] Refactor ResolveSQLOnFile
    omit 3151d9793521 [SPARK-49801][INFRA][FOLLOWUP] Sync pandas version in release environment
    omit 7a4f3c4c4938 [SPARK-50345][BUILD] Upgrade Kafka to 3.9.0
    omit 19b8250588bf [SPARK-50331][INFRA][FOLLOW-UP] Skip Torch/DeepSpeed tests in MacOS PySpark Daily test
    omit b582daca568f [MINOR][DOCS] Fix a HTML/Markdown syntax error in sql-migration-guide.md
    omit 0157778a3d4b [SPARK-50358][SQL][TESTS] Update postgres docker image to 17.1
    omit 2185f3cbb328 [SPARK-50359][PYTHON] Upgrade PyArrow to 18.0
    omit b7cf448aa409 [SPARK-49550][FOLLOWUP][SQL][DOC] Switch Hadoop to 3.4.1 in IsolatedClientLoader and docs
    omit 87917675c401 [SPARK-48344][SQL] Prepare SQL Scripting for addition of Execution Framework
    omit c149dcbc733f [SPARK-50352][PYTHON][DOCS] Refine docstrings for window/aggregation functions - part 3
    omit 37497e6ad836 [SPARK-50335][PYTHON][DOCS][FOLLOW-UP] Make percentile doctests more stable
    omit 19509d07983d Revert "[SPARK-49002][SQL] Consistently handle invalid locations in WAREHOUSE/SCHEMA/TABLE/PARTITION/DIRECTORY"
    omit f1b68d897e49 [SPARK-50315][SQL] Support custom metrics for V1Fallback writes
    omit 87a5b37ec3c4 [SPARK-50313][SQL][TESTS][FOLLOWUP] Regenerate golden files for Java 21
    omit b74aa8c2d6e7 [SPARK-50340][SQL] Unwrap UDT in INSERT input query
    omit 5a57efdcee9e [SPARK-50313][SQL][TESTS][FOLLOWUP] Restore some tests in *SQLQueryTestSuite for ANSI mode
    omit 6d4798103502 [SPARK-50331][INFRA] Add a daily test for PySpark on MacOS
    omit e1477a34d745 [SPARK-50298][PYTHON][CONNECT] Implement verifySchema parameter of createDataFrame in Spark Connect
    omit b61411d9eba4 [SPARK-50328][INFRA] Add a separate docker file for SparkR
    omit fa36e8b6080a [SPARK-50335][PYTHON][DOCS] Refine docstrings for window/aggregation functions - part 2
    omit 400a8d3797bd Revert "[SPARK-49787][SQL] Cast between UDT and other types"
    omit 05750def2da6 [MINOR][PYTHON][DOCS] Fix the type hint of `histogram_numeric`
    omit 8b2d0328c213 [SPARK-45265][SQL][BUILD][FOLLOWUP] Add `-Xss64m` for Maven testing of the `sql/hive` module
    omit a01856de2001 [SPARK-50330][SQL] Add hints to Sort and Window nodes
    omit b6265286eca9 [SPARK-50313][SQL][TESTS] Enable ANSI in SQL *SQLQueryTestSuite by default
    omit 281a8e18d416 [SPARK-50309][DOCS] Document `SQL Pipe` Syntax
    omit 007c31df6da7 [SPARK-50236][SQL] Assign appropriate error condition for `_LEGACY_ERROR_TEMP_1156`: `COLUMN_NOT_DEFINED_IN_TABLE `
    omit 11e47064d0c7 [SPARK-50320][CORE] Make `--remote` an official option by removing `experimental` warning
    omit 77e006f2a494 [SPARK-50327][SQL] Factor out function resolution to be reused in the single-pass Analyzer
    omit d3170028da1d [SPARK-50322][SQL] Fix parameterized identifier in a sub-query
    omit cc81ed0ac253 [SPARK-50325][SQL] Factor out alias resolution to be reused in the single-pass Analyzer
    omit cf90271af8b2 [MINOR] Fix code style for if/for/while statements
    omit 3f5e8468d204 [SPARK-50237][SQL] Assign appropriate error condition for `_LEGACY_ERROR_TEMP_2138-9`: `CIRCULAR_CLASS_REFERENCE`
    omit e615e3f51bf9 [SPARK-50049][SQL] Support custom driver metrics in writing to v2 table
    omit 3237885000e9 [SPARK-50312][SQL] SparkThriftServer createServer parameter passing error when kerberos is true
    omit c90efae2b4aa [SPARK-50318][SQL] Add IntervalUtils.makeYearMonthInterval to deduplicate code between interpreted and codegen
    omit e0a83f663896 [SPARK-50317][BUILD] Upgrade ORC to 2.0.3
    omit c2343f763251 [SPARK-45265][SQL] Support Hive 4.0 metastore
    omit 0aee601dcc98 [SPARK-50153][SQL] Add `name` to `RuleExecutor` to make printing `QueryExecutionMetrics`'s logs clearer
    omit c1968a1bd85f [SPARK-50216][SQL][TESTS] Update `CollationBenchmark` to invoke `collationNameToId` outside of cases
    omit aea9e875fba4 [SPARK-50291][PYTHON] Standardize verifySchema parameter of createDataFrame in Spark Classic
    omit 0b1b6766b472 [SPARK-50092][SQL] Fix PostgreSQL connector behaviour for multidimensional arrays
    omit 09d6b32e69d6 [SPARK-48755][DOCS][PYTHON][FOLLOWUP] Add PySpark doc for `transformWithState`
    omit 6bee2681f1f3 [SPARK-50299][BUILD] Upgrade jupiter-interface to 0.13.1 and Junit5 to 5.11.3
    omit 2fd470263714 [SPARK-49913][SQL] Add check for unique label names in nested labeled scopes
    omit 891f694207ea [SPARK-50306][PYTHON][CONNECT] Support Python 3.13 in Spark Connect
    omit 33378a6f86e0 [SPARK-50304][INFRA] Remove `(any|empty).proto` from RAT exclusion
    omit 5cc60f467088 [SPARK-50300][BUILD] Use mirror host instead of `archive.apache.org`
    omit 05508cf7cb9d [SPARK-42838][SQL] Assign a name to the error class _LEGACY_ERROR_TEMP_2000
    omit 87ad4b4a2cfb [SPARK-50139][INFRA][SS][PYTHON] Introduce scripts to re-generate and checking StateMessage_pb2.py and StateMessage_pb2.pyi
    omit 7b1b450bb65b Revert [SPARK-50215][SQL] Refactored StringType pattern matching in jdbc code stack
    omit 558fc89f8ccf [SPARK-49611][SQL][FOLLOW-UP] Make collations TVF consistent and return null on no result for country and language
    omit bc9b2597ea2d [SPARK-50066][SQL] Codegen Support for `SchemaOfXml` (by Invoke & RuntimeReplaceable)
    omit bd94419c988b [SPARK-50226][SQL] Correct MakeDTInterval and MakeYMInterval to catch Java exceptions
    omit 898bff21c992 [SPARK-50245][SQL][TESTS] Extended CollationSuite and added tests where SortMergeJoin is forced
    omit 6fb1d4381912 [SPARK-50246][SQL] Assign appropriate error condition for `_LEGACY_ERROR_TEMP_2167`: `INVALID_JSON_RECORD_TYPE`
    omit ede05fa500fe [SPARK-50248][SQL] Assign appropriate error condition for `_LEGACY_ERROR_TEMP_2058`: `INVALID_PARTITION_VALUE`
    omit 40ffdefc2e11 [SPARK-50250][SQL] Assign appropriate error condition for `_LEGACY_ERROR_TEMP_2075`: `UNSUPPORTED_FEATURE.WRITE_FOR_BINARY_SOURCE`
    omit 82040bbd41cc [SPARK-49563][SQL] Add SQL pipe syntax for the WINDOW operator
    omit cc84e7f1824f [SPARK-50296][PYTHON][CONNECT] Avoid using a classproperty in threadpool for Python Connect client
    omit e29db6ecfa57 [SPARK-50283][INFRA] Add a separate docker file for linter
    omit 158aeb071147 [SPARK-38912][PYTHON] Remove the comment related to classmethod and property
    omit dc6fba58b9fc [SPARK-50290][SQL] Add a flag to disable DataFrameQueryContext creation
    omit a84ca5e46e63 [SPARK-50241][SQL] Replace NullIntolerant Mixin with Expression.nullIntolerant method
    omit 26330355836f [SPARK-49249][SPARK-49122] Artifact isolation in Spark Classic
    omit 1198117a82ab [SPARK-50130][SQL][PYTHON][TEST] Add more test for outer reference
    omit 3c3d1a6a9f6d [SPARK-48898][SQL] Add Variant shredding functions
    omit 4002a5352d54 [SPARK-50238][PYTHON] Add Variant Support in PySpark UDFs/UDTFs/UDAFs
    omit 4f4eb22f5e9e [SPARK-50234][PYTHON][SQL] Improve error message and test for transpose DataFrame API
    omit 916bf533a3eb [SPARK-50259][BUILD] Update Parquet to 1.14.4
    omit e7ebda6994d3 [SPARK-48273][SQL] Revert "[] Fix late rewrite of PlanWithUnresolvedIdentifier
    omit 00bff28cd66d [SPARK-50056][SQL] Codegen Support for ParseUrl (by Invoke & RuntimeReplaceable)
    omit 4f95a7f4dd2c [SPARK-50260][CONNECT] Refactor and optimize Spark Connect execution and session management
    omit 0a66c371cd4c [SPARK-50130][SQL][PYTHON] Add DataFrame APIs for scalar and exists subqueries
    omit 5432cef483c4 [SPARK-50152][SS] Support handleInitialState with state data source reader
    omit d96c623c7601 Revert "[SPARK-50222][PYTHON][FOLLOWUP] Support `spark.submit.appName` in PySpark"
    omit 7ba279a3ff76 [SPARK-50118][CONNET] Reset isolated state cache when tasks are running
    omit 2df08b1de737 [SPARK-50282][ML] Simplify `TargetEncoderModel.transform`
    omit 039f796c893a [SPARK-50279][DOCS][PYTHON] Add DataFrame.plot to DataFrame API reference
    omit 0aa854129d37 [SPARK-50280][PYTHON] Refactor result sorting and empty bin filling in `compute_hist`
    omit cf9ca4261e45 [SPARK-50278][BUILD] Upgrade `netty-tcnative` to 2.0.69.Final
    omit a2f8805dea9b [SPARK-50268][SQL][TESTS] Upgrade oracle jdbc driver to `ojdbc17:23.6.0.24.10`
    omit 6dd5c5fb5754 [MINOR][DOCS][PYTHON] Fix groupped aggreagte pandas UDF example in df.groupby.agg
    omit a08859760a41 [SPARK-50262][SQL] Forbid specification complex types during altering collation
    omit 7dbd17d5ea23 [SPARK-50275][SS][PYTHON] Enable test_pandas_transform_with_state unit test
    omit 6fd77b1e7f5b [SPARK-50264][SQL][CONNECT] Add missing methods to DataStreamWriter
    omit 655061aaf728 [SPARK-50102][SQL][CONNECT] Add shims need for missing public SQL methods
    omit e490dd737dbf [SPARK-50274][CORE] Guard against use-after-close in DirectByteBufferOutputStream
    omit 2cdbedeaebdf [SPARK-50267][ML] Improve `TargetEncoder.fit` with DataFrame APIs
    omit 8f5d8d424644 [SPARK-50270][SS][PYTHON] Added custom state metrics for TransformWithStateInPandas
    omit 202400ae40dc [SPARK-50273][SS] Improve logging for RocksDB lock acquire/release cases
    omit b23905ac0c62 [SPARK-50222][PYTHON][FOLLOWUP] Support `spark.submit.appName` in PySpark
    omit 5d757993f4cf [SPARK-50269][SQL][TESTS] Further improve collation support testing for various collations
    omit e4638c888cf3 [SPARK-49899][PYTHON][SS] Support deleteIfExists for TransformWithStateInPandas
    omit 60acd2fe442c [SPARK-50261][SQL] Fix error signature for invalid fraction of a second
    omit 01adf10405fb [SPARK-50256][SQL] Add lightweight validation to check if a logical plan becomes unresolved after every optimizer rule
    omit a45e819da0ce [SPARK-50230][SQL] Added logic to support reading unknown collation name as utf8_binary
    omit 9858ab6d9c8e [SPARK-50055][SQL] Add TryMakeInterval alternative
    omit d9c596c040b0 [SPARK-50251][PYTHON] Add `getSystemProperty` to PySpark `SparkContext`
    omit 49b081187fb5 [SPARK-50253][SS] Stream-Stream Join should not fetch checkpoint ID if not supported
    omit 07301ddb889b [SPARK-50135][BUILD] Upgrade ZooKeeper to 3.9.3
    omit 1f9a12cf7511 [SPARK-50255][PYTHON] Avoid unnecessary casting in `compute_hist`
    omit 755316e0a4fd [SPARK-50254][PS][TESTS] Enable test `DataFramePlotParityTests`
    omit a4391364dd24 [SPARK-50247][CORE] Define `BLOCK_MANAGER_REREGISTRATION_FAILED` as `ExecutorExitCode`
    omit f2d39b958deb [SPARK-50208][CORE][FOLLOWUP] Rename `useDriverIdAsAppId` variable to `useDriverIdAsAppName`
    omit abe990e58eee [SPARK-37178][ML] Add Target Encoding to ml.feature
    omit 99d27c970101 [MINOR][SQL] Add more description to SKIP_TYPE_VALIDATION_ON_ALTER_PARTITION
    omit 0cb7a4273107 [SPARK-48590][BUILD] Upgrade netty to `4.1.114.Final`
    omit 800faf0abfa3 [SPARK-50235][SQL] Clean up ColumnVector resource after processing all rows in ColumnarToRowExec
    omit 737a65e0ec41 [SPARK-50096][SQL] Assign appropriate error condition for `_LEGACY_ERROR_TEMP_2150`: `TUPLE_SIZE_EXCEEDS_LIMIT`
    omit 4418f24a03f4 [SPARK-50156][SQL] Integrate `_LEGACY_ERROR_TEMP_2113` into `UNRECOGNIZED_STATISTIC`
    omit fe70ca4cf9e0 [SPARK-50228][SQL] Move the `RewriteCollationJoin` rule to `FinishAnalysis`
    omit 4771638b3a68 [SPARK-50229] Reduce memory usage on driver for wide schemas by reducing the lifetime of AttributeReference objects created during logical planning
    omit 0ee26599d68e [SPARK-50011][INFRA] Add a separate docker file for doc build
    omit 36410f073cb9 [SPARK-49601][SS][PYTHON] Support Initial State Handling for TransformWithStateInPandas
    omit 9fbefb3e575c Revert "Revert "[SPARK-50227][CONNECT][PYTHON] Upgrade buf plugins to `v28.3`""
    omit 6c4d3446b445 Revert "[SPARK-50135][BUILD] Upgrade ZooKeeper to 3.9.3"
    omit 52df0cdf24e1 [SPARK-50162][SS][TESTS] Add tests for loading snapshot with given version for transformWithState operator state and state data source reader
    omit 667cff76b248 Revert "[SPARK-50227][CONNECT][PYTHON] Upgrade buf plugins to `v28.3`"
    omit 2c4f748e8924 [SPARK-50112][SQL] Allowing the TransformWithState operator to use Avro encoding
    omit b81da764b9c5 [SPARK-49995][SQL] Add named argument support to more TVFs
    omit 066a30ecacbc [SPARK-50231][PYTHON] Make function `instr` accept Column `substring`
    omit b7e70feef54a [SPARK-50233][PYTHON][CONNECT][TESTS] Skip PipelineTests if torch is not installed
    omit fd3b61033304 [SPARK-50222][CORE] Support `spark.submit.appName`
    omit acdccfe0b3e3 [SPARK-50232][PYTHON][CONNECT] Add 'protobuf==5.28.3' in dev/requirements.txt
    omit 6eee2717d7b5 [SPARK-50028][CONNECT] Replace global locks in Spark Connect server listener with fine-grained locks
    omit 57f6824e78e2 [SPARK-50227][CONNECT][PYTHON] Upgrade buf plugins to `v28.3`
    omit 74d82665bc2e [SPARK-50219][SQL] Refactor `ApplyCharTypePadding` so that helper methods can be used in single-pass resolver
    omit 46fe10afc140 [SPARK-50215][SQL] Refactored StringType pattern matching in jdbc code stack
    omit 273b02fd6cdc [SPARK-50214][SQL] From json/xml should not change collations in the given schema
    omit 642a62b6970f [SPARK-50224][SQL] The replacements of IsValidUTF8|ValidateUTF8|TryValidateUTF8|MakeValidUTF8 shall be NullIntolerant
    omit 47063a6a75f7 [SPARK-50128][SS] Add stateful processor handle APIs using implicit encoders in Scala
    omit 0d2d031c2d90 [SPARK-50218][BUILD] Upgrade checkstyle to 10.20.0
    omit 2afd6134d21e [SPARK-50217][BUILD][TESTS] Upgrade `bouncycastle` to 1.79
    omit 98dd2b024003 [SPARK-49615][ML] Make all ML feature transformers dataset schema validation conforming "spark.sql.caseSensitive" config
    omit 3985a76e6a3a [SPARK-50189][BUILD][SQL] Upgrade ICU4J to `76.1`
    omit c53dac05058c [SPARK-50211][PYTHON][CONNECT][TESTS] Skip FeatureTests if torch is not installed
    omit a0b4205d9251 [SPARK-50151][SS][ROCKSDB HARDENING] Fix ineffective file reuse bug in the new file management change
    omit 91c01dbad451 [SPARK-50196][CONNECT] Fix Python error context to use a proper context
    omit 9cf98ed41b2d [SPARK-50210][CORE] Fix `SparkSubmit` to show REST API `kill` response properly
    omit f33b142cc3f5 [SPARK-50209][R][INFRA] Use R 4.4.2 in `windows` R GitHub Action Windows job
    omit e305e7035ce7 [SPARK-49696][SQL][TESTS] Disable flaky test `Cancellation APIs in SparkSession are isolated`
    omit 934134e99aed [SPARK-50163][SS] Fix the RocksDB extra acquireLock release due to the completion listener
    omit fe88d1d70fa0 [SPARK-50154][SQL] Assign appropriate error condition for `_LEGACY_ERROR_TEMP_0043`: `INVALID_RESET_COMMAND_FORMAT`
    omit f2b41b034a8f [SPARK-50208][CORE] Support `spark.master.useDriverIdAsAppName.enabled`
    omit fc77b1420739 [SPARK-50204][SQL] Factor out `HiveTableRelation` read path resolution
    omit d6b733492d3e [SPARK-50033][SQL] Add a hint to logical.Aggregate() node
    omit 362d1c7361e2 [SPARK-49530][PYTHON] Get active session from dataframes
    omit 2cb72322033d [SPARK-50184][SPARK-48997][SS][TESTS] Make test case `: maintenance threads with exceptions unload only themselves` use a self-cleaning `rootLocation`
    omit fcfbf8e660db [SPARK-50199][PYTHON][TESTS] Use Spark 3.4.4 instead of 3.0.1 in `test_install_spark`
    omit 0030fe0ad34c [SPARK-50200][PYTHON] Remove unused `have_numpy` testing utility from `sqlutils.py`
    omit ce899404f59d [SPARK-50195][CORE] Fix `StandaloneRestServer` to propagate `spark.app.name` to `SparkSubmit` properly
    omit b526924f58e0 [SPARK-50197][PYTHON][INFRA] Install Jinja2 for Python 3.13 build
    omit f67ae598115d [SPARK-50198][ML][CONNECT] Skip CrossValidatorTestsOnConnect if torch is not installed
    omit c02727093e59 [SPARK-50193][SS] Fix exception handling for validating time modes
    omit d33b3bef86e9 [SPARK-50166][SQL][BUILD] Fix shade and relocation rule of `sql/core` module
    omit f1c7aa0b78c3 [SPARK-50192][K8S] Simplify symbolic link creation of `spark-examples.jar`
    omit a84e6811881a [SPARK-50190][PYTHON] Remove direct dependency of Numpy from Histogram
    omit 125c1da1810a Revert "[SPARK-48139][CONNECT][TESTS] Try stabilising multi-thread tests in CI"
    omit 793a7dfeefca [SPARK-50140][BUILD] Upgrade `zstd-jni` to 1.5.6-7
    omit 60d68034e6a7 [SPARK-50187][INFRA] Use latest Python dependencies for the build of Python client 3.5 <> 4.0 server
    omit 082bdb7f33e2 [SPARK-49980][CORE][SQL] Fix potential file stream leaks caused by interruption in canceled tasks
    omit 77630e9f058a [SPARK-50169][SQL] Improve performance of `RegExpReplace`
    omit 165c754652eb [SPARK-50183][PYTHON] Unify the internal functions for Pandas API and PySpark Plotting
    omit 1a691131f5c6 [SPARK-50173][PS][SQL] Make pandas expressions accept more datatypes
    omit beea9c36725e [SPARK-50172][BUILD] Upgrade scala-maven-plugin to 4.9.2
    omit a6c1920f6a6f [SPARK-50185][EXAMPLE][TESTS] Unify `*.sh` file naming patterns by replacing `_` with `-`
    omit 7c0e66c4794a [SPARK-50182][EXAMPLE] Add `submit-sql.sh` REST API example
    omit d39e5f1a55d3 [SPARK-50181][INFRA][DOCS] Remove `run-tests-jenkins`-related stuff
    omit 069c46c3fd49 [SPARK-50179][CORE] Make `spark.app.name` property optional in REST API
    omit 6ba13ee35e0c [SPARK-50178][INFRA] Use `PyArrow>=18.0.0` for Python 3.13
    omit 2ace2ebf7ae1 [SPARK-49513][SS] Add Support for timer in transformWithStateInPandas API
    omit f9252a908a03 [SPARK-50177][BUILD] Upgrade `Arrow` to 18.0.0
    omit cb5938363ff5 [SPARK-50075][SQL][PYTHON][CONNECT] Add DataFrame APIs for table-valued functions
    omit 428f75942ecf [SPARK-50144][SS] Address the limitation of metrics calculation with DSv1 streaming sources
    omit 18bbddd2fe2d [SPARK-50068][SQL] Refactor `TypeCoercion` and `AnsiTypeCoercion` to separate single node transformations
    omit bb8b691b0f66 [SPARK-48139][CONNECT][TESTS] Try stabilising multi-thread tests in CI
    omit e8c79634b2a1 [SPARK-50171][PYTHON] Make numpy optional for KDE plot
    omit 01464b9ec1ca [SPARK-50124][SQL] LIMIT/OFFSET should preserve data ordering
    omit 73a2b84ad924 [SPARK-47094][SQL][TEST][FOLLOWUP] SPJ : fix bucket reducer function
    omit 91dd5e3d74b7 [SPARK-50148][SQL] Make `StaticInvoke` compatible with the method that declare throw exception
    omit d5bc478a4989 [SPARK-50165][BUILD] Upgrade jackson to 2.18.1
    omit 94abea8572fe [SPARK-50167][PYTHON][CONNECT] Improve PySpark plotting error messages and imports
    omit b7ac624d56c2 [SPARK-50174][SQL] Factor out `UnresolvedCatalogRelation` resolution
    omit 2316fed65192 [SPARK-50170][PYTHON] Move `_invoke_internal_function_over_columns` to `pyspark.sql.utils`
    omit 214f95819f83 [SPARK-49766][SPARK-49939][SPARK-49967][SPARK-49966][SQL][FOLLOWUP] Remove redundant whether is null check
    omit 8f82d0f3dc17 [SPARK-49663][SQL] Enable RTRIM suggestions in collation expressions
    omit 413242bea60d [SPARK-50117][BUILD][SS] Change to using `maven/sbt` plugin to generate `StateMessage.java`
    omit f86c6eabee81 [SPARK-49562][SQL] Add SQL pipe syntax for aggregation
    omit b6c569fe4d70 [SPARK-50159][PYTHON][DOCS][TESTS] Refine docstrings for aggregation functions - part 1
    omit dbd946986dab [SPARK-50161][BUILD] Upgrade Kafka to 3.8.1
    omit 6a36c43604a6 [SPARK-49506][SQL] Optimize ArrayBinarySearch for foldable array
    omit 4d3004835209 [SPARK-50155][INFRA] Move `scala` and `java` files to their default folders
    omit bb29a3f8e658 [SPARK-50069][SQL] Integrate `_LEGACY_ERROR_TEMP_0028` into `UNSUPPORTED_FROM_TO_EXPRESSION`
    omit 56593f4fb77b [SPARK-49637][SQL] Changed error message for INVALID_FRACTION_OF_SECOND
    omit 9968a32594d4 [SPARK-50071][SQL][PYTHON] Add try_make_timestamp(_ltz and _ntz) and related tests
    omit 87f5a8676368 [SPARK-49297][CORE][TESTS] Fix race condition in BlockManagerDecommissionIntegrationSuite
    omit d371ec03c030 [SPARK-50149][INFRA] Update INFRA docker image to use `jammy-20240911.1`
    omit 0a06322d1826 [SPARK-50147][PYTHON][DOCS][TESTS] Refine docstring for trigonometric functions
    omit 9116932a82c6 [SPARK-50145][BUILD] Upgrade Jetty to 11.0.24
    omit 1cd72c6b4a0e [SPARK-49993][SQL] Improve error messages for Sum and Average
    omit cde8e4a82e20 [SPARK-49707][BUILD] Upgrade rocksdbjni to 9.7.3
    omit bb15eb7b91ab [SPARK-50143][BUILD] Fix `protobuf` module Maven compilation
    omit 51bd96d3b70d [SPARK-50141][PYTHON] Make `lpad` and `rpad` accept Column type arguments
    omit 03e051b5e400 [SPARK-49999][PYTHON][CONNECT] Support optional "column" parameter in box, kde and hist plots
    omit d2ffccbe9cd0 [SPARK-50100][INFRA] Remove `branch-3.4`-related CIs and comments
    omit 55381092bb87 [SPARK-50138][PYTHON][DOCS][TESTS] Refine docstring for basic functions
    omit eed1530790b6 [SPARK-49665][SQL] Trim collation support for string functions
    omit 9b2129da49db [SPARK-50136][BUILD] Upgrade `curator` to 5.7.1
    omit 3f54a5ff657d [SPARK-50135][BUILD] Upgrade ZooKeeper to 3.9.3
    omit 9909817aef91 [SPARK-48912][SQL][TESTS] Improve collation support testing for various collations
    omit b8c2a320bf17 [SPARK-50046][SS] Use stable order of EventTimeWatermark node to calculate watermark
    omit c08616307fe7 [SPARK-50015][BUILD] Upgrade `grpcio*` to 1.67.0 and `grpc-java` to 1.67.1
    omit 4de286aed61c [SPARK-50123][TESTS] Move BitmapExpressionUtilsSuite & ExpressionImplUtilsSuite from java to scala test sources folder
    omit f74947e071a6 [SPARK-50121][INFRA] Add `Python 3.9` Daily Python GitHub Action job
    omit 250f8affd04e [SPARK-50107][SQL][BUILD] Upgrade `protobuf-java` to 4.28.3
    omit b0dc6546a0b7 [SPARK-50106][PYTHON] Upgrade `protobuf` Python package to 5.28.3
    omit 5a43c978a8dc [SPARK-50060][SQL] Disabled conversion between different collated types in TypeCoercion and AnsiTypeCoercion
    omit 3da8f7066892 [SPARK-50015][SQL] Assign appropriate error condition for `_LEGACY_ERROR_TEMP_1125`: `MISSING_DATABASE_FOR_V1_SESSION_CATALOG`
    omit 413a65bbe6b8 [SPARK-50061][SQL] Enable analyze table for collated columns
    omit c976c80d94e2 [SPARK-50097][SQL] Assign appropriate error condition for `_LEGACY_ERROR_TEMP_1248`: `ALTER_TABLE_SERDE_FOR_DATASOURCE_TABLE`
    omit c558d4dafea2 [SPARK-50031][SQL][FOLLOW-UP] Add tests for `try_parse_url` in the connect module
    omit 6e82d5946ae7 [SPARK-50109][PYTHON][DOCS] Refine the docstring for more date functions
    omit b0b959c99d2c [SPARK-50108][INFRA][PYTHON][TESTS] Upgrade PyPy CI to 3.10
    omit ea32ce268be9 [SPARK-50098][PYTHON][FOLLOWUP] Update `_minimum_googleapis_common_protos_version` in `setup.py`
    omit cfe14c9c275b [SPARK-50098][PYTHON] Upgrade the minimum version of `googleapis-common-protos` to 1.65.0
    omit 95b2d27079c2 [SPARK-50099][K8S][DOCS] Update `YuniKorn` docs with `1.6.0`
    omit a0cbc91e84d1 [SPARK-46461][BUILD] Clean up unavailable `SQL/Hive console settings` from SparkBuild.scala
    omit e55a80651db8 [SPARK-50093][SQL] Collations that use ICU should have the version of the used ICU library
    omit 1985b9c5a591 [SPARK-50062][SQL] Support collations by `InSet`
    omit da609ba81d63 [SPARK-50079][SQL] Assign appropriate error condition for `_LEGACY_ERROR_TEMP_2013`: `NEGATIVE_VALUES_IN_FREQUENCY_EXPRESSION`
    omit 4ac20ad6f211 [SPARK-50077][SQL] Introduce a new pattern object for LogicalRelation to help avoiding default full params pattern
    omit 1d6e0a992c95 [SPARK-50094][PYTHON][CONNECT] Better error message when using memory profiler on editors with no line numbers
    omit 867de47a4151 [SPARK-50090] Refactor ResolveBinaryArithmetic to separate single-node transformation
    omit aeddc3f55e60 [SPARK-50034][CORE] Fix Misreporting of Fatal Errors as Uncaught Exceptions in `SparkUncaughtExceptionHandler`
    omit 18f01ecc1b57 [SPARK-49982][SQL] Fix negative caching in InMemoryRelation
    omit b72bf79f5a26 [SPARK-49854][SQL] Clone artifact manager during session clone
    omit 1373bedf5158 Revert "[SPARK-50007][SQL][SS] Provide default values for metrics on observe API when physical node is lost in executed plan"
    omit be19a3ed7e3f [SPARK-50085][PYTHON] Make `lit(ndarray)` with np.int8 respect the numpy datatype
    omit d65c85e1c012 [SPARK-50086][PYTHON][DOCS] Refine the docstring of multiple datetime functions
    omit 7066f061264b [SPARK-50088][SQL] Refactor `UnresolvedStarBase.expand`
    omit dcb4177822ad [SPARK-50084][SQL] Assign appropriate error condition for `_LEGACY_ERROR_TEMP_3168`: `MISSING_TIMEOUT_CONFIGURATION`
    omit 5e9dec777697 [SPARK-49638][SQL] Remove the ANSI config suggestion in INVALID_URL
    omit be18b7cef3b3 [MINOR][PYTHON][DOCS] Fix the type hint for `extract`, `date_part` and `datepart`
    omit 35d2ef99180d [SPARK-49902][SQL] Catch underlying runtime errors in `RegExpReplace`
    omit c8fb2112ef62 [SPARK-50078][SQL] Assign appropriate error condition for `_LEGACY_ERROR_TEMP_0038`: `DUPLICATED_CTE_NAMES`
    omit 369c40cdb24a [SPARK-50067][SQL] Codegen Support for SchemaOfCsv(by Invoke & RuntimeReplaceable)
    omit 2cb7a16a2f9c [SPARK-50080][SQL][TESTS] Add benchmark cases for parquet adaptive bloom filter in BloomFilterBenchmark
    omit 51e915a0ca2d [SPARK-50064][PYTHON][TESTS][FOLLOW-UP] Make pysaprk-ml tests passing without optional dependencies
    omit 2957069efbce [SPARK-49985][SQL] Remove support for interval types in Variant
    omit 2bf41a69e56f [SPARK-50050][PYTHON][CONNECT] Make `lit` accept `str` and `bool` type numpy ndarray
    omit 008381518fa3 [SPARK-50065][SQL][TESTS] Upgrade `MySQL` & `Mariadb` docker image version
    omit abc49865055f [SPARK-50031][SQL] Add the `TryParseUrl` expression
    omit 3ec28ff545ab [SPARK-49934][SQL] Add implicit cast for accessing collated map with literal
    omit 6e3dcdbf5adc [SPARK-48413][SQL][FOLLOW-UP] Fix ALTER COLUMN with collation
    omit 3015e3e283f7 [SPARK-50070][SQL] Integrate `_LEGACY_ERROR_TEMP_0039` into `UNSUPPORTED_SQL_STATEMENT`
    omit a7b686b6ebff [SPARK-50054][PYTHON][CONNECT] Support histogram plots
    omit a37688047c7b [SPARK-50059][CONNECT][PYTHON] API compatibility check for Structured Streaming I/O
    omit 61db25138617 [SPARK-50051][PYTHON][FOLLOW-UP] Fix data type check for Numpy 1.x
    omit b142a195593d [SPARK-49708][BUILD] Upgrade `RoaringBitmap` to `1.3.0`
    omit ff0653fa0b22 [SPARK-50047][BUILD] Upgrade jersey to 3.0.16
    omit 693ebd9b616c [SPARK-50057][SQL] Assign appropriate error condition for `_LEGACY_ERROR_TEMP_1049`: `INVALID_ATTRIBUTE_NAME_SYNTAX`
    omit aaa704485932 [SPARK-49461][SS] Persistent Checkpoint ID to commit logs and read it back
    omit dc35ba87a1aa [SPARK-50051][PYTHON][CONNECT] Make `lit` works with empty numpy ndarray
    omit e7cdb5a67145 [SPARK-49944][DOCS] Fix broken main.js import and fix image links for streaming documentation
    omit 2c904e4b682e [SPARK-49849][CONNECT][PYTHON] API compatibility check for Structured Streaming Query Management
    omit d5419ab0960e [SPARK-50058][SQL] Factor out plan normalization functions to later use those in single-pass Analyzer testing
    omit 09b7bdff48a6 [SPARK-50064][PYTHON][TESTS] Make pysaprk-ml-connect tests passing without optional dependencies
    omit 70c9b1f978a2 [SPARK-49552][CONNECT][FOLLOW-UP] Make 'randstr' and 'uniform' deterministic in Scala Client
    omit 91ae1022198b [SPARK-50052][PYTHON] Make NumpyArrayConverter support empty str ndarray
    omit b4eb03469a90 [SPARK-49967][SQL] Codegen Support for `StructsToJson`(`to_json`)
    omit f86df1e37b47 [SPARK-50036][CORE][PYTHON] Include SPARK_LOG_SCHEMA in the context of REPL shell
    omit 3e597e2ec7f6 [SPARK-50053][SQL] Turn `_LEGACY_ERROR_TEMP_2104` into `INTERNAL_ERROR`
    omit f9a5de475caa [SPARK-50029][SQL] Make `StaticInvoke` compatible with the method that return `Any`
    omit 738dfa346bde [SPARK-50035][SS] Add support for explicit handleExpiredTimer function part of the stateful processor
    omit 26325440a2b7 [SPARK-49851][CONNECT][PYTHON] API compatibility check for Protobuf
    omit 4ed908e96af5 [SPARK-50048][SQL] Assign appropriate error condition for `_LEGACY_ERROR_TEMP_2114`: `UNRECOGNIZED_STATISTIC`
    omit c1198fa2b45c [SPARK-49821][SS][PYTHON] Implement MapState and TTL support for TransformWithStateInPandas
    omit 537ca919c87a [SPARK-49850][CONNECT][PYTHON] API compatibility check for Avro
    omit f581bad90258 [SPARK-50037][SQL] Refactor AttributeSeq.resolve(...)
    omit d2e322314c78 [SPARK-50001][PYTHON][PS][CONNECT] Adjust "precision" to be part of kwargs for box plots
    omit 32cc2ddc8e9a [SPARK-50044][PYTHON] Refine the docstring of multiple math functions
    omit 450891109f44 [SPARK-50027][SQL] Move Identity Column SQL parsing code to DataTypeAstBuilder
    omit f2f309970ff5 [SPARK-50038][SQL] Assign appropriate error condition for `_LEGACY_ERROR_TEMP_0008`: `MERGE_WITHOUT_WHEN`
    omit 76ea894bc69f [SPARK-50039][CONNECT][PYTHON] API compatibility check for Grouping
    omit ae75cac50d37 [SPARK-50040][PYTHON][TESTS] Make pysaprk-connect tests passing without optional dependencies
    omit 164a0f46d657 [MINOR][DOCS][PYTHON] Clarify that profilers show the accumulated results
    omit 4883689e90bd [SPARK-50042][PYTHON] Upgrade numpy 2 for python linter
    omit 1ca2d5620d00 [MINOR][TESTS] Fix appname from `sql.protobuf.functions` to `sql.connect.protobuf.functions`
    omit a31da6a37ef6 [SPARK-50024][PYTHON][CONNECT] Switch to use logger instead of warnings module in client
    omit 64218d71951e [MINOR][DOCS] Add Development Version of docs in README.md
    omit d5550f65a5da [SPARK-49530][PYTHON][CONNECT] Support kde/density plots
    omit 25b03f9d3471 [SPARK-50023][PYTHON][CONNECT] API compatibility check for Functions
    omit 14ed86e24dba [SPARK-50030][PYTHON][CONNECT] API compatibility check for Window
    omit f8d92224b9af [SPARK-49550][BUILD] Upgrade Hadoop to 3.4.1
    omit b1d1f10f96b1 [SPARK-49846][SS] Add numUpdatedStateRows and numRemovedStateRows metrics for use with transformWithState operator
    omit 6f710cd9c561 Revert "[SPARK-50011][INFRA] Add a separate docker file for doc build"
    omit aaecab3d3c8b Revert "[SPARK-50011][INFRA][FOLLOW-UP] Refresh the image cache job"
    omit 9d0e31dda53d [SPARK-50022][CORE][UI] Fix `MasterPage` to hide App UI links when UI is disabled
    omit ae87ce6c7f9e [SPARK-50026][SQL] Fix Identity Column metadata get
    omit a1fc7e66883a [SPARK-49966][SQL] Use `Invoke` to implement `JsonToStructs`(`from_json`)
    omit ff47dd9516e8 [SPARK-50021][CORE][UI] Fix `ApplicationPage` to hide App UI links when UI is disabled
    omit 4c6367255d75 [SPARK-50004][SQL] Integrate `_LEGACY_ERROR_TEMP_3327` into `FIELD_NOT_FOUND`
    omit 7e3b1680b09f [SPARK-49990][SQL] Improve performance of `randStr`
    omit 7e9017ec8f6e [SPARK-50025][SQL] Integrate `_LEGACY_ERROR_TEMP_1253` into `EXPECT_VIEW_NOT_TABLE`
    omit 0705e1022923 [SPARK-50007][SQL][SS] Provide default values for metrics on observe API when physical node is lost in executed plan
    omit b0414b19a0ea Revert "[SPARK-49909][SQL] Fix the pretty name of some expressions"
    omit 25c55a86c39f [SPARK-50016][SQL] Improve explicit collation mismatch error
    omit 693d008de0e4 [SPARK-49808][SQL] Fix a deadlock in subquery execution due to lazy vals
    omit fae0c12fea88 [SPARK-49558][SQL][FOLLOW-UP] Run `./build/mvn scalafmt:format` to fix CI
    omit af0cc8ef3918 [SPARK-50018][SQL] Make AbstractStringType serializable
    omit 55c558a9fb00 [SPARK-50010][SQL] Expand implicit collation mismatch error
    omit 2724909656fe [SPARK-50011][INFRA][FOLLOW-UP] Refresh the image cache job
    omit ff5e0a11f6a1 [SPARK-49558][SQL] Add SQL pipe syntax for LIMIT/OFFSET and ORDER/SORT/CLUSTER/DISTRIBUTE BY
    omit 5697df70b3b7 [SPARK-49411][SS] Communicate State Store Checkpoint ID between driver and stateful operators
    omit 82802494fa35 [SPARK-49802][SS] Add support for read change feed for map and list types used in stateful processors
    omit d90145db7b84 [SPARK-49991][SQL] Make HadoopMapReduceCommitProtocol respect 'mapreduce.output.basename' to generate file names
    omit 75b86667ee76 [SPARK-49829][SS] Fix the bug on the optimization on adding input to state store in stream-stream join
    omit 78308da2e1e3 [SPARK-48773][FOLLOW-UP] spark.conf.set should not fail when setting `spark.default.parallelism`
    omit eb73143be6fc [SPARK-50019][INFRA] Install BASIC_PIP_PKGS except `pyarrow` in Python 3.13 image
    omit bdd34647c42a [SPARK-50014][INFRA] Use `grpcio*` 1.67.0 in Python 3.13 image
    omit 91becf140cfe [SPARK-49962][SQL] Simplify AbstractStringTypes class hierarchy
    omit 6362e0c30fab [SPARK-49936][BUILD] Upgrade `datasketches-java` to 6.1.1
    omit 120ae9ac889a [SPARK-49958][PYTHON] Python API for string validation functions
    omit 6bcad771246b [SPARK-50008][PS][CONNECT] Avoid unnecessary operations in `attach_distributed_sequence_column`
    omit d144144e3133 [SPARK-50003][PYTHON][DOCS] Add pie plot doc examples and correct area plot parameter type hints
    omit 24d9c97916c1 [SPARK-50002][PYTHON][CONNECT] API compatibility check for I/O
    omit 360ce0a82bc9 [SPARK-50011][INFRA] Add a separate docker file for doc build
    omit f1f04b097d2f [SPARK-49988][BUILD] Remove unused Hadoop dependency management
    omit b078c0d6e2ad [SPARK-50009][INFRA] Exclude pandas/resource/testing from `pyspark-core` module
    omit 0dd6a2a46441 [SPARK-49997][SQL] Integrate `_LEGACY_ERROR_TEMP_2165` into `MALFORMED_RECORD_IN_PARSING`
    omit 0b7cc3709e6b [SPARK-49971][SQL] Assign proper error condition for _LEGACY_ERROR_TEMP_1097
    omit 371a09a292ef [SPARK-49998][SQL] Integrate `_LEGACY_ERROR_TEMP_1252` into `EXPECT_TABLE_NOT_VIEW`
    omit 2469a6f29c0e [SPARK-49495][INFRA][FOLLOW-UP] Disable GitHub Pages workflow in forked repository
    omit f96a6f8e3a0b [SPARK-49259][SS] Size based partition creation during kafka read
    omit 8405c9b57a1f [SPARK-49770][SS][ROCKSDB HARDENING] Improve RocksDB SST file mapping management, and fix issue with reloading same version with existing snapshot
    omit 74dbd9ab9cff [SPARK-49996][SQL][TESTS] Upgrade `mysql-connector-j` to 9.1.0
    omit c8bedb49e9b9 [SPARK-49848][PYTHON][CONNECT] API compatibility check for Catalog
    omit 175d56310fae [SPARK-49977][SQL] Use stack-based iterative computation to avoid creating many Scala List objects for deep expression trees
    omit e374b94a9c8b [SPARK-49979][SQL] Fix AQE hanging issue when collecting twice on a failed plan
    omit 070f2bdfb968 [SPARK-49876][CONNECT] Get rid of global locks from Spark Connect Service
    omit 948aeba93e1a [SPARK-49947][SQL][TESTS] Upgrade `MsSql` docker image version
    omit 9af705d27cae [SPARK-49951][SQL] Assign proper error condition for _LEGACY_ERROR_TEMP_(1099|3085)
    omit baa5f408a098 [SPARK-49945][PS][CONNECT] Add alias for `distributed_id`
    omit 224d3ba1a2cd [SPARK-49986][INFRA] Restore `scipy` installation in dockerfile
    omit e92bf3746ebf [SPARK-49978][R] Move sparkR deprecation warning to package attach time
    omit f5e6b05e486e [SPARK-49643][SQL] Merge _LEGACY_ERROR_TEMP_2042 into ARITHMETIC_OVERFLOW
    omit 31a411773a3e [SPARK-49057][SQL][TESTS][FOLLOWUP] Handle `_LEGACY_ERROR_TEMP_2235` error case
    omit f860af67db34 [SPARK-48155][FOLLOWUP][SQL] AQEPropagateEmptyRelation for left anti join should check if remain child is just BroadcastQueryStageExec
    omit 60200ae195a1 [SPARK-49957][SQL] Scala API for string validation functions
    omit bcfe62b9988f [SPARK-49983][CORE][TESTS] Fix `BarrierTaskContextSuite.successively sync with allGather and barrier` test case to be robust
    omit a3b91247b320 [SPARK-49981][CORE][TESTS] Fix `AsyncRDDActionsSuite.FutureAction result, timeout` test case to be robust
    omit 8d1cb76066e4 [SPARK-49987][SQL] Fix the error prompt when `seedExpression` is non-foldable in `randstr`
    omit 39112e4f2f8c [SPARK-49946][CORE] Require an error class in `SparkOutOfMemoryError`
    omit 2a1301133138 [SPARK-49966][SQL] Codegen Support for JsonToStructs(`from_json`)
    omit f3b2535d8d92 [SPARK-49970][SQL] Assign proper error condition for _LEGACY_ERROR_TEMP_2069
    omit 861b5e98e6e4 [SPARK-49948][PS][CONNECT] Add parameter "precision" to pandas on Spark box plot
    omit 0e75d19a736a [SPARK-49792][PYTHON][BUILD] Upgrade to numpy 2 for building and testing Spark branches
    omit add4a9c6bd5d [SPARK-49922][BUILD] Upgrade `sbt-assembly` to `2.3.0`
    omit 111473dec898 [SPARK-49964][BUILD] Remove `ws-rs-api` package
    omit 0f6bc3bce9c4 [SPARK-49973][INFRA] Upgrade python to 3.11 for non-python tests
    omit 856cfe7dc02b [SPARK-49969][BUILD] Simplify dependency management in YARN module
    omit 0b488280bef0 [SPARK-49911][SQL] Fix semantic of support binary equality
    omit 1269b35a7e6d [SPARK-49954][SQL] Codegen Support for SchemaOfJson (by Invoke & RuntimeReplaceable)
    omit 5f2bd5c10dc7 [SPARK-49959][SQL] Fix ColumnarArray.copy() to read nulls from the correct offset
    omit f2bd31453885 [SPARK-49916][SQL] Throw appropriate Exception for type mismatch between ColumnType and data type in some rows
    omit 14c01ebda5a8 [SPARK-49974][SQL] Move resolveRelations(...) out of the Analyzer.scala
    omit 2ccdabaa79ac [SPARK-49956] Disabled collations with collect_set expression
    omit c3176a79de62 [SPARK-49451][SQL][FOLLOW-UP] Improve duplicate key exception test
    omit 217e0da917c7 [SPARK-49965][BUILD] Upgrade ASM to 9.7.1
    omit 488f68090b22 [SPARK-49929][PYTHON][CONNECT] Support box plots
    omit 74aed77853ca [SPARK-49661][SQL] Implement trim collation hashing and comparison
    omit 96c495368098 [SPARK-49955][SQL] null value does not mean corrupted file when parsing JSON string RDD
    omit 0606512065b2 [SPARK-49864][SQL][FOLLOW-UP] Fix default suggestion for binary arithmetic overflow
    omit d53fa23e4a1e [SPARK-49952][SQL] Assign proper error condition for _LEGACY_ERROR_TEMP_1142
    omit eeb044ea4a5d [SPARK-49949][PS] Avoid unnecessary analyze task in `attach_sequence_column`
    omit 560748c2b148 [SPARK-49892][SQL] Assign proper error class for _LEGACY_ERROR_TEMP_1136
    omit a2ad4d4db6f1 [MINOR][CORE] Fix the regenerate command in `SparkThrowableSuite`
    omit 5b9b8da3ddec [SPARK-49904][SQL] Assign proper error condition for _LEGACY_ERROR_TEMP_2140
    omit 36b2a4e3eab7 [SPARK-49891][SQL] Assign proper error condition for _LEGACY_ERROR_TEMP_2271
    omit 1aae16089601 [SPARK-49928][PYTHON][TESTS] Refactor plot-related unit tests
    omit 1abfd490d072 [SPARK-49943][PS] Remove `timestamp_ntz_to_long` from `PythonSQLUtils`
    omit 54fd408316be [SPARK-49939][SQL] Codegen Support for json_object_keys (by Invoke & RuntimeReplaceable)
    omit 083f44d5e6c8 [MINOR][SQL] Improved broadcast timeout message prompt
    omit 62ade5f3f42b [SPARK-49924][SQL] Keep `containsNull` after `ArrayCompact` replacement
    omit ed4847ffe07e [SPARK-49937][INFRA] Ban call the method `SparkThrowable#getErrorClass`
    omit 1244c5a0f548 [SPARK-49766][SQL] Codegen Support for `json_array_length` (by `Invoke` & `RuntimeReplaceable`)
    omit 6734d4883e76 [SPARK-49932][CORE] Use `tryWithResource` release `JsonUtils#toJsonString` resources to avoid memory leaks
    omit 1fb3d57c1083 [SPARK-49935][BUILD] Exclude `spark-connect-shims` from `assembly` module
    omit cf657e5e2e49 [SPARK-49927][SS][PYTHON][TESTS] pyspark.sql.tests.streaming.test_streaming_listener to wait for longer
    omit 3ecfe8ef7b95 [SPARK-49930][SS] Ensure that socket updates are flushed on exception from the python worker
    omit 04ec55ed8106 [MINOR][PYTHON] Minor refine `LiteralExpression`
    omit c79e2d6370cc [SPARK-49925][SQL] Add tests for order by with collated strings
    omit 6d0b8389a20e [SPARK-49748][CORE][FOLLOWUP] Add `getCondition` and deprecate `getErrorClass` in `QueryCompilationErrorsSuite`
    omit 8e1d317307d9 [SPARK-49615] Bugfix: Make ML column schema validation conforms with spark config `spark.sql.caseSensitive`
    omit 5104d1d434c1 [SPARK-49915][SQL] Handle zeros and ones in ReorderAssociativeOperator
    omit ed0a63f93f57 Revert "[SPARK-49927][SS] pyspark.sql.tests.streaming.test_streaming_listener to wait longer"
    omit b93281ed0ed9 Revert "[SPARK-48567][PYTHON][TESTS][FOLLOW-UP] Make the query scope higher so finally can access to it"
    omit c0d396f862f4 Revert "[SPARK-49927][SS][PYTHON][TESTS][FOLLOW-UP] Fixes `q.lastProgress.batchId` to `q.lastProgress.progress.batchId`"
    omit cacd261e825a [SPARK-49927][SS][PYTHON][TESTS][FOLLOW-UP] Fixes `q.lastProgress.batchId` to `q.lastProgress.progress.batchId`
    omit 4666972a3190 [SPARK-48567][PYTHON][TESTS][FOLLOW-UP] Make the query scope higher so finally can access to it
    omit 2af653688c20 [SPARK-49927][SS] pyspark.sql.tests.streaming.test_streaming_listener to wait longer
    omit b056e0b12786 [SPARK-49569][BUILD][FOLLOWUP] Exclude `spark-connect-shims` from `sql/core` module
    omit e589ccd9c18e [SPARK-49920][INFRA] Install `R` for `ubuntu 24.04` when GA run `k8s-integration-tests`
    omit ab1315b95b8e [SPARK-49756][SQL] Postgres dialect supports pushdown datetime functions
    omit e693af0a0ada [SPARK-49748][CORE] Add `getCondition` and deprecate `getErrorClass` in `SparkThrowable`
    omit 38d66fde2eec [SPARK-49908][SQL] Assign proper error condition for _LEGACY_ERROR_TEMP_0044
    omit f003638f1015 [SPARK-49542][SQL] Partition transform exception evaluate error
    omit f0498f056ddf [SPARK-49549][SQL] Assign a name to the error conditions _LEGACY_ERROR_TEMP_3055, 3146
    omit d7772f27b8a8 [SPARK-49782][SQL] ResolveDataFrameDropColumns rule resolves UnresolvedAttribute with child output
    omit 0912217b38f4 [SPARK-48714][PYTHON][FOLLOW-UP] Skip tests if test class is not available
    omit ea60e935fb11 [SPARK-49918][CORE] Use read-only access to conf in `SparkContext` where appropriate
    omit 7e82e290fb05 [SPARK-49905][SQL][SS] Use different ShuffleOrigin for the shuffle required from stateful operators
    omit 6ed4bdf7fdba [MINOR][PYTHON][TESTS] Reduce the python worker error log of `test_toDF_with_schema_string`
    omit f69d03e0cf45 [SPARK-43838][SQL][FOLLOWUP] Replace `HashSet` with `HashMap` to improve performance of `DeduplicateRelations`
    omit fed9a8da3d41 [SPARK-49569][BUILD][FOLLOWUP] Adds `scala-library` maven dependency to the `spark-connect-shims` module to fix Maven build errors
    omit 97a5aa6ef7ca [SPARK-49873][SQL] Assign proper error class for _LEGACY_ERROR_TEMP_1325
    omit b1ff7672cba1 [SPARK-49857][SQL] Add storageLevel to Dataset localCheckpoint API
    omit b565a8df9fef [SPARK-49418][CONNECT][SQL] Shared Session Thread Locals
    omit 52538f0d9bd1 [SPARK-49909][SQL] Fix the pretty name of some expressions
    omit 135cbc6e89d7 [SPARK-49564][SQL] Add SQL pipe syntax for set operations
    omit 5e27eec38475 [SPARK-49863][SQL] Fix NormalizeFloatingNumbers to preserve nullability of nested structs
    omit c1f18a00bb88 [SPARK-49022][CONNECT][SQL][FOLLOW-UP] Parse unresolved identifier to keep the behavior same
    omit 5f64e80843ae [SPARK-49895][SQL] Improve error when encountering trailing comma in SELECT clause
    omit 80ae411b178b [SPARK-49569][CONNECT][SQL] Add shims to support SparkContext and RDD
    omit 345a2be41320 [SPARK-49906][SQL] Introduce and use CONFLICTING_DIRECTORY_STRUCTURES error for PartitioningUtils
    omit 92e79e36c79c [SPARK-49901][BUILD] Upgrade dropwizard metrics to 4.2.28
    omit 83f732cb1c3f [SPARK-49903][BUILD] Upgrade `Parquet` to 1.14.3
    omit ef142c4f204f [SPARK-49898][CORE] Fix documentation and default for event log task metrics accumulator logging flag from SPARK-42204
    omit 78135dc204e4 [SPARK-49889][PYTHON] Add argument `trim` for functions`trim/ltrim/rtrim`
    omit 51af177c9ef0 [SPARK-49564][SQL] Add SQL pipe syntax for the JOIN operator
    omit c6b09c0b71e7 [SPARK-49894][PYTHON][CONNECT] Refine the string representation of column field operations
    omit d8aca18d98a3 [SPARK-49852][SQL] Fix deadlock caused by explain string generation
    omit 724e1bf6b5bd [SPARK-49896][K8S][DOCS] Drop K8s v1.28 Support
    omit 5132ab1f7b33 [SPARK-49877][SQL] Change classifyException function signature: add isRuntime argument
    omit be546ff5cd08 [SPARK-49890][PYTHON] Extract the preparation of df.sample to parent class
    omit a177ed1227cc Revert "[SPARK-49808][SQL] Fix a deadlock in subquery execution due o lazy vals"
    omit 9ce05648a6ce [SPARK-49773][SQL] Uncaught Java exception from `make_timestamp()` with bad timezone
    omit 06a8f6b6d6f6 [SPARK-49744][SS][PYTHON] Implement TTL support for ListState in TransformWithStateInPandas
    omit 06c70ba6c80c [MINOR][SQL] Move iterator.hasNext into try block in executeTask
    omit 37f2966b6423 [SPARK-49866][SQL] Improve the error message for describe table with partition columns
    omit 3e69b40300af [SPARK-49683][SQL] Block trim collation
    omit d8c04cf2fb75 [SPARK-49836][SQL][SS] Fix possibly broken query when window is provided to window/session_window fn
    omit 0c653db0b2fe [SPARK-49879][CORE] Move `TransportCipherUtil` to a separate file to eliminate Java compilation warnings
    omit fcda935bc48e [SPARK-49864][SQL] Improve message of BINARY_ARITHMETIC_OVERFLOW
    omit 4cf9d14d84c0 [SPARK-49806][SQL][CONNECT] Remove redundant `blank space` after `show` in `Scala` and `Connect` clients
    omit 13e5504115c5 [SPARK-49874][SQL] Remove trim and ltrim collation specifiers
    omit f694ea82dbc8 [SPARK-49867][SQL] Improve the error message when index is out of bounds when calling GetColumnByOrdinal
    omit 8a51ca7d66eb [SPARK-47702][CORE] Remove Shuffle service endpoint from the locations list when RDD block is removed form a node
    omit e59483c9e28c [SPARK-49868][K8S] Fix race condition in executor failure tracking
    omit 780759b70245 [SPARK-48357][SQL][FOLLOW-UP] Add doc comment for LOOP statement exec node
    omit 58f0f3be2204 [SPARK-49668][SQL] Implement collation key support for trim collation
    omit e1dbeb4de862 [SPARK-49871][CORE][TESTS] Fix `(Ssl)CoarseGrainedExecutorBackendSuite` to reduce test resource and increase timeout
    omit 3dfedf69cc39 [SPARK-49842][BUILD] Add `byte-buddy` dependency for modules that depend on `mockito-core` to ensure `sbt test` uses the correct `byte-buddy` with Java 21
    omit bd3e2eb02170 [SPARK-49751][CONNECT] Fix deserialization of SparkListenerConnectServiceStarted event
    omit de9b9c85e287 [SPARK-49870][PYTHON] Add Python 3.13 support in Spark Classic
    omit 98da5e1ab4ac [MINOR][PYTHON][TESTS] Skip test_artifact if grpc isn't installed
    omit a38505c9455f [SPARK-49869][INFRA] Add NumPy in Python 3.13 image
    omit 96666d49feb3 [SPARK-49859][CONNECT] Replace multiprocessing.ThreadPool with ThreadPoolExecutor
    omit 29312bc0c971 [SPARK-49824][SS][CONNECT] Improve logging in SparkConnectStreamingQueryCache
    omit b9a327479422 [SPARK-46647][INFRA] Add `unittest-xml-reporting` into Python 3.12 image
    omit 901bb33c23b3 [SPARK-49860][PYTHON][INFRA] Add `Python 3.13` Daily Python Github Action job
    omit 68fd17d09e83 [SPARK-49861][INFRA] Add `Python 3.13` to Infra docker image
    omit 38f067dfcef9 [SPARK-49358][SQL] Mode expression for map types with collated strings
    omit 036db74b97f8 [SPARK-47341][SQL] Fix inaccurate documentation of RuntimeConfig#get
    omit c1ecab4d77b4 [SPARK-49541][BUILD] Upgrade log4j2 to 2.24.1
    omit 216f761bcb12 [SPARK-48357][SQL] Support for LOOP statement
    omit ce5762649435 [SPARK-49837][SQL][TESTS] Add more tests for NULLIF function
    omit d97acc17dd0b [SPARK-49853][SQL][TESTS] Increase test timeout of `PythonForeachWriterSuite` to `60s`
    omit 18dbaa5a070c [SPARK-49560][SQL] Add SQL pipe syntax for the TABLESAMPLE operator
    omit 077a31989c99 [SPARK-49843][SQL] Fix change comment on char/varchar columns
    omit 3551a9ee6d38 [SPARK-49845][CORE] Make `appArgs` and `environmentVariables` optional in REST API
    omit 3093ad68d2a3 [MINOR] Fix a typo in First aggregate expression
    omit 97e9bb3ac4b6 [SPARK-48700][SQL] Mode expression for complex types (all collations)
    omit c0a1ea2a4c42 [SPARK-49795][CORE][SQL][SS][DSTREAM][ML][MLLIB][K8S][YARN][EXAMPLES] Clean up deprecated Guava API usage
    omit 8d0f6fb90221 [SPARK-49826][BUILD] Upgrade jackson to 2.18.0
    omit da106f86260b [SPARK-49840][INFRA] Use `MacOS 15` in `build_maven_java21_macos14.yml`
    omit 123361137bbe [SPARK-49732][CORE][K8S] Spark deamons should respect `spark.log.structuredLogging.enabled` conf
    omit d68048b06a04 [SPARK-49833][K8S] Support user-defined annotations for OnDemand PVCs
    omit a7fa2700e0f0 [SPARK-48196][SQL] Turn QueryExecution lazy val plans into LazyTry
    omit 3065dd92ab8f [SPARK-49561][SQL] Add SQL pipe syntax for the PIVOT and UNPIVOT operators
    omit dbfa909422ad [SPARK-49816][SQL] Should only update out-going-ref-count for referenced outer CTE relation
    omit 97ae372634b1 [SPARK-49819] Disable CollapseProject for correlated subqueries in projection over aggregate correctly
    omit c54c017e9309 [SPARK-49666][SQL] Add feature flag for trim collation feature
    omit d85e7bc0beb4 [SPARK-49811][SQL] Rename StringTypeAnyCollation
    omit 885c3fac7246 [SPARK-49823][SS] Avoid flush during shutdown in rocksdb close path
    omit 039fd13eacb1 [SPARK-49749][CORE] Change log level to debug in BlockManagerInfo
    omit 8dfecc1463ff [SPARK-49434][SPARK-49435][CONNECT][SQL] Move aggregators to sql/api
    omit 47d2c9ca064e [SPARK-49712][SQL] Remove encoderFor from connect-client-jvm
    omit 550c2071bf8e [SPARK-49822][SQL][TESTS] Update postgres docker image to 17.0
    omit 4c12c78801b8 [SPARK-49814][CONNECT] When Spark Connect Client starts, show the `spark version` of the `connect server`
    omit f9a2077fd32f [SPARK-49810][PYTHON] Extract the preparation of `DataFrame.sort` to parent class
    omit 0c1905951f8c [SPARK-49820][PYTHON] Change `raise IOError` to `raise OSError`
    omit d813f5467e93 [SPARK-49417][CONNECT][SQL] Add Shared StreamingQueryManager interface
    omit 4d70954b1aeb [SPARK-49817][BUILD] Upgrade `gcs-connector` to `2.2.25`
    omit b6681fbf32fa [SPARK-49787][SQL] Cast between UDT and other types
    omit 6dc628c31cdf [SPARK-49809][BUILD] Use `sbt.IO` in `SparkBuild.scala` to avoid naming conflicts with `java.io.IO` in Java 23
    omit dd692e90b738 [SPARK-49801][FOLLOWUP][INFRA] Update `pandas` to 2.2.3 in `pages.yml` too
    omit d7abddc454ff [SPARK-49808][SQL] Fix a deadlock in subquery execution due to lazy vals
    omit 9b739d415cd5 [SPARK-49757][SQL] Support IDENTIFIER expression in SET CATALOG statement
    omit f18c4e7722b4 [SPARK-49805][SQL][ML] Remove private[xxx] functions from `function.scala`
    omit 5d701f2d5add [SPARK-49804][K8S] Fix to use the exit code of executor container always
    omit 27d4a77f2a8c [SPARK-49801][PYTHON][PS][BUILD] Update `pandas` to 2.2.3
    omit 488c3f604490 [SPARK-49776][PYTHON][CONNECT] Support pie plots
    omit 09b7aa67ce64 [SPARK-49803][SQL][TESTS] Increase `spark.test.docker.connectionTimeout` to 10min
    omit fc9d421a2345 [SPARK-49211][SQL][FOLLOW-UP] Support catalog in QualifiedTableName
    omit 339dd5b93316 [SPARK-49791][SQL] Make DelegatingCatalogExtension more extendable
    omit 54e62a158ead [SPARK-49800][BUILD][K8S] Upgrade `kubernetes-client` to 6.13.4
    omit 624eda5030eb [SPARK-49444][SQL] Modified UnivocityParser to throw runtime exceptions caused by ArrayIndexOutOfBounds with more user-oriented messages
    omit 87b5ffb22082 [SPARK-49797][INFRA] Align the running OS image of `maven_test.yml` to `ubuntu-latest`
    omit 218051a566c7 [MINOR][SQL][TESTS] Use `formatString.format(value)` instead of `value.formatted(formatString)`
    omit 7b20e5841a85 [SPARK-49286][CONNECT][SQL] Move Avro/Protobuf functions to sql/api
    omit a116a5bf708d [SPARK-49416][CONNECT][SQL] Add Shared DataStreamReader interface
    omit fe1cf3200223 [SPARK-49656][SS] Add support for state variables with value state collection types and read change feed options
    omit 913a0f7813c5 [SPARK-49784][PYTHON][TESTS] Add more test for `spark.sql`
    omit 562977928772 [SPARK-49786][K8S] Lower `KubernetesClusterSchedulerBackend.onDisconnected` log level to debug
    omit c0984e70469d [SPARK-49609][PYTHON][TESTS][FOLLOW-UP] Avoid import connect modules when connect dependencies not installed
    omit 80d6651cf6a1 [SPARK-48195][FOLLOWUP] Accumulator reset() no longer needed in CollectMetricsExec.doExecute()
    omit 09209f0ff503 [SPARK-49775][SQL][FOLLOW-UP] Use SortedSet instead of Array with sorting
    omit 1f2e7b87db76 [SPARK-49731][K8S] Support K8s volume `mount.subPathExpr` and `hostPath` volume `type`
    omit 983f6f434af3 [SPARK-49746][BUILD] Upgrade Scala to 2.13.15
    omit d23023202185 [SPARK-49745][SS] Add change to read registered timers through state data source reader
    omit 0ccf53ae6faa [SPARK-49609][PYTHON][FOLLOWUP] Correct the typehint for `filter` and `where`
    omit c362d500acba [SPARK-49775][SQL][TESTS] Make tests of `INVALID_PARAMETER_VALUE.CHARSET` deterministic
    omit e1b2ac55b4b9 [SPARK-49767][PS][CONNECT] Refactor the internal function invocation
    omit 7f0ecd4221a7 [SPARK-49764][PYTHON][CONNECT] Support area plots
    omit 46c5accaa551 [SPARK-49609][PYTHON][TESTS][FOLLOW-UP] Skip Spark Connect tests if dependencies are not found
    omit 5134c6889673 [SPARK-49765][DOCS][PYTHON] Adjust documentation of "spark.sql.pyspark.plotting.max_rows"
    omit 9aa11d1ee480 [SPARK-49772][SS] Remove ColumnFamilyOptions and add configs directly to dbOptions in RocksDB
    omit e2d2ab510632 [SPARK-49552][PYTHON] Add DataFrame API support for new 'randstr' and 'uniform' SQL functions
    omit a4fb6cbfda22 [SPARK-49743][SQL] OptimizeCsvJsonExpr should not change schema fields when pruning GetArrayStructFields
    omit 828b1f94734a [SPARK-49463] Support ListState for TransformWithStateInPandas
    omit 0c234bb1a68c [SPARK-49369][CONNECT][SQL] Add implicit Column conversions
    omit 5fb0ff9e10b1 [SPARK-49282][CONNECT][SQL] Create a shared SparkSessionBuilder interface
    omit 29ed2729492a [SPARK-49688][CONNECT] Fix a data race between interrupt and execute plan
    omit 0a7b98532fd2 [SPARK-49585][CONNECT] Replace executions map in SessionHolder with operationID set
    omit afe8bf945e1a [SPARK-49771][PYTHON] Improve Pandas Scalar Iter UDF error when output rows exceed input rows
    omit 55d0233d19cc [SPARK-49713][PYTHON][FOLLOWUP] Make function `count_min_sketch` accept long seed
    omit dedf5aa91827 [SPARK-49750][DOC] Mention delegation token support in K8s mode
    omit 73d6bd7c35b5 [SPARK-49630][SS] Add flatten option to process collection types with state data source reader
    omit 982028ea7fc6 [SPARK-49609][PYTHON][CONNECT] Add API compatibility check between Classic and Connect
    omit 6bdd151d5775 [SPARK-49694][PYTHON][CONNECT] Support scatter plots
    omit 438a6e7782ec [SPARK-49753][BUILD] Upgrade ZSTD-JNI to 1.5.6-6
    omit 64ea50e87c70 [SPARK-49607][PYTHON] Update the sampling approach for sampled based plots
    omit 35e5d290deee [SPARK-49760][YARN] Correct handling of `SPARK_USER` env variable override in app master
    omit 742265ebb742 [SPARK-49429][CONNECT][SQL] Add Shared DataStreamWriter interface
    omit 94d288e08f2b [MINOR][PYTHON][DOCS] Fix the docstring of `to_timestamp`
    omit 1086256a81f1 [SPARK-49415][CONNECT][SQL] Move SQLImplicits to sql/api
    omit 3b5c1d6baeb2 [SPARK-49747][CONNECT] Migrate connect/ files to structured logging
    omit fec1562b0ea0 [SPARK-49755][CONNECT] Remove special casing for avro functions in Connect
    omit e1637e3fbe0a [SPARK-48712][SQL][FOLLOWUP] Check whether input is valid utf-8 string or not before entering fast path
    omit 44ec70f5103f [SPARK-49626][PYTHON][CONNECT] Support horizontal and vertical bar plots
    omit d2e8c1cb60e3 [SPARK-48195][CORE] Save and reuse RDD/Broadcast created by SparkPlan
    omit 3c81f076ab9c [SPARK-49653][SQL] Single join for correlated scalar subqueries
    omit 0eeb61fb64e0 [SPARK-49734][PYTHON] Add `seed` argument for function `shuffle`
    omit 719b57a32e0f [SPARK-49752][YARN] Remove workaround for YARN-3350
    omit 067f8f188eb2 [SPARK-48355][SQL][TESTS][FOLLOWUP] Enable a SQL Scripting test in ANSI and non-ANSI modes
    omit b6420969b5df [SPARK-49741][DOCS] Add `spark.shuffle.accurateBlockSkewedFactor` to config docs page
    omit 4f640e2485d2 [SPARK-49495][DOCS] Document and Feature Preview on the master branch via Live GitHub Pages Updates
    omit 19906468d145 Revert "[SPARK-49495][DOCS] Document and Feature Preview on the master branch via Live GitHub Pages Updates"
    omit bbbc05cbf971 [SPARK-49495][DOCS] Document and Feature Preview on the master branch via Live GitHub Pages Updates
    omit 0b05b1aa72ce [SPARK-48782][SQL][TESTS][FOLLOW-UP] Enable ANSI for malformed input test in ProcedureSuite
    omit fc8b94544163 [SPARK-49460][SQL] Followup: fix potential NPE risk
    omit f235bab24761 [SPARK-49451][FOLLOW-UP] Add support for duplicate keys in from_json(_, 'variant')
    omit 70bd606cc865 [SPARK-49641][DOCS] Include `table_funcs` and `variant_funcs` in the built-in function list doc
    omit bdea09128760 [SPARK-49557][SQL] Add SQL pipe syntax for the WHERE operator
    omit f76a9b1135e7 [SPARK-49738][SQL] Endswith bug fix
    omit f3785fadec30 [SPARK-49737][SQL] Disable bucketing on collated columns in complex types
    omit 22a7edce0a7c [SPARK-49531][PYTHON][CONNECT] Support line plot with plotly backend
    omit 3d8c078ddefe [SPARK-49704][BUILD] Upgrade `commons-io` to 2.17.0
    omit bb8294c64990 [SPARK-49729][SQL][DOCS] Forcefully check `usage` and correct the non-standard writing of 4 expressions
    omit 4d97574425e6 [SPARK-49733][SQL][DOCS] Delete `ExpressionInfo[between]` from `gen-sql-api-docs.py` to avoid duplication
    omit 46b0210edb4e [SPARK-49699][SS] Disable PruneFilters for streaming workloads
    omit b37863d23271 [MINOR][FOLLOWUP] Fix rat check for .nojekyll
    omit c009cd061c49 [SPARK-49392][SQL][FOLLOWUP] Catch errors when failing to write to external data source
    omit 6352c12f607b [MINOR][INFRA] Disable 'pages build and deployment' action
    omit d4665fa1df71 [SPARK-49677][SS] Ensure that changelog files are written on commit and forceSnapshot flag is also reset
    omit a5ac80af8e94 [SPARK-49713][PYTHON][CONNECT] Make function `count_min_sketch` accept number arguments
    omit ca726c10925a [SPARK-49721][BUILD] Upgrade `protobuf-java` to 3.25.5
    omit 04455797bfb3 [SPARK-49720][PYTHON][INFRA] Add a script to clean up PySpark temp files
    omit 6d1815eceea2 [SPARK-49718][PS] Switch `Scatter` plot to sampled data
    omit 92cad2abd54e [SPARK-49716][PS][DOCS][TESTS] Fix documentation and add test of barh plot
    omit f0fb0c89ec29 [SPARK-49719][SQL] Make `UUID` and `SHUFFLE` accept integer `seed`
    omit 94dca78c128f [SPARK-49693][PYTHON][CONNECT] Refine the string representation of `timedelta`
    omit 398457af5987 [SPARK-49422][CONNECT][SQL] Add groupByKey to sql/api
    omit 4068fbcc0de5 [SPARK-49717][SQL][TESTS] Function parity test ignore private[xxx] functions
    omit a060c236d314 [SPARK-49667][SQL] Disallowed CS_AI collators with expressions that use StringSearch
    omit ac34f1de92c6 [SPARK-48280][SQL][FOLLOW-UP] Add expressions that are built via expressionBuilder to Expression Walker
    omit 492d1b14c0d1 [SPARK-48782][SQL] Add support for executing procedures in catalogs
    omit 3bdf146bbee5 [SPARK-49611][SQL][FOLLOW-UP] Fix wrong results of collations() TVF
    omit f3c8d26eb0c3 Revert "[SPARK-49422][CONNECT][SQL] Add groupByKey to sql/api"
    omit 8861f0f9af3f Revert "[SPARK-49495][DOCS] Document and Feature Preview on the master branch via Live GitHub Pages Updates"
    omit 376382711e20 Revert "[SPARK-49495][DOCS][FOLLOWUP] Fix Pandoc installation for GitHub Pages publication action"
    omit 58d73fe8e7cb Revert "[SPARK-49495][DOCS][FOLLOWUP] Enable GitHub Pages settings via .asf.yml"
    omit af45902d33c4 [SPARK-49422][CONNECT][SQL] Add groupByKey to sql/api
    omit 3b34891e5b9c [SPARK-49684][CONNECT] Remove global locks from session and execution managers
    omit db8010b4c8be [SPARK-49568][CONNECT][SQL] Remove self type from Dataset
    omit 5c48806a2941 [SPARK-49688][CONNECT][TESTS] Fix a sporadic `SparkConnectServiceSuite` failure
    omit 669e63a34012 [SPARK-49673][CONNECT] Increase CONNECT_GRPC_ARROW_MAX_BATCH_SIZE to 0.7 * CONNECT_GRPC_MAX_MESSAGE_SIZE
    omit 25d6b7a280f6 [SPARK-49692][PYTHON][CONNECT] Refine the string representation of literal date and datetime
    omit a6f6e07b7031 [SPARK-48939][AVRO] Support reading Avro with recursive schema reference
    omit fbf81ebaef49 [SPARK-47263][SQL] Assign names to the legacy conditions _LEGACY_ERROR_TEMP_13[44-46]
    omit ed3a9b1aa929 [SPARK-49691][PYTHON][CONNECT] Function `substring` should accept column names
    omit b86e5d2ab1fb [SPARK-49495][DOCS][FOLLOWUP] Enable GitHub Pages settings via .asf.yml
    omit 7de71a2ec78d [SPARK-49495][DOCS][FOLLOWUP] Fix Pandoc installation for GitHub Pages publication action
    omit 4590538df095 [SPARK-49682][BUILD] Upgrade joda-time to 2.13.0
    omit fd8e99b9df55 [SPARK-49249][SPARK-49320] Add new tag-related APIs in Connect back to Spark Core
    omit 31cd6991558c [SPARK-49424][CONNECT][SQL] Consolidate Encoders.scala
    omit b1807095bef9 [SPARK-49495][DOCS] Document and Feature Preview on the master branch via Live GitHub Pages Updates
    omit a7f191ba5947 [SPARK-49640][PS] Apply reservoir sampling in `SampledPlotBase`
    omit 6fc176f4f34d [SPARK-49413][CONNECT][SQL] Create a shared RuntimeConfig interface
    omit c38844c9ecc6 [SPARK-49687][SQL] Delay sorting in `validateAndMaybeEvolveStateSchema`
    omit 6393afada79a [SPARK-49505][SQL] Create new SQL functions "randstr" and "uniform" to generate random strings or numbers within ranges
    omit f586ffbf47ee [SPARK-49498][SQL][TESTS] Check differences between SR_AI and SR_Latn_AI collations
    omit dd8d1270679c [SPARK-49650][SQL][DOCS] Updates references to deprecated Hive JDBC client params in tests and docs
    omit db8468105bdb [SPARK-49659][SQL] Add a nice user-facing error for scalar subqueries inside VALUES clause
    omit 370453adba17 [SPARK-49678][CORE] Support `spark.test.master` in `SparkSubmitArguments`
    omit 294af6e31639 [SPARK-49680][PYTHON] Limit `Sphinx` build parallelism to 4 by default
    omit 8d78f5b02c25 [DOCS][MINOR] Remove spark-sql-api dependency from connect docs
    omit 2113f109b8d7 [SPARK-49611][SQL] Introduce TVF `collations()` & remove the `SHOW COLLATIONS` command
    omit 738db079c0b6 [SPARK-49646][SQL] fix subquery decorrelation for union/set operations when parentOuterReferences has references not covered in collectedChildOuterReferences
    omit b4f4d9b7a7d4 [SPARK-49655][BUILD] Link `python3` to `python3.9` in `spark-rm` Docker image
    omit 931ab065df39 [SPARK-48824][SQL] Add Identity Column SQL syntax
    omit 1346531ccc6e [SPARK-48355][SQL][TESTS][FOLLOWUP] Disable a test case failing on non-ANSI mode
    omit fa6a0786bb4b Revert "[SPARK-49531][PYTHON][CONNECT] Support line plot with plotly backend"
    omit 017b0ea71e03 [SPARK-49556][SQL] Add SQL pipe syntax for the SELECT operator
    omit 2250b35be6a2 [SPARK-49649][DOCS] Make `docs/index.md` up-to-date for 4.0.0
    omit df0e34c5a1c3 [SPARK-49648][DOCS] Update `Configuring Ports for Network Security` section with JWS
    omit d3eb99f79e50 [SPARK-49647][TESTS] Change SharedSparkContext so that its SparkConf loads defaults
    omit 08a26bb56cfb [SPARK-48779][SQL][TESTS] Improve collation support testing - add golden files
    omit bbe6d573d95e [SPARK-49392][SQL] Catch errors when failing to write to external data source
    omit b9b64fe37e93 [SPARK-49244][SQL] Further exception improvements for parser/interpreter
    omit f92e9489fb23 [SPARK-49234][BUILD][FOLLOWUP] Add `LICENSE-xz.txt` to `licenses-binary` folder
    omit 9fc58aa4c075 [SPARK-49488][SQL] MySQL dialect supports pushdown datetime functions
    omit 5533c81e3453 [SPARK-48355][SQL] Support for CASE statement
    omit aa54ed17832f [SPARK-48549][SQL][DOCS][FOLLOWUP] Add migration guide for SQL function `sentences` behavior changes
    omit 319e7cc7d0e7 [SPARK-49628][SQL] ConstantFolding should copy stateful expression before evaluating
    omit e7cf246fb763 [SPARK-49624][BUILD] Upgrade `aircompressor` to 2.0.2
    omit 61814876b26c [SPARK-43354][PYTHON][TESTS] Re-enable `test_create_dataframe_from_pandas_with_day_time_interval` in PyPy3.9
    omit 23e61f6b1845 [SPARK-49621][SQL][TESTS] Remove the flaky `EXEC IMMEDIATE STACK OVERFLOW` test case
    omit f69b518446e2 [SPARK-49620][INFRA] Fix `spark-rm` and `infra` docker files to create `pypy3.9` links
    omit 3b8dddac65bc [SPARK-49531][PYTHON][CONNECT] Support line plot with plotly backend
    omit 875def391665 [SPARK-49081][SQL][DOCS] Add data source options docs of `Protobuf`
    omit d2d293e3fb57 [SPARK-49598][K8S] Support user-defined labels for OnDemand PVCs
    omit 317eddb7390c [SPARK-49606][PS][DOCS] Improve documentation of Pandas on Spark plotting API
    omit 98f0d9f32322 [SPARK-49605][SQL] Fix the prompt when `ascendingOrder` is `DataTypeMismatch` in `SortArray`
    omit 1f24b2d72ed6 [SPARK-44811][BUILD] Upgrade Guava to 33.2.1-jre
    omit 1a0791d006e2 [SPARK-49261][SQL] Don't replace literals in aggregate expressions with group-by expressions
    omit bc54eac57e5c [SPARK-42846][SQL] Remove error condition _LEGACY_ERROR_TEMP_2011
    omit c5c880e690c3 [SPARK-49591][SQL] Add Logical Type column to variant readme
    omit 19aad9ee36ed [SPARK-49578][SQL][TESTS][FOLLOWUP] Regenerate Java 21 golden file for `postgreSQL/float4.sql` and `postgreSQL/int8.sql`
    omit 8023504e69fd [SPARK-49594][SS] Adding check on whether columnFamilies were added or removed to write StateSchemaV3 file
    omit 0f4d289b7932 [SPARK-48906][SQL] Introduce `SHOW COLLATIONS LIKE ...` syntax to show all collations
    omit b466f32077e3 [SPARK-49544][CONNECT] Replace coarse-locking in SparkConnectExecutionManager with ConcurrentMap
    omit 3cb8d6e59999 [SPARK-49548][CONNECT] Replace coarse-locking in SparkConnectSessionManager with ConcurrentMap
    omit 07f5b2c1c5ff [SPARK-49155][SQL][SS] Use more appropriate parameter type to construct `GenericArrayData`
    omit 591a60df788a [SPARK-49602][BUILD] Fix `assembly/pom.xml` to use `{project.version}` instead of `{version}`
    omit e037953d4c58 [SPARK-48549][SQL][PYTHON] Improve SQL function `sentences`
    omit a9502d42a500 [SPARK-49310][BUILD] Upgrade `Parquet` to 1.14.2
    omit d72e8f9e1263 [SPARK-43838][SQL][FOLLOWUP] Improve DeduplicateRelations performance
    omit c5fd509ad3c0 [SPARK-49085][CONNECT][BUILD][FOLLOWUP] Remove the erroneous `type` definition for `spark-protobuf` from `sql/connect/server/pom.xml`
    omit 480d5a73bbd3 [SPARK-47259][SQL] Assign names to error conditions for interval errors
    omit aee9f60b3966 [SPARK-49600][PYTHON] Remove `Python 3.6 and older`-related logic from `try_simplify_traceback`
    omit 7b456e4dfb34 [SPARK-49596][SQL] Improve performance of `FormatString`
    omit 14de06e19d38 [SPARK-49574][CONNECT][SQL] ExpressionEncoder tracks the AgnosticEncoder that created it
    omit e63b5601c1bd [SPARK-49595][CONNECT][SQL] Fix `DataFrame.unpivot/melt` in Spark Connect Scala Client
    omit 70482f6f82b1 [SPARK-49599][BUILD] Upgrade snappy-java to 1.1.10.7
    omit cc6d6f17bdee [SPARK-49519][SQL] Merge options of table and relation when constructing FileScanBuilder
    omit e28c33bcf784 [SPARK-49578][SQL] Remove the ANSI config suggestion in CAST_INVALID_INPUT and CAST_OVERFLOW
    omit 44d08cb97eff [MINOR][CONNECT] Fix missing request metadata in RPC call
    omit 50955cb934ff [SPARK-48358][SQL][TESTS][FOLLOWUP] Use `condition` instead of `errorClass` in `SqlScriptingInterpreterSuite`
    omit 3d7ddfc83a5f [SPARK-49398][SQL] Improve the error for parameters in the query of CACHE TABLE and CREATE VIEW
    omit da893223353e [SPARK-48358][SQL] Support for REPEAT statement
    omit 6d8b842784ae [SPARK-49593][SS] Throw RocksDB exception to the caller on DB close if an error is seen
    omit fdf3b096f060 [SPARK-49553][PYTHON][DOCS] Remove the experimental API notes for pandas related functions
    omit 782d0ca7dd01 [SPARK-49582][PYTHON][CONNECT] Improve "dispatch_window_method" utility and docstring
    omit ea4b82e97d94 [MINOR][SQL][TESTS] Check `sqlState` in `checkError()`
    omit 8f662722235a [SPARK-49443][SQL][FOLLOWUP] Fix formatting of a comment for `to_variant_object()`
    omit e0a2c747644c [SPARK-49579][SQL][TESTS][FOLLOWUP] Use `condition` instead of `errorClass` in `SparkThrowableSuite` and in `DateTimeFormatterHelperSuite`
    omit 0fd91c7805b8 [SPARK-49579][SQL][TESTS] Rename `errorClass` to `condition` in `checkError()`
    omit 3709c2e3395b [SPARK-49443][SQL][PYTHON] Implement to_variant_object expression and make schema_of_variant expressions print OBJECT for for Variant Objects
    omit 7355864e2cf8 [SPARK-49583][SQL] Define the error sub-condition `SECONDS_FRACTION` for invalid seconds fraction pattern
    omit ab7aea144da4 [MINOR][DOCS] Fix scaladoc for `FlatMapGroupsInArrowExec` and `FlatMapCoGroupsInArrowExec`
    omit e918fb65f9bc [SPARK-49085][FOLLOWUP] Update the scope of `spark-protobuf`
    omit 726999f098d9 [SPARK-49575][SS] Add logging for lock release only if acquiredThreadInfo is not null
    omit 5f2d839ac0c5 [SPARK-49538][SQL][TESTS] Detect unused error message parameters
    omit f5d1f35adb60 [SPARK-49576][INFRA] Upload Python logs in CI
    omit 3633cfbf2d13 [SPARK-49539][SS] Update internal col families start identifier to a different one
    omit 8732528216fc [SPARK-49191][SS] Add support for reading transformWithState map state variables with state data source reader
    omit c4a396b50392 [SPARK-49536] Handle error in python streaming data source record prefetching
    omit a69c5eaa7f53 [SPARK-49567][PYTHON] Use `classic` instead of `vanilla` from PySpark code base
    omit b0c5642f696e [SPARK-49287][SPARK-49432][CONNECT][SQL] Move streaming classes to sql/api
    omit 14e490854f98 [SPARK-49551][SS] Improve RocksDB log for replayChangelog
    omit 20643bb538c3 [SPARK-49246][SQL][TESTS][FOLLOWUP] Fix flaky CollationSuite
    omit 8bc26d0236b4 [SPARK-49534][CORE][FOLLOWUP] Update outdated code comments related to the `AbstractCommandBuilder#isJarAvailable` method
    omit 23d22b1ef641 [SPARK-49534][CORE] No longer prepend `sql/hive`and `sql/hive-thriftserver` when `spark-hive_xxx.jar` is not in the classpath
    omit dc3333bcc599 [SPARK-49501][SQL] Fix double-escaping of table location
    omit 532aaafec4f1 [SPARK-49006] Implement purging for OperatorStateMetadataV2 and StateSchemaV3 files
    omit a3b918eb94c1 [SPARK-49540][PS] Unify the usage of `distributed_sequence_id`
    omit 37b39b41d07c [SPARK-49246][SQL][FOLLOW-UP] The behavior of SaveAsTable should not be changed by falling back to v1 command
    omit 3ed5a4d9dd02 [SPARK-48823][DOCS] Improve clarity in `lag` docstring
    omit c159033ee146 [SPARK-48376][SQL][FOLLOWUP] Add ITERATE statement fixes
    omit 29129645942a [MINOR][TEST] Fix the spell mistake
    omit f9a8ca54e7ad [SPARK-42204][CORE] Add option to disable redundant logging of TaskMetrics internal accumulators in event logs
    omit 3a4ea84f9928 [SPARK-49502][CORE] Avoid NPE in SparkEnv.get.shuffleManager.unregisterShuffle
    omit d5b79c0cfdc3 [SPARK-48348][SPARK-48376][FOLLOWUP][SQL] Replace parseScript with runSqlScript in SQL Scripting Interpreter test suite
    omit b5e345caa875 [SPARK-49533][CORE][TESTS] Change default `ivySettings` in the `IvyTestUtis#withRepository` function to use `.ivy2.5.2` as the Default Ivy User Dir
    omit 62cdc564d5b3 [SPARK-49532][DOCS][PS] Improve documentation of "plotting.sample_ratio" option
    omit 68718fe6fda8 [SPARK-49284][CONNECT][SQL] Create a shared Catalog interface
    omit 13a102da17be [MINOR][INFRA] Upgrade actions in test_report.yml workflow
    omit 002bf8735938 [SPARK-49511][SQL][FOLLOW-UP] Fix `@see` links to recover Scaladoc build
    omit 5d1d44ff1201 [SPARK-49467][SS] Add support for state data source reader and list state
    omit fdeb288f955b [SPARK-49511][SQL] Apply formatting rules to sql/api
    omit 62344cdac156 [SPARK-47948][PYTHON][FOLLOWUP] Bump Pandas to 2.0.0 in `requirements.txt`
    omit 20f2cb0365b5 [SPARK-47737][PYTHON][FOLLOWUP] Bump PyArrow to 10.0.0 in `requirements.txt`
    omit e9a6230ae09c [SPARK-44239][SQL][FOLLOWUP] Do not disable vector memory optimization when hugeVectorThreshold=0 to align its document
    omit 9269a0bfed56 [SPARK-49525][SS][CONNECT] Minor log improvement to Server Side Streaming Query ListenerBus Listener
    omit e49e31d27c27 [SPARK-49307][CONNECT][SQL] Add Kryo serialization to agnostic encoder framework
    omit 23bea28acfc9 [SPARK-49383][SQL][PYTHON][CONNECT] Support Transpose DataFrame API
    omit 26e59f26b10c [SPARK-49492][CONNECT] Reattach attempted on inactive ExecutionHolder
    omit 4cc34bf82b12 [SPARK-49517][BUILD] Upgrade `netty-tcnative` to 2.0.66.Final
    omit 2c1e69c074a5 [SPARK-49396][SQL] Modify nullability check for CaseWhen expression
    omit f3d2ebd74e50 [SPARK-49526][CONNECT] Support Windows-style paths in ArtifactManager
    omit a6203cc54825 [SPARK-49326][SS] Classify Error class for Foreach sink user function error
    omit d75f550b9fed [SPARK-49303][SS] Implement TTL for ValueState in transformWithStateInPandas API
    omit 7f88344e6026 [SPARK-49521][DOCS] Remove the ambiguous term “constructor” from the documentation for logical plan nodes
    omit 2e2715db59e4 [SPARK-49518][K8S][BUILD] Change to using `build-helper-maven-plugin` to manage the code for `volcano`
    omit f7292cbb6a35 [SPARK-48493][PYTHON][FOLLOW-UP] Fix documentation for Arrow batch support in Python Data Source
    omit f7cfeb534d92 [SPARK-49152][SQL][FOLLOWUP] DelegatingCatalogExtension should also use V1 commands
    omit 9676b1c48cba [SPARK-48348][SPARK-48376][SQL] Introduce `LEAVE` and `ITERATE` statements
    omit 182353d74e43 [SPARK-49523][CONNECT] Increase maximum wait time for connect server to come up for testing
    omit 9a7b6e5c31bf [SPARK-49474][SS] Classify Error class for FlatMapGroupsWithState user function error
    omit 37f2fa99c31d [SPARK-49408][SQL] Use IndexedSeq in ProjectingInternalRow
    omit 628cd751a909 Revert "[SPARK-49241][CORE] Add `OpenTelemetryPush` Sink with `opentelemetry` profile"
    omit f37977771c94 [SPARK-49458][CONNECT][PYTHON] Supply server-side session id via ReattachExecute
    omit 5375ce2acfe2 [SPARK-48965][SQL] Use the correct schema in `Dataset#toJSON`
    omit 0c25fbeaef83 [SPARK-49085][SQL] Remove special casing for Protobuf functions in Connect
    omit 3f3b52969585 [SPARK-49427][CONNECT][SQL] Create a shared interface for MergeIntoWriter
    omit e76c6c9de083 [SPARK-49414][CONNECT][SQL] Add Shared DataFrameReader interface
    omit 75e53b78b67e [SPARK-49512][K8S][DOCS] Drop K8s v1.27 Support
    omit 9f2312d1e5bc [SPARK-49478][CONNECT] Handle null metrics in ConnectProgressExecutionListener
    omit cc3df38ce4b7 [SPARK-48493][PYTHON] Enhance Python Datasource Reader with direct Arrow Batch support for improved performance
    omit 2ed6c3e511f3 [SPARK-49509][CORE] Use `Platform.allocateDirectBuffer` instead of `ByteBuffer.allocateDirect`
    omit c5293ecb017b [SPARK-49426][CONNECT][SQL] Create a shared interface for DataFrameWriterV2
    omit 4a79e73035ea [SPARK-49445][UI] Support show tooltip in the progress bar of UI
    omit 3e22c47146e7 [SPARK-48960][CONNECT] Makes spark-submit works with Spark connect
    omit 90a236eefd50 [MINOR][DOCS] Fix site.SPARK_VERSION pattern in RDD Programming Guide
    omit 339d1c9d9d50 [SPARK-49202][PS] Apply `ArrayBinarySearch` for histogram
    omit 39d4bd8b3d99 [SPARK-49504][BUILD] Add `jjwt` profile
    omit 7d564b7d1e53 [SPARK-49477][PYTHON] Improve pandas udf invalid return type error message
    omit 6165b12c7a64 [SPARK-49249][SPARK-49122][SQL] CONNECT] Add missing doc for the `addArtifact` API
    omit 8301f92a1c60 [SPARK-49422][CONNECT][SQL] Create a shared interface for KeyValueGroupedDataset
    omit e49dfcb73bab [SPARK-49486][BUILD] Upgrade tink to 1.15.0
    omit 48f9cc7e716d [SPARK-49494][DOCS] Use spark logos from spark-website in docs
    omit 7508f6df0f44 [SPARK-47995][INFRA][PYTHON] Refresh testing image for pyarrow 17
    omit 270327d6f8b6 [SPARK-49044][SQL] ValidateExternalType should return child in error
    omit 2a6080c5b4ee [SPARK-49347][R][FOLLOW-UP] Use dedicated environment variable for SparkR deprecation warning
    omit 85d4da8b0dbc [SPARK-49497][BUILD] Upgrade protobuf-java to 3.25.4
    omit 04725ee71ca7 [SPARK-49470][UI] Update dataTables stylesheets and javascripts from 1.13.5 to 1.13.11
    omit 0a3315ecce24 [MINOR][INFRA] Exclude .python-version from git
    omit 32b054c40602 [SPARK-49203][SQL] Add expression for `java.util.Arrays.binarySearch`
    omit ce1fca991860 [SPARK-49420][CONNECT][SQL] Add shared interface for DataFrameNaFunctions
    omit d4e2dac3465c [SPARK-49425][CONNECT][SQL] Create a shared DataFrameWriter
    omit 783c055d05d9 [SPARK-49441][ML] `StringIndexer` sort arrays in executors
    omit 8879df5fc12b [SPARK-49451] Allow duplicate keys in parse_json
    omit ec7570e220e6 [SPARK-49275][SQL] Fix return type nullness of the xpath expression
    omit c274c5a1aecd [SPARK-49476][SQL] Fix nullability of base64 function
    omit cef3c86e046e [SPARK-49480][CORE] Fix NullPointerException from `SparkThrowableHelper.isInternalError`
    omit dc82285610e6 [SPARK-49483][BUILD] Upgrade `commons-lang3` to 3.17.0
    omit 6b879c2ca1f0 [SPARK-49455][SQL][TESTS] Refactor `StagingInMemoryTableCatalog` to override the non-deprecated functions
    omit fb8d01acf166 [SPARK-48682][SQL][FOLLOW-UP] Changed initCap behaviour with UTF8_BINARY collation
    omit c58148da5496 [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled`
    omit 9cec3c4f7c1b [SPARK-49460][SQL] Remove `cleanupResource()` from EmptyRelationExec
    omit 7b54d9c554fb [SPARK-49466][BUILD] Upgrade `zstd-jni` to 1.5.6-5
    omit 23469fd93721 [SPARK-49459][CORE][SHUFFLE] Support `CRC32C` for Shuffle Checksum
    omit 343471dac4b9 [SPARK-49409][CONNECT] Adjust the default value of CONNECT_SESSION_PLAN_CACHE_SIZE
    omit 528ba0e0db7d [SPARK-49423][CONNECT][SQL] Consolidate Observation in sql/api
    omit 0863af4254aa [SPARK-49249][SPARK-49122][CONNECT][SQL] Add `addArtifact` API to the Spark SQL Core
    omit df4256836d71 [MINOR] Add `artifacts` to `.gitignore`
    omit 493ca987ca93 [SPARK-49378][DOCS][SS] Break apart the Structured Streaming Programming Guide
    omit 53c1f31dc26b [SPARK-49119][SQL] Fix the inconsistency of syntax `show columns` between v1 and v2
    omit 0602020eb3b3 [SPARK-49252][CORE] Make`TaskSetExcludeList` and `HeathTracker` independent
    omit 03180ece177d [SPARK-49421][CONNECT][SQL] Create a shared RelationalGroupedDataset interface
    omit 2fc21adc636c [SPARK-49456][DOCS] Fix hash fragment scrolling behavior on versioned Spark documentation
    omit cf487b9f81fb [SPARK-49363][SS][TESTS] Add unit tests for potential RocksDB state store SST file mismatch
    omit ebe635eb1d44 [SPARK-48482][PYTHON][FOLLOWUP] Revert dropDuplicates and dropDuplicatesWIthinWatermark should accept variable length args
    omit c7a9c1e0776e [SPARK-49021][SS] Add support for reading transformWithState value state variables with state data source reader
    omit 8a85f22ebb80 [SPARK-49454][SQL] Avoid double normalization in the cache process
    omit 1e6765910c44 [SPARK-49410][SQL][TESTS] Update collation benchmarks
    omit dad2b763f004 [SPARK-49419][CONNECT][SQL] Create shared DataFrameStatFunctions
    omit effcf22a029f [SPARK-49457][BUILD] Remove uncommon curl option `--retry-all-errors`
    omit fec68678c0cd [SPARK-49028][CONNECT][SQL] Create a shared SparkSession
    omit d78199ae2a67 [SPARK-49406][SQL][TESTS] Add three test cases for `from_avro/to_avro` from the perspective of `GenericRecord`
    omit c728c223f8eb [SPARK-49449][SQL][DOCS] Remove string-from-metadata and binary-from-metadata
    omit d048dfe6d10d [SPARK-49446][BUILD] Upgrade jetty to 11.0.23
    omit af5e0a267e5a [SPARK-46037][SQL] Correctness fix for Shuffled Hash Join build left without codegen
    omit ebb1975ae4c1 [SPARK-49447][K8S] Fix `spark.kubernetes.allocation.batch.delay` to prevent small values less than 100
    omit a3cb06407029 [SPARK-49306][PYTHON][SQL] Create SQL function aliases for 'zeroifnull' and 'nullifzero'
    omit 620f16ed0afe [SPARK-48400][CORE] Promote `PrometheusServlet` to `DeveloperApi`
    omit 269381a44bf5 [SPARK-48781][SQL] Add Catalog APIs for loading stored procedures
    omit aa208baf6414 [SPARK-49313][BUILD][TESTS] Upgrade `DB2` & `MySQL` & `Postgres` & `Mariadb` docker image version
    omit 8cfc88f5fb2c [SPARK-49086][CONNECT] Move ML function registration to SparkSessionExtensions
    omit aaf61e69a0dc [SPARK-49029][CONNECT][SQL] Create shared Dataset interface
    omit 57b6bc114e33 [MINOR][SQL] Fix the incorrect method `@link` tag in `StagingTableCatalog`
    omit df3d4fd9ca77 [SPARK-49438][SQL] Fix the pretty name of the `FromAvro` & `ToAvro` expression
    omit 07816316c265 [SPARK-49439][SQL] Fix the pretty name of the `FromProtobuf` & `ToProtobuf` expression
    omit 0b0933c52585 [SPARK-49050] Enabling deleteIfExists operator in TWS with Virtual Column Families
    omit 54edfd3cdb1b [SPARK-49412][PS] Compute all box plot metrics in single job
    omit 1c9cde59ba65 [SPARK-49402][PYTHON][FOLLOW-UP] Manually load ~/.profile in Spark Connect notebook
    omit 6d8235f3b2bb [SPARK-49357][CONNECT][PYTHON] Vertically truncate deeply nested protobuf message
    omit 2eb2e39eda79 [SPARK-49241][CORE] Add `OpenTelemetryPush` Sink with `opentelemetry` profile
    omit 9315b9c04704 [SPARK-49405][SQL] Restrict charsets in JsonOptions
    omit 212a59d7a5be [SPARK-49393][SQL] Fail by default in deprecated catalog plugin APIs
    omit 0895471ad3b2 [MINOR] Fix spelling mistake in comment
    omit 1bdbe14ef4d1 [SPARK-43242][CORE] Fix throw 'Unexpected type of BlockId' in shuffle corruption diagnose
    omit c106c7760ea6 [SPARK-49017][SQL][TESTS][FOLLOW-UP] Add testing for CreateTableAs with parameters
    omit eb79fc46ac5d [SPARK-49204][SQL][FOLLOWUP] Fix IndexOutOfBoundsException when dealing with surrogate pairs
    omit e1b5d65aeecc [SPARK-49384][BUILD] Upgrade rocksdbjni to 9.5.2
    omit 9fc1e05224d2 [SPARK-49402][PYTHON] Fix Binder integration in PySpark documentation
    omit 7448f2596e79 [SPARK-49355][SQL] `levenshtein` should check whether the `collation` values of all parameter types are the same
    omit f7bb15c21b12 [SPARK-49347][R][FOLLOW-UP] Set SPARK_TESTING in Windows build
    omit d64d1f76ff28 [SPARK-49366][CONNECT] Treat Union node as leaf in dataframe column resolution
    omit f394cd3878bd [SPARK-48441][SQL][FOLLOWUP] Fix StringTrim behaviour for UTF8_LCASE collation
    omit e10a789273c9 [SPARK-49311][SQL] Make it possible for large 'interval second' values to be cast to decimal
    omit ccef4df7abb0 [SPARK-49354][SQL] `split_part` should check whether the `collation` values of all parameter types are the same
    omit 7e4d6bdba0ff [SPARK-47258][SQL] Assign names to error classes _LEGACY_ERROR_TEMP_127[0-5]
    omit b1ddec5757ae [SPARK-49391][PS] Box plot select outliers by distance from fences
    omit d7f0c8fca7bd [SPARK-49314][BUILD][TESTS] Upgrade `h2` to 2.3.232, `postgresql` to 42.7.4 and `mssql` to 12.8.1.jre11
    omit 30152d0ee938 [SPARK-49387][PYTHON] Fix type hint for `accuracy` in `percentile_approx` and `approx_percentile`
    omit 8409da3d1815 [SPARK-49382][PS] Make frame box plot properly render the fliers/outliers
    omit f59607995bbd [SPARK-49385][K8S] Fix `getReusablePVCs` to use `podCreationTimeout` instead of `podAllocationDelay`
    omit 5706942bc6ac [SPARK-49381][K8S][TESTS] Fix `SPARK-41388` test case to assert the result
    omit 4d5b4b568078 [SPARK-49370][BUILD] `maven.scaladoc.skip` should not affect test code compilation
    omit d4296539748e [SPARK-49347][R] Deprecate SparkR
    omit fb13344556af [SPARK-49372][SS] Ensure that latestSnapshot is set to none on close to avoid subsequent use
    omit 4b7191d43b4b [SPARK-49367][PS] Parallelize the KDE computation for multiple columns (plotly backend)
    omit 69b09bc52c29 [SPARK-49368][CONNECT] Avoid accessing protobuf lite classes directly
    omit 0c18fc072b05 [SPARK-49365][PS] Simplify the bucket aggregation in hist plot
    omit ee97caafadf7 [SPARK-49352][SQL] Avoid redundant array transform for identical expression
    omit af45052981b8 [SPARK-49373][PROTOBUF][TESTS] Fix `ProtobufFunctionsSuite` to be environment-independent
    omit e95618f4e7df [SPARK-49027][CONNECT][SQL] Share Column API between Class and Connect
    omit d84f1a3575c4 [SPARK-49217][CORE] Support separate buffer size configuration in UnsafeShuffleWriter
    omit 70c9b94218b9 [SPARK-49268][CORE] Log IO exceptions in SHS history provider
    omit a18ba89df9aa [SPARK-49342][SQL] Make TO_AVRO SQL function 'jsonFormatSchema' argument optional
    omit 97e0a88b1d04 [SPARK-49359][SQL] Allow StagedTableCatalog implementations to fall back to non-atomic write
    omit 08d69ffae935 [SPARK-49344][PS] Support `json_normalize` for Pandas API on Spark
    omit 853731dc5731 [SPARK-49333][SQL] Shutdown timeout thread while cleaning up SparkExecuteStatementOperation
    omit b57d863afda1 [SPARK-49353][SQL] Update docs related to `UTF-32` encoding/decoding
    omit 3e8cd994a8df [SPARK-47262][SQL] Assign names to error conditions for parquet conversions
    omit 98c7f519c09a [SPARK-49339][INFRA] Use `setup-minikube` GitHub Action
    omit 309c543ae853 [SPARK-49334][SQL] `str_to_map` should check whether the `collation` values of all parameter types are the same
    omit 596098b5fe61 [SPARK-49223][ML] Simplify the StringIndexer.countByValue with builtin functions
    omit e93c5fbe81d2 [SPARK-49301][SS] Chunk arrow data passed to Python worker
    omit ab3cd910e0c9 [SPARK-49306][SQL] Create new SQL functions 'zeroifnull' and 'nullifzero'
    omit b796ecbe3275 [SPARK-49121][SQL] Support `from_protobuf` and `to_protobuf` for SQL functions
    omit 4ceacbe862dd [SPARK-49345][CONNECT] Make sure using the current running Spark Session
    omit cc96ede2b9e2 [SPARK-49341] Remove `connector/docker` in favor of `Apache Spark Operator`
    omit 72c1df39c049 [SPARK-48796][SQL] Optimze parquet binary getBytes with getBytesUnsafe to avoid copy cost
    omit 0cb28c49f5f1 [SPARK-49338][CORE][SQL][K8S] Make Spark Deamons support `spark.log.structuredLogging.enabled` system property
    omit a885365897ac [SPARK-49113] Dont assert on translation bugs - silently swallow the exception
    omit f1cb7a730b18 [SPARK-49318][SQL] Preempt low priority error on LCA until end of check analysis to improve error experience
    omit 9b9a7a7478d1 [SPARK-48628][CORE] Add task peak on/off heap memory metrics
    omit 99b793965f01 [SPARK-48638][FOLLOW][CONNECT] Fix documentation for ExecutionInfo
    omit 37f93e5acfc0 [SPARK-49335][BUILD] Upgrade Maven to 3.9.9
    omit 4d9c30982459 [SPARK-49336][CONNECT] Limit the nesting level when truncating a protobuf message
    omit 3edc9c23a723 [SPARK-49323][CONNECT] Move MockObserver from Spark Connect Server's test folder to the Server's main folder
    omit a31a5acbd389 [SPARK-49327][BUILD] Upgrade `commons-compress` to 1.27.1
    omit 7f1a0c62a03c [SPARK-49142][FOLLOWUP][CONNECT][PYTHON] Use PySparkLogger for Spark Connect client log
    omit ae0585226479 [SPARK-49300][CORE] Fix Hadoop delegation token leak when tokenRenewalInterval is not set
    omit b6164e6ed717 [SPARK-49246][SQL] TableCatalog#loadTable should indicate if it's for writing
    omit 33059396bd2d [SPARK-49026][CONNECT] Add ColumnNode to Proto conversion
    omit bc7bfbcbfe58 [SPARK-49274][CONNECT] Support java serialization based encoders
    omit af4c738ff468 [SPARK-49269][SQL] Eagerly evaluate VALUES() list in AstBuilder
    omit ba208b9ca999 [SPARK-49292][BUILD] Upgrade dropwizard metrics to 4.2.27
    omit 47c0b55e06d1 [SPARK-49190][K8S][FOLLOW-UP] Add how to populate executor log url vars with env vars
    omit 899fad4710be [SPARK-49043][SQL] Fix interpreted codepath group by on map containing collated strings
    omit b4a802901174 [SPARK-49312][PYTHON] Improve error message for `assertSchemaEqual`
    omit 4be1ddc4a12f [SPARK-49267][SQL][TESTS] Add collation support unit tests for StringTranslate
    omit 1ae482ac7e64 [SPARK-49075][BUILD] Upgrade JUnit5 related to the latest version
    omit e64f620fe8fd [SPARK-48796][SS] Load Column Family Id from RocksDBCheckpointMetadata for VCF when restarting
    omit 8fbbcb09454c [SPARK-49025][CONNECT] Make Column implementation agnostic
    omit 5d2d6a314044 [SPARK-49131][SS] TransformWithState should properly set implicit grouping keys even with lazy iterators
    omit a6a62e564bed [SPARK-49263][CONNECT] Spark Connect python client: Consistently handle boolean Dataframe reader options
    omit 542b24a95960 [SPARK-49266][SQL][TESTS] Add collation support unit tests for FindInSet
    omit 11b682cf5b7c [SPARK-49017][SQL] Insert statement fails when multiple parameters are being used
    omit dd259b0b2784 [SPARK-49016][SQL] Restore the behavior that queries from raw CSV files are disallowed when only include corrupt record column and assign name to `_LEGACY_ERROR_TEMP_1285`
    omit 5b2d2149b615 [SPARK-49257][INFRA] Make `bundler install` can retry
    omit 56d1e48860e3 [SPARK-49264][SQL][TESTS] Add collation support unit tests for Contains, StartsWith, and EndsWith
    omit 866cfaa851c5 [SPARK-49260][CONNECT] No longer prepend the classes path of `sql/core` module in Spark Connect Shell
    omit 469ebcf4fe41 [SPARK-45891][FOLLOW-UP][VARIANT] Address post-merge comments
    omit a767f5cb1704 [SPARK-49022] Use Column Node API in Column
    omit 8566bc630218 [SPARK-49265][SQL][TESTS] Add collation support unit tests for Upper, Lower, and InitCap
    omit 291647daf483 [SPARK-49250][SQL] Improve error message for nested UnresolvedWindowExpression in CheckAnalysis
    omit e63c3986e2e4 [SPARK-48066][SQL][TESTS] Enable ANSI for TPCDSQueryBenchmark
    omit 2e47be75d3aa [SPARK-49213][SQL][TESTS] Add collation support unit tests for comparison and equality
    omit 5166cb3161c2 [SPARK-49245][SQL] Refactor some analyzer rules
    omit e3a44307fe73 [SPARK-49039][UI] Reset checkbox when executor metrics are loaded in the Stages tab
    omit 366f9ecb3f35 [SPARK-48873][SQL][FOLLOW-UP] Add useUnsafeRow to JSON options
    omit 3050ba00e883 [SPARK-48803][FOLLOWUP] Remove unused error-condition `_LEGACY_ERROR_TEMP_2088`
    omit 7a670d7ae19e [SPARK-49255][BUILD][K8S] Upgrade `kubernetes-client` to 6.13.3
    omit bcb963ecce6c [SPARK-49254][BUILD] Upgrade mima-core & sbt-mima-plugin to 1.1.4
    omit 29b1381ef185 [MINOR][DOCS] Use `SparkSession.builder.getOrCreate()` over the deprecated `build()`
    omit ac5aa7dcb0a5 [SPARK-49251][BUILD] Upgrade ORC to 2.0.2
    omit 3e867a69fd12 [SPARK-48966][SQL] Improve error message with invalid unresolved column reference in UDTF call
    omit 470e3f948123 [SPARK-49243][INFRA][K8S] Add `OpenContainers` Annotations in docker images
    omit def42d44405a [SPARK-48755][SS][PYTHON] transformWithState pyspark base implementation and ValueState support
    omit b9fbdf010efe [SPARK-49211][SQL] V2 Catalog can also support built-in data sources
    omit 0219d60224ad [SPARK-48882][SS] Assign names to streaming output mode related error classes
    omit b98ac058ab88 [MINOR][SQL] Remove unnecessary asserts from loop iterators
    omit 7b43a6fdf0f9 [MINOR][SQL][TESTS] Changes the `test:runMain` in the code comments to `Test/runMain`
    omit 00b8cb31de1c [SPARK-49137][SQL][FOLLOWUP] Assign a reasonable error condition, when the Boolean condition in the if statement returns an empty row
    omit 5a8794025652 [SPARK-47257][SQL] Assign names to error classes _LEGACY_ERROR_TEMP_105[3-4] and _LEGACY_ERROR_TEMP_1331
    omit f87476f10040 [SPARK-49242][BUILD] Upgrade commons-cli to 1.9.0
    omit c8cf3947a09f [SPARK-45590][BUILD] Upgrade okio to 1.17.6 from 1.15.0
    omit d8bb37e82b46 [SPARK-47265][SQL][FOLLOWUP] V2SessionCatalog#createTable should not fail
    omit 89bdc5047b10 [SPARK-48947][SQL] Use lowercased charset name to decrease cache missing in Charset.forName
    omit 0c24ae11164d [SPARK-49240][BUILD] Add `scalastyle` and `checkstyle` rules to avoid `URL` constructors
    omit a0fc398e1c11 [SPARK-49152][SQL][FOLLOWUP] table location string should be Hadoop Path string
    omit e0b3f2abe8ba [SPARK-49235][SQL] Refactor ResolveInlineTables rule so it doesn't traverse the whole tree
    omit 02795a36171c [SPARK-49222][SQL] Clear useless methods in QueryCompilationErrors
    omit 7e761888ea00 [SPARK-48618][SQL] Utilize the `ErrorCode` and `SQLState` returned in SQLException to make errors more accurate
    omit 1947646f2b63 [SPARK-49234][BUILD] Upgrade `xz` to `1.10`
    omit 36327329779c [MINOR][DOCS] Explicitly mention RDD/SparkContext are unsupported in Spark Connect
    omit 55c0ea58bc0a [SPARK-48913][BUILD][FOLLOWUP] Remove `txw2` from `LICENSE-binary` and delete `LICENSE-txw2.txt`
    omit 2fedf0fb7b68 [SPARK-49232][INFRA][TESTS] Remove `3.4.2` from `SKIP_SPARK_RELEASE_VERSIONS`
    omit b9b14f95c725 [SPARK-48967][SQL] Improve performance and memory footprint of "INSERT INTO ... VALUES" Statements
    omit 67d2888c476b [SPARK-49226][SQL] Clean-up UDF code generation
    omit 3fcf041a1809 Revert "[SPARK-48628][CORE] Add task peak on/off heap memory metrics"
    omit 56a90add735c [SPARK-49166][SQL] Support OFFSET in correlated subquery
    omit f4f24503bbde [SPARK-49225][SQL][CONNECT] Add ColumnNode sql & normalize
    omit 717cedfc8cbc [SPARK-49221][BUILD] Fix sbt compilation warnings related to `Regular tasks always evaluate task dependencies (.value) regardless of if expressions`
    omit b8060e585873 [SPARK-49224][TESTS] Regenerate benchmark results
    omit 9b6907dfb140 [SPARK-48913][SQL] Implement IndentingXMLStreamWriter
    omit 0c54125bbb28 [SPARK-49038][SQL] SQLMetric should report the raw value in the accumulator update event
    omit 20d98a1ca594 [SPARK-49216][CORE] Fix to not log message context with explicitly LogEntry constructed when Structured Logging conf is off
    omit bfddd53d98da [SPARK-48760][SQL][FOLLOWUP] Override `hashCode` method for `TableChange.ClusterBy`
    omit f08165003cff [SPARK-42307][SQL] Assign name for error _LEGACY_ERROR_TEMP_2232
    omit 3f3d02473265 [SPARK-49205][SQL] KeyGroupedPartitioning should inherit HashPartitioningLike
    omit d82c69520c9b [SPARK-49207][SQL] Fix one-to-many case mapping in SplitPart and StringSplitSQL
    omit e77c8fbe5940 [SPARK-49204][SQL] Fix surrogate pair handling in StringReplace
    omit e38e13506b09 [SPARK-49204][SQL] Fix surrogate pair handling in StringInstr and StringLocate
    omit 7feea93ce4a3 [SPARK-49204][SQL] Fix surrogate pair handling in SubstringIndex
    omit 175deba21740 [SPARK-49204][SQL] Fix surrogate pair handling in StringTrim
    omit 2e96fc294565 [SPARK-49146][SS] Move assertion errors related to watermark missing in append mode streaming queries to error framework
    omit e7e082663b94 [SPARK-48463][ML] Make Binarizer, Bucketizer, VectorAssembler, FeatureHasher, QuantizeDiscretizer, OnehotEncoder, StopWordsRemover, Imputer, Interactor supporting nested input columns
    omit acfe84712a8e [SPARK-48347][SQL] Support for WHILE statement
    omit 4aa8afa96655 [SPARK-49215][PYTHON][DOCS] Document the NaN handling in df.na.drop
    omit 5501bdad2f23 [SPARK-49199][BUILD] `build/mvn` should check project local installed Maven ahead
    omit 8133294d6c2c [SPARK-49179][SQL] Fix v2 multi bucketed inner joins throw AssertionError
    omit e72d21c299a4 [SPARK-49187][BUILD] Upgrade slf4j to 2.0.16
    omit 70b814b558be [SPARK-49201][PS][PYTHON][CONNECT] Reimplement `hist` plot with Spark SQL
    omit 111302991006 [SPARK-49195][CONNECT] Embed script level parsing logic into SparkSubmitCommandBuilder
    omit 93cf355ab05d [SPARK-49197][CORE] Redact `Spark Command` output in `launcher` module
    omit 7aa83ad20306 [SPARK-49208][SQL][TESTS] Add more tests for negative MONTH intervals
    omit 654a99783047 [SPARK-49078][SQL][FOLLOWUP] Clear unused showColumnsNotSupportedForV2TablesError in QueryCompilationErrors and add more uts
    omit d09e1c57ac14 [SPARK-49196][BUILD] Upgrade `kubernetes-client` to 6.13.2
    omit dba555512056 [SPARK-47741][SQL][TESTS][FOLLOWUP] Make `EXEC IMMEDIATE STACK OVERFLOW` in `ExecuteImmediateEndToEndSuite` check the intercepted exception
    omit 2973097b4fae [SPARK-49206][CORE][UI] Add `Environment Variables` table to Master `EnvironmentPage`
    omit e3ba74b65b9d [SPARK-49200][SQL] Fix null type non-codegen ordering exception
    omit 2fb8dffe40ff [SPARK-46632][SQL] Fix subexpression elimination when equivalent ternary expressions have different children
    omit f33aa0abbcda [SPARK-48778][SQL][TESTS] Improve collation support testing - add unit tests for string expressions
    omit cb7919bb5ecd [SPARK-49164][SQL] Fix not NullSafeEqual in predicate of SQL query in JDBC Relation
    omit 2465cb0d35ae [SPARK-49152][SQL] V2SessionCatalog should use V2Command
    omit 436a6d1a1899 [SPARK-49198][CONNECT] Prune more jars required for Spark Connect shell
    omit 37ef6a3bec5a [SPARK-49173][CONNECT] Change Spark Connect shell prompt from `@` to `scala>`
    omit e893b32196af [SPARK-49188][SQL] Internal error on concat_ws called on array of arrays of string
    omit 5c1222511cd1 [SPARK-49185][PS][PYTHON][CONNECT] Reimplement `kde` plot with Spark SQL
    omit ed04e160940a [SPARK-49183][SQL] V2SessionCatalog.createTable should respect PROP_IS_MANAGED_LOCATION
    omit b09302994b1a [SPARK-47430][SQL] Rework group by map type to fix bind reference exception
    omit 879924491944 [SPARK-49114][SS][FOLLOWUP] Add missing error classes into json definition file
    omit 3ab97c1a4eaa [SPARK-49137][SQL] When the Boolean condition in the `if statement` is invalid, an exception should be thrown
    omit 11e0d2df672d [SPARK-49169][BUILD] Upgrade `commons-compress` to 1.27.0
    omit d1af1f5951b6 [SPARK-49171][CONNECT][DOCS] Update Spark Shell with Spark Connect
    omit 6e3b38476037 [SPARK-48936][CONNECT][FOLLOW-UP] Makes spark-shell with Spark connect work with SPARK_REMOTE environment variable
    omit 9459c5a6e3d9 [SPARK-49178][SQL] Optimize performance of `Row#getSeq` to match the performance when using Spark 3.5 with Scala 2.12
    omit 567d58c2c161 [SPARK-49193][SQL] Improve the performance of RowSetUtils.toColumnBasedSet
    omit dee53bb31edd [SPARK-48936][CONNECT] Makes spark-shell works with Spark connect
    omit 5ef040ebe366 [SPARK-49177][INFRA][DOCS] Make GA's `build_error_docs` run only once
    omit 584b6dbfc99f [SPARK-49190][K8S] Add `SPARK_EXECUTOR_ATTRIBUTE_(APP|EXECUTOR)_ID` if `CUSTOM_EXECUTOR_LOG_URL` is defined
    omit eb859def4402 [SPARK-49182][DOCS][PYTHON] Stop publish site/docs/{version}/api/python/_sources dir
    omit d8aff6e04941 [SPARK-48937][SQL] Add collation support for StringToMap string expressions
    omit fd3069ab113c [SPARK-49163][SQL] Attempt to create table based on broken parquet partition data results should return user-facing error
    omit 731a104df307 [SPARK-47261][SQL] Assign better name for errors _LEGACY_ERROR_TEMP_1172, _LEGACY_ERROR_TEMP_1173, and _LEGACY_ERROR_TEMP_1174
    omit 81948bb61d5f [SPARK-49174][BUILD][DOCS] Exclude the dir `docs/util` from `_site`
    omit 8a0986fc4e25 [SPARK-49170][BUILD] Upgrade snappy to 1.1.10.6
    omit 832e4d034827 [SPARK-49176][K8S][DOCS] Fix `spark.ui.custom.executor.log.url` docs by adding K8s
    omit c121bb557447 [SPARK-49111][SQL] Move withProjectAndFilter to the companion object of DataSourceV2Strategy
    omit e0d435d59162 [SPARK-49172][PYTHON][DOCS] Refine the type hints in functions
    omit 44a84dd507ac [SPARK-47410][SQL][FOLLOWUP] Limit part of StringType API to private[sql]
    omit 19fbdaa54cd7 [SPARK-49142][CONNECT][PYTHON] Follow up to revert proto to string performance cost
    omit 01a6991a338c [SPARK-49138][SQL] Fix CollationTypeCasts of several expressions
    omit 61d8c3b2b45f [SPARK-47410][SQL][FOLLOWUP] Update StringType interface for collation support
    omit b54b3006f946 [SPARK-49147][CORE] Mark `KryoRegistrator` with DeveloperApi interface
    omit 43d881a94f2e [SPARK-49139][SQL] Enable collations by default
    omit a713a66e1c4f [SPARK-49142][CONNECT][PYTHON] Lower Spark Connect client log level to debug
    omit af70aafd330f [SPARK-49154][K8S][INFRA][DOCS] Upgrade `Volcano` to 1.9.0
    omit b3dd626defd0 [SPARK-49151][BUILD] Upgrade `slf4j` to 2.0.14
    omit d9f358fb0123 [SPARK-49124][BUILD] Upgrade tink to 1.14.1
    omit 7e68b28d5bfa [SPARK-49150][BUILD] Upgrade `commons-lang3` to 3.16.0
    omit d84a3e9b3c17 [MINOR][CORE] Fix a typo in ResourceProfile comment
    omit d0d98c15182d [SPARK-49125][SQL] Allow duplicated column names in CSV writing
    omit 712b94659b0c [SPARK-49143][K8S][DOCS] Update `YuniKorn` docs with v1.5.2
    omit 8a8f65d2a2ec [SPARK-48843][FOLLOWUP] Adding more tests that would have failed had the fix ap…
    omit 435a01a7fc2d [SPARK-49115][DOCS] Add docs for state metadata source for operators using schema format v2 such as transformWithState
    omit 717a6da11d0b [SPARK-48628][CORE] Add task peak on/off heap memory metrics
    omit 5481d39177bb [SPARK-49141][SQL] Mark variant as hive incompatible data type
    omit 7769ef11f367 [SPARK-49082][SQL] Widening type promotions in `AvroDeserializer`
    omit ce4f185a98be [SPARK-49134][INFRA] Support retry for deploying artifacts to Nexus staging repository
    omit 490f0bc4ed66 [SPARK-48911][SQL][FOLLOWUP][TESTS] Fix test for collect_set
    omit 46acdf453517 [MINOR][SQL] Add parquet nanosAsLong behavior change to 3.2 migration guide
    omit e7f790084cae [SPARK-49128][CORE] Support custom History Server UI title
    omit 514f272273d7 [SPARK-49126][CORE] Move `spark.history.ui.maxApplications` config definition to `History.scala`
    omit b3877894acb5 [SPARK-49089][CONNECT] Move hardcoded Catalyst Expressions to the internal function registry
    omit eda290c86ffb [SPARK-49024][CONNECT] Add support for functions to column node
    omit c3985ac256e4 [SPARK-48989][SQL][FOLLOWUP] Fix SubstringIndex codegen
    omit 2db05db9e9d1 [SPARK-49098][SQL] Add write options for INSERT
    omit bb584de1442e [SPARK-48911][SQL][TESTS] Improve collation support testing for various expressions
    omit 9fb9cff6e1cb [SPARK-49120][BUILD] Bump Gson 2.11.0
    omit 0346b186f6eb [SPARK-49004][CONNECT] Use separate registry for Column API internal functions
    omit 6246b18635f0 [MINOR][INFRA] Mute `stale` Github Action in forks
    omit 5b59129e70b5 [SPARK-49047][PYTHON][CONNECT][FOLLOWUP] Catch potential truncation failure
    omit babe7a05a6e4 [SPARK-48978][SQL] Implement ASCII fast path in collation support for UTF8_LCASE
    omit 97fe0e896c6a [SPARK-49099][SQL] CatalogManager.setCurrentNamespace should respect custom session catalog
    omit 6de95a2fbdb4 [SPARK-47359][SQL][FOLLOWUP] Remove unused genCode for StringTranslate
    omit 2a752105091e [SPARK-49048][SS] Add support for reading relevant operator metadata at given batch id
    omit b477753450f3 [SPARK-49109][SQL] Rename leftover BinaryLcase to Lcase
    omit a9bc38a74edb [SPARK-49114] Sub-categorize cannot load state store errors
    omit 2e86051b692b [SPARK-49116][K8S] Fix `InvalidDefaultArgInFrom` in Python/R binding Dockerfiles
    omit ed753089926c [MINOR][INFRA] Add `bin/docker-image-tool.sh` to KUBERNETES label
    omit bfa0d13f3f6b [SPARK-49083][CONNECT] Allow from_xml and from_json to natively work with json schemas
    omit da5912a8b1e0 Revert "[MINOR][SQL] Remove orphans in ProtoToParsedPlanTestSuite and PlanGenerationTestSuite"
    omit d9c590299668 [SPARK-49117][K8S] Fix `docker-image-tool.sh` to be up-to-date
    omit 6d32472b2a42 [SPARK-49112][CONNECT][TEST] Make `createLocalRelationProto` support `TimestampType`
    omit de8ee94bc44f [SPARK-49014][BUILD] Bump Apache Avro to 1.12.0
    omit 5f9c87007379 [SPARK-49097][INFRA] Add Python3 environment detection for the `build_error_docs` method in `build_api_decs.rb`
    omit e8227a8e2442 [SPARK-49018][SQL] Fix approx_count_distinct not working correctly with collation
    omit 5250ff3dbd7e [SPARK-48791][CORE][FOLLOW-UP] Fix regression caused by immutable conversion on TaskMetrics#externalAccums
    omit 79cb09627b43 Revert "[SPARK-48763][TESTS][FOLLOW-UP] Update project location in PlanGenerationTestSuite"
    omit e5b6b5ff6f54 [SPARK-48338][SQL] Improve exceptions thrown from parser/interpreter
    omit d431d4034219 [MINOR][SQL] Remove orphans in ProtoToParsedPlanTestSuite and PlanGenerationTestSuite
    omit 76a1ca5e1959 [SPARK-49060][CONNECT] Clean up Mima rules for SQL-Connect binary compatibility checks
    omit 682eb1b63a4a [SPARK-49107][SQL] `ROUTINE_ALREADY_EXISTS` supports RoutineType
    omit dbc8f0146fbe [SPARK-49108][EXAMPLE] Add `submit_pi.sh` REST API example
    omit f1f686dbf57c [SPARK-48763][TESTS][FOLLOW-UP] Update project location in PlanGenerationTestSuite
    omit 6bf6088e7719 [SPARK-49063][SQL] Fix Between with ScalarSubqueries
    omit f99291ac864d [SPARK-48346][SQL] Support for IF ELSE statements in SQL scripts
    omit f01eafd79f3b [SPARK-49057][SQL] Do not block the AQE loop when submitting query stages
    omit 94f88723eed3 [SPARK-49078][SQL] Support show columns syntax in v2 table
    omit b76f0b9c5d91 [SPARK-49047][PYTHON][CONNECT] Truncate the message for logging
    omit 4e69e1640629 [SPARK-49104][CORE][DOCS] Document `JWSFilter` usage in Spark UI and REST API and rename parameter to `secretKey`
    omit 250a4cca2d75 [SPARK-48834][SQL][TESTS][FOLLOWUP] Add `assume(shouldTestPandasUDFs)` to PythonUDFSuite complex variant input test
    omit 44b10ba00041 [MINOR][DOCS] Fix broken links in streaming docs
    omit 18db040ede12 [SPARK-49106][DOCS] Documented `Prometheus` endpoints
    omit b6e1a0d297c0 [SPARK-48731][INFRA] Upgrade `docker/build-push-action` to v6
    omit dbba92a1b4ea [SPARK-48949][SQL] SPJ: Runtime partition filtering
    omit b17e50886242 [SPARK-49105][BUILD][TESTS] Upgrade `ojdbc11` to 23.5.0.24.07 and OracleDatabaseOnDocker docker image tag to `oracle-free:23.5-slim`
    omit bf02b9b206a9 [SPARK-49103][CORE] Support `spark.master.rest.filters`
    omit 7612f13b5eac [SPARK-47894][CORE][FOLLOWUP] Add a trailing slash to `MasterPage`'s `Environment` page link
    omit aaf602a5099d [SPARK-48480][SS][CONNECT] StreamingQueryListener should not be affected by spark.interrupt()
    omit 9e35d04cb030 [SPARK-48931][SS][FOLLOWUP] Reduce Cloud Store List API cost for state store maintenance task
    omit 94b572a60111 [SPARK-49080][SQL][TEST] Upgrade `mssql-jdbc` to 12.8.0.jre11 and MsSQLServer docker image tag to `2022-CU14-ubuntu-22.04`
    omit 5f2d3b085d2d [SPARK-48763][FOLLOWUP] Make `dev/lint-scala` error message more accurate
    omit 6631abc2f170 [SPARK-49094][SQL] Fix ignoreCorruptFiles non-functioning for hive orc impl with mergeSchema off
    omit 3da31b07c4df [SPARK-49090][CORE] Support `JWSFilter`
    omit 080e7eb0de06 [SPARK-49000][SQL][FOLLOWUP] Improve code style and update comments
    omit c248b0651b1d [SPARK-48763][CONNECT][BUILD][FOLLOW-UP] Move Spark Connect common/server into sql directory
    omit 6e66be7de9bd [MINOR][DOCS] Fix typos in docs/sql-ref-number-pattern.md
    omit aca0d24c8724 [SPARK-49093][SQL] GROUP BY with MapType nested inside complex type
    omit 5a5390c90642 [SPARK-49071][SQL] Remove ArraySortLike trait
    omit cbe36338ac10 [SPARK-49072][DOCS] Fix abnormal display of text content which contains two $ in one line but non-formula in docs
    omit a9daed17444d [MINOR][DOCS] Fix typos in `docs/sql-data-sources-xml.md`
    omit f15570dbfe19 Revert "SPARK-49004"
    omit a0f8de58c4b6 Revert "Touch-ups after cherry-pick"
    omit 275d3c27aa0d Touch-ups after cherry-pick
    omit c8db813644f9 SPARK-49004
    omit 6706c41cb42c [SPARK-49070][SS][SQL] TransformWithStateExec.initialState is rewritten incorrectly to produce invalid query plan
    omit 08b1fb55d738 [SPARK-49077][SQL][TESTS] Remove `bouncycastle-related` test dependencies from `hive-thriftserver` module
    omit abf9bac69680 [SPARK-49076][SQL] Fix the outdated `logical plan name` in `AstBuilder's` comments
    omit b0f92a9261a3 [SPARK-49002][SQL] Consistently handle invalid locations in WAREHOUSE/SCHEMA/TABLE/PARTITION/DIRECTORY
    omit 63c08632f2d6 [SPARK-49065][SQL] Rebasing in legacy formatters/parsers must support non JVM default time zones
    omit c3bb64e367ac [MINOR][PS][DOCS] Add `DataFrame.plot.kde` to API reference
    omit bf3ad7e94afb [SPARK-49074][SQL] Fix variant with `df.cache()`
    omit 06ed91ac6e75 [SPARK-49066][SQL][TESTS] Refactor `OrcEncryptionSuite` and make `spark.hadoop.hadoop.security.key.provider.path` effective only within `OrcEncryptionSuite`
    omit 8b826abd0ee8 [SPARK-44638][SQL][TESTS] Add test for Char/Varchar in JDBC custom options
    omit dfa21332f20f [SPARK-49000][SQL] Fix "select count(distinct 1) from t" where t is empty table by expanding RewriteDistinctAggregates
    omit 0e07873d368f [SPARK-48977][SQL] Optimize string searching under UTF8_LCASE collation
    omit 5954ed19b9dd [SPARK-48964][SQL][DOCS] Fix the discrepancy between implementation, comment and documentation of option `recursive.fields.max.depth` in ProtoBuf connector
    omit 43c05a73e678 [SPARK-49064][BUILD] Upgrade Kafka to 3.8.0
    omit 34a52ad2ce37 [SPARK-49067][SQL] Move utf-8 literal into internal methods of UrlCodec class
    omit bb5ed954575c [SPARK-49056][SQL] ErrorClassesJsonReader cannot handle null properly
    omit 98c365f6e652 [SPARK-48725][SQL] Integrate CollationAwareUTF8String.lowerCaseCodePoints into string expressions
    omit 5929412e65cd [SPARK-49003][SQL] Fix interpreted code path hashing to be collation aware
    omit 40d94b64e658 [SPARK-49031] Implement validation for the TransformWithStateExec operator using OperatorStateMetadataV2
    omit acb2fecb8c17 [SPARK-49059][CONNECT] Move `SessionHolder.forTesting(...)` to the test package
    omit 33e463ef647d [SPARK-48762][SQL] Introduce clusterBy DataFrameWriter API for Python
    omit e5736441724d [SPARK-49054][SQL] Column default value should support current_* functions
    omit 1ec211e2e294 [SPARK-49053][PYTHON][ML] Make model save/load helper functions accept spark session
    omit 74e200e772b9 [MINOR][DOCS] Fix typos in `docs/sql-migration-guide.md`
    omit 08943104fad9 [SPARK-48997][SS] Implement individual unloads for maintenance thread pool thread failures
    omit 6ff93eaca57e [SPARK-48829][BUILD] Upgrade `RoaringBitmap` to 1.2.1
    omit 76bbcc5d8c19 [SPARK-49032][SS] Add schema path in metadata table entry, verify expected version and add operator metadata related test for operator metadata format v2
    omit f9d63158bb98 [SPARK-48989][SQL] Fix error result of throwing an exception of `SUBSTRING_INDEX` built-in function with Codegen
    omit 332bb0eaf2e1 [SPARK-49049][DOCS] Document MasterPage custom title conf and REST API server-side env variable replacements
    omit 80c44329fec4 [SPARK-48865][SQL][FOLLOWUP] Add failOnError argument to replace TryEval in TryUrlDecode
    omit 308669fc3019 [SPARK-48900] Add `reason` field for all internal calls for job/stage cancellation
    omit 0a1f5ab37b0e [SPARK-48710][PYTHON][FOLLOWUP] PySpark rdd test should not fail on optional dependencies
    omit 0b48d3f61b72 [SPARK-48740][SQL] Catch missing window specification error early
    omit 7e529c3dbfcb [SPARK-49041][PYTHON][CONNECT] Raise proper error for `dropDuplicates` when wrong `subset` is given
    omit 6d4d76463144 [SPARK-49040][SQL] Fix doc `sql-ref-syntax-aux-exec-imm.md`
    omit db036537ab55 [SPARK-48910][SQL] Use HashSet/HashMap to avoid linear searches in PreprocessTableCreation
    omit efc6a75fc8d3 [SPARK-48901][SPARK-48916][SS][PYTHON] Introduce clusterBy DataStreamWriter API
    omit 112a52d58816 [SPARK-49035][PYTHON] Eliminate TypeVar `ColumnOrName_`
    omit 80223bb190ae [SPARK-48985][CONNECT] Connect Compatible Expression Constructors
    omit c463e078881e [SPARK-49013] Change key in collationsMap for Map and Array types in scala
    omit 10849d94bc48 [SPARK-49034][CORE] Support server-side `sparkProperties` replacement in REST Submission API
    omit 388ca1e3064b [SPARK-45891][SQL][PYTHON][VARIANT] Add support for interval types in the Variant Spec
    omit d7966cfd5212 [SPARK-49010][SQL][XML] Add unit tests for XML schema inference case sensitivity
    omit d23ff11d8726 [SPARK-49033][CORE] Support server-side `environmentVariables` replacement in REST Submission API
    omit 163e512c5320 [SPARK-48986][CONNECT][SQL] Add ColumnNode Intermediate Representation
    omit 7e678a07ecd1 [SPARK-49009][SQL][PYTHON] Make Column APIs and functions accept Enums
    omit b02eeff99af8 [SPARK-48999][SS] Divide PythonStreamingDataSourceSimpleSuite
    omit d98480e8b68a [SPARK-49012][SQL][BUILD] Add bouncycastle-related test dependencies to the `hive-thriftserver` module to fix the Maven daily test
    omit 5ccf9ba958f4 [SPARK-48998][ML] Meta algorithms save/load model with SparkSession
    omit 2363aec0c14e [SPARK-49007][CORE] Improve `MasterPage` to support custom title
    omit e73ede7939f2 [SPARK-45787][SQL] Support Catalog.listColumns for clustering columns
    omit f3b819ec642c [SPARK-48503][SQL] Allow grouping on expressions in scalar subqueries, if they are bound to outer rows
    omit 78b83fa542a5 [SPARK-48996][SQL][PYTHON] Allow bare literals for __and__ and __or__ of Column
    omit 4999469173df [SPARK-48849][SS] Create OperatorStateMetadataV2 for the TransformWithStateExec operator
    omit cf95e75cf5eb [MINOR][DOCS] Update doc `sql/README.md`
    omit 5c19505c5f16 [SPARK-48844][FOLLOWUP][TESTS] Cleanup duplicated data resource files in hive-thriftserver test
    omit bafce5de394b [SPARK-48761][SQL] Introduce clusterBy DataFrameWriter API for Scala
    omit 8c625eaaf646 [SPARK-48943][SQL][TESTS][FOLLOWUP] Fix the `h2` filter push-down test case failure with ANSI mode off
    omit 5d787e252942 [SPARK-47764][FOLLOW-UP] Change to use ShuffleDriverComponents.removeShuffle to remove shuffle properly
    omit 34e65a8e7251 [SPARK-48990][SQL] Unified variable related SQL syntax keywords
    omit d68cde812c6f [SPARK-48991][SQL] Move path initialization into try-catch block in FileStreamSink.hasMetadata
    omit 8597b78271fc [SPARK-48988][ML] Make `DefaultParamsReader/Writer` handle metadata with spark session
    omit 0c9b072e1808 [SPARK-48833][SQL][VARIANT] Support variant in `InMemoryTableScan`
    omit 22eb6c4b0a82 [SPARK-48567][SS][FOLLOWUP] StreamingQuery.lastProgress should return the actual StreamingQueryProgress
    omit 239d77b86ca4 [SPARK-48338][SQL] Check variable declarations
    omit 4de4ed10ebd5 [SPARK-48935][SQL][TESTS] Make `checkEvaluation` directly check the `Collation` expression itself in UT
    omit 090ad9f3f400 [SPARK-48961][PYTHON] Make the parameter naming of `PySparkException` consistent with JVM
    omit 4e20a2a6319e [SPARK-48931][SS] Reduce Cloud Store List API cost for state store maintenance task
    omit f828146264d6 [SPARK-48975][PROTOBUF] Remove unnecessary `ScalaReflectionLock` definition from `protobuf`
    omit 881b2142955e [SPARK-48976][SQL][DOCS] Improve the docs related to `variable`
    omit 3fdad6a00dbf [SPARK-48981] Fix simpleString method of StringType in pyspark for collations
    omit b9447db52e87 [SPARK-48987][INFRA] Make `curl` retry 3 times in `bin/mvn`
    omit 877c3f2bea92 [SPARK-48974][SQL][SS][ML][MLLIB] Use `SparkSession.implicits` instead of `SQLContext.implicits`
    omit fdcf975a2d11 [SPARK-48414][PYTHON] Fix breaking change in python's `fromJson`
    omit 118167f444f0 [SPARK-48928] Log Warning for Calling .unpersist() on Locally Checkpointed RDDs
    omit c69f02ec4e5a [SPARK-48752][FOLLOWUP][PYTHON][DOCS] Use explicit name for line number in log
    omit fba4c8c20e52 [SPARK-48970][PYTHON][ML] Avoid using SparkSession.getActiveSession in spark ML reader/writer
    omit 285489b02250 [SPARK-48957][SS] Return sub-classified error class on state store load for hdfs and rocksdb provider
    omit 71f3f1f5d88e [SPARK-48972][PYTHON] Unify the literal string handling in functions
    omit a809740cf6ec [SPARK-48893][SQL][PYTHON][DOCS] Add some examples for `linearRegression` built-in functions
    omit 4cc41ea63f94 [SPARK-48943][TESTS] Upgrade `h2` to 2.3.230 and enhance the test coverage of behavior changes of `asin` and `acos` complying Standard SQL
    omit 0b6cb3e92a03 [SPARK-48914][SQL][TESTS] Add OFFSET operator as an option in the subquery generator
    omit f70d41acefe4 [SPARK-48703][SQL][TESTS] Upgrade `mssql-jdbc` to 12.6.3.jre11
    omit 3b0731537feb [MINOR][INFRA] Add more know translations for contributors
    omit 47726027867a [SPARK-48929] Fix view internal error and clean up parser exception context
    omit bc7a5c0a47b3 [SPARK-48958][BUILD] Upgrade `zstd-jni` to 1.5.6-4
    omit c17fc5a189f9 [SPARK-48963][INFRA] Support `JIRA_ACCESS_TOKEN` in translate-contributors.py
    omit 3af2ea973c45 [SPARK-48959][SQL] Make `NoSuchNamespaceException` extend `NoSuchDatabaseException` to restore the exception handling
    omit be1afd504a44 [SPARK-48941][PYTHON][ML] Replace RDD read / write API invocation with Dataframe read / write API
    omit 8816e9cbccc3 [SPARK-48962][INFRA] Make the input parameters of `workflows/benchmark` selectable
    omit 42d14799c983 [MINOR][DOCS] Fix some typos in `LZFBenchmark`
    omit 6508b1f5e187 [MINOR][PYTHON] Fix type hint for `from_utc_timestamp` and `to_utc_timestamp`
    omit df510ae1a021 [SPARK-48955][SQL] `ArrayCompact`'s datatype should be `containsNull = false`
    omit 2fb3c3571cc0 [SPARK-48891][SS] Refactor StateSchemaCompatibilityChecker to unify all state schema formats
    omit 15c98e0e5e07 [SPARK-48954] try_mod() replaces try_remainder()
    omit 41f67ddbc503 [SPARK-48592][INFRA] Add structured logging style script and GitHub workflow
    omit 578509845752 [SPARK-48938][PYTHON] Improve error messages when registering Python UDTFs
    omit 163256851ac0 [SPARK-48944][CONNECT] Unify the JSON-format schema handling in Connect Server
    omit 69d0c6ac392f [SPARK-48945][PYTHON] Simplify regex functions with `lit`
    omit c629dc0017df [SPARK-48836] Integrate SQL schema with state schema/metadata
    omit 1fa53c77050e [SPARK-48946][SQL] NPE in redact method when session is null
    omit af5eb085b0ac [SPARK-47307][SQL][FOLLOWUP] Promote spark.sql.legacy.chunkBase64String.enabled from a legacy/internal config to a regular/public one
    omit 3f6e2d6ef354 [SPARK-48498][SQL][FOLLOWUP] do padding for char-char comparison
    omit 1645046fbb83 [SPARK-48940][BUILD] Upgrade `Arrow` to 17.0.0
    omit c439bc8d685e [SPARK-48933][BUILD] Upgrade `protobuf-java` to `3.25.3`
    omit d3a90559dc79 Revert "[SPARK-47307][DOCS][FOLLOWUP] Add a migration guide for the behavior change of base64 function"
    omit 110b55857017 [SPARK-48915][SQL][TESTS][FOLLOWUP] Add some uncovered predicates(!=, <, <=, >, >=) for correlation in `GeneratedSubquerySuite`
    omit a90cb0a9fb15 [SPARK-48495][DOCS][FOLLOW-UP] Fix Table Markdown in Shredding.md
    omit a54daa196c1c [SPARK-48934][SS] Python datetime types converted incorrectly for setting timeout in applyInPandasWithState
    omit fab6d83ac70f [SPARK-48921][SQL] ScalaUDF encoders in subquery should be resolved for MergeInto
    omit 723c1c5d30a5 [SPARK-48890][CORE][SS] Add Structured Streaming related fields to log4j ThreadContext
    omit 83a97917d46d [SPARK-48388][SQL] Fix SET statement behavior for SQL Scripts
    omit 1a428c160664 [SPARK-36680][SQL][FOLLOWUP] Files with options should be put into resolveDataSource function
    omit 546da0d5522e [MINOR][SQL][TESTS] Enable test case `testOrcAPI` in `JavaDataFrameReaderWriterSuite`
    omit 3b4c423fec08 [SPARK-48752][PYTHON][CONNECT][DOCS] Introduce `pyspark.logger` for improved structured logging for PySpark
    omit 3831d1ddf0c6 [SPARK-48623][CORE] Migrate FileAppender logs to structured logging
    omit 41b37ae88416 [SPARK-48900] Add `reason` field for `cancelJobGroup` and `cancelJobsWithTag`
    omit 9d4ebf7ec451 [SPARK-48915][SQL][TESTS] Add some uncovered predicates(!=, <=, >, >=) in test cases of `GeneratedSubquerySuite`
    omit 2cd4a4a558d6 [SPARK-48932][BUILD] Upgrade `commons-lang3` to 3.15.0
    omit f26eeb0fa9ba [SPARK-48926][SQL][TESTS] Use `checkError` method to optimize exception check logic related to `UNRESOLVED_COLUMN` error classes
    omit 8b1891438dd6 [SPARK-48510][CONNECT][FOLLOW-UP-MK2] Fix for UDAF `toColumn` API when running tests in Maven
    omit 171c7947d1af [SPARK-48924][PS] Add a pandas-like `make_interval` helper function
    omit 1e17c392b4de [SPARK-48930][CORE] Redact `awsAccessKeyId` by including `accesskey` pattern
    omit ebf8da12d12f [SPARK-48927][CORE] Show the number of cached RDDs in `StoragePage`
    omit 5d16c3134c44 [SPARK-48907][SQL] Fix the value `explicitTypes` in `COLLATION_MISMATCH.EXPLICIT`
    omit 74ca836400b6 [SPARK-48923][SQL][TESTS] Fix the incorrect logic of `CollationFactorySuite`
    omit 41a2bebe725d [SPARK-48865][SQL] Add try_url_decode function
    omit 3a245558be88 [SPARK-48889][SS] testStream to unload state stores before finishing
    omit b2e0a4dfb8d2 [SPARK-47307][DOCS][FOLLOWUP] Add a migration guide for the behavior change of base64 function
    omit c9868b5dadc2 [SPARK-48510][CONNECT][FOLLOW-UP] Fix for UDAF `toColumn` API when running tests in Maven
    omit 2ce5d7280624 [SPARK-48903][SS] Set the RocksDB last snapshot version correctly on remote load
    omit c0f6db83db7a [SPARK-48883][ML][R] Replace RDD read / write API invocation with Dataframe read / write API
    omit 3755d51eb5b8 [SPARK-48892][ML] Avoid per-row param read in `Tokenizer`
    omit 0e8d403fca3c [SPARK-48909][ML][MLLIB] Uses SparkSession over SparkContext when writing metadata
    omit 2e1a39cc6701 [SPARK-48896][ML][MLLIB] Avoid repartition when writing out the metadata
    omit d7f633a53495 [SPARK-48873][SQL] Use UnsafeRow in JSON parser
    omit 3923d7f8fcaf [SPARK-48902][BUILD] Upgrade `commons-codec` to 1.17.1
    omit d0a1c4d0a680 [SPARK-48846][PYTHON][DOCS][FOLLOWUP] Add a missing param doc in python api `partitioning` functions docs
    omit b9b7669fce5c [SPARK-45190][SPARK-48897][PYTHON][CONNECT] Make `from_xml` support StructType schema
    omit 4a44a9ec4a44 [SPARK-48884][PYTHON] Remove unused helper function `PythonSQLUtils.makeInterval`
    omit c0943b349de7 [SPARK-48885][SQL] Make some subclasses of RuntimeReplaceable override replacement to lazy val
    omit d8820a07eb82 [SPARK-47172][DOCS][FOLLOWUP] Fix spark.network.crypto.ciphersince version field on security page
    omit 23080acba7c7 [SPARK-45155][CONNECT] Add API Docs for Spark Connect JVM/Scala Client
    omit 5dafa4c421d3 [SPARK-48350][SQL] Introduction of Custom Exceptions for Sql Scripting
    omit 5ff6a52426f5 [SPARK-48886][SS] Add version info to changelog v2 to allow for easier evolution
    omit 40899c1aca54 [SPARK-48899][K8S] Fix `ENV` key value format in K8s Dockerfiles
    omit b6c05259a0b9 [MINOR][SQL][TESTS] Fix compilation warning `adaptation of an empty argument list by inserting () is deprecated`
    omit 95de02e80dee [SPARK-48441][SQL] Fix StringTrim behaviour for non-UTF8_BINARY collations
    omit 9bff2c8bc505 [SPARK-48463][ML] Make StringIndexer supporting nested input columns
    omit d343b2335993 [SPARK-48894][TESTS] Upgrade `docker-java` to 3.4.0
    omit 08554732f4fd [SPARK-48880][CORE] Avoid throw NullPointerException if driver plugin fails to initialize
    omit 23b2849afa3a [SPARK-48888][SS] Remove snapshot creation based on changelog ops size
    omit effd4d85a312 [SPARK-48834][SQL] Disable variant input/output to python scalar UDFs, UDTFs, UDAFs during query compilation
    omit 206cc1a554ef [SPARK-48613][SQL] SPJ: Support auto-shuffle one side + less join keys than partition keys
    omit e07dc5aac80f [SPARK-48714][SPARK-48794][FOLLOW-UP][PYTHON][DOCS] Add `mergeInto` to API reference
    omit c4766a2d4ea2 [SPARK-48895][R][INFRA] Use R 4.4.1 in `windows` R GitHub Action job
    omit cc321373e223 Revert "[SPARK-48883][ML][R] Replace RDD read / write API invocation with Dataframe read / write API"
    omit b4cd2ecf0616 [SPARK-48495][SQL][DOCS] Describe shredding scheme for Variant
    omit e5e751b98f9e [SPARK-48887][K8S] Enable `spark.kubernetes.executor.checkAllContainers` by default
    omit 115c6e42586f [SPARK-47911][SQL][FOLLOWUP] Rename UTF8 to UTF-8 in spark.sql.binaryOutputStyle
    omit 8e4bbdff80a1 [SPARK-48440][SQL] Fix StringTranslate behaviour for non-UTF8_BINARY collations
    omit 0fa5787d0a6b [SPARK-48883][ML][R] Replace RDD read / write API invocation with Dataframe read / write API
    omit e20db137d2de [SPARK-48510] 2/2] Support UDAF `toColumn` API in Spark Connect
    omit 8d3d4f9b900d [SPARK-47307][SQL] Add a config to optionally chunk base64 strings
    omit 236d95738b6e [SPARK-48845][SQL] GenericUDF catch exceptions from children
    omit d74785359c50 [SPARK-48666][SQL] Do not push down filter if it contains PythonUDFs
    omit 918ca333a900 [SPARK-48878][PYTHON][DOCS] Add doctests for `options` in json functions
    omit 745d602940a7 [SPARK-48876][BUILD] Upgrade Guava used by the connect module to 33.2.1-jre
    omit 60fc6a0710ae [SPARK-48794][CONNECT] df.mergeInto support for Spark Connect (Scala and Python)
    omit 2bec82c05895 [MINOR][TESTS] Remove unused test jar (udf_noA.jar)
    omit 920df93c41ed [SPARK-48877][PYTHON][DOCS] Test the default column name of array functions
    omit 4c7edd2a2048 [SPARK-48864][SQL][TESTS] Refactor `HiveQuerySuite` and fix bug
    omit 9cbd5dd4e747 [SPARK-48871] Fix INVALID_NON_DETERMINISTIC_EXPRESSIONS validation in…
    omit 5bbe9c850aaa [SPARK-48842][DOCS] Document non-determinism of max_by and min_by
    omit 0e940e2080f0 Revert "Remove unused test jar (#47309)"
    omit b560e4ebbd24 Remove unused test jar (#47309)
    omit d274c2cdcbeb [SPARK-48874][SQL][DOCKER][BUILD][TESTS] Upgrade `MySQL` docker image version to `9.0.0`
    omit cbe6846c477b [SPARK-48760][SQL] Fix CatalogV2Util.applyClusterByChanges
    omit 2bee9a611b8e [SPARK-46654][DOCS][FOLLOW-UP] Remove obsolete TODO item
    omit 73eab92d30bf [SPARK-48841][SQL] Include `collationName` to `sql()` of `Collate`
    omit ef71600e1378 [SPARK-46743][SQL][FOLLOW UP] Count bug after ScalarSubqery is folded if it has an empty relation
    omit c1a8966e3091 [SPARK-48872][PYTHON] Reduce the overhead of _capture_call_site
    omit 06bebb884961 [SPARK-48852][CONNECT] Fix string trim function in connect
    omit 52958d9c34d9 [SPARK-48717][FOLLOWUP][PYTHON][SS] Catch Cancelled Job Group wrapped by Py4JJavaError in StreamExecution
    omit 47f85be055d3 [SPARK-48850][DOCS][SS][SQL] Add documentation for new options added to State Data Source
    omit 71cf25ec7b0a [SPARK-48623][CORE] Structured logging migrations [Part 3]
    omit 4501285a49e4 [SPARK-48775][SQL][STS] Replace SQLContext with SparkSession in STS
    omit 297a9d2ac773 [MINOR][SQL][TESTS] Remove a duplicate test case in `CSVExprUtilsSuite`
    omit f8052ddd4208 [SPARK-46738][PYTHON] Reenable a group of doctests
    omit dbc420f02e81 [SPARK-48793][SQL][TESTS][S] Unify v1 and v2 `ALTER TABLE .. `DROP|RENAME` COLUMN ...` tests
    omit 896c15e2a30c [SPARK-48773] Document config "spark.default.parallelism" by config builder framework
    omit 261dbf4a9047 [SPARK-48866][SQL] Fix hints of valid charset in the error message of INVALID_PARAMETER_VALUE.CHARSET
    omit 31d5ea1c3bc4 [SPARK-48863][SQL] Fix ClassCastException when parsing JSON with "spark.sql.json.enablePartialResults" enabled
    omit b4e3c2ad5ac8 [SPARK-48851][SQL] Change the value of `SCHEMA_NOT_FOUND` from `namespace` to `catalog.namespace`
    omit 225b6cbf1297 [SPARK-48280][SQL][FOLLOWUP] Improve collation testing surface area using expression walking
    omit 633238ad84b2 [SPARK-48858][PYTHON] Remove deprecated `setDaemon` method call of `Thread` in `log_communication.py`
    omit 2e3a6fbb342d [SPARK-48529][SQL] Introduction of Labels in SQL Scripting
    omit 97a0f6449f8f [SPARK-48726][SS] Create the StateSchemaV3 file format, and write this out for the TransformWithStateExec operator
    omit 52d95653ada7 [SPARK-48763][FOLLOWUP] Make `dev/lint-scala` error message more accurate
    omit 0d131a023f58 [SPARK-48459][CONNECT][PYTHON][FOLLOWUP] Ignore to_plan from with_origin
    omit 66769888138d [SPARK-48862][PYTHON][CONNECT] Avoid calling `_proto_to_string` when INFO level is not enabled
    omit ddf87b433487 [SPARK-48860][TESTS] Update `ui-test` to use `ws` 8.18.0
    omit 4d13c2231cc4 [SPARK-48763][CONNECT][BUILD] Move connect server and common to builtin module
    omit 0fdebcca1857 Revert "[SPARK-48823][DOCS] Improve clarity in `lag` docstring"
    omit e38383acc121 [SPARK-48844][SQL] USE INVALID_EMPTY_LOCATION instead of UNSUPPORTED_DATASOURCE_FOR_DIRECT_QUERY when path is empty
    omit 8ca1822b4f71 [SPARK-48823][DOCS] Improve clarity in `lag` docstring
    omit 9fa246145f4e [MINOR][DOCS] Add example to `countDistinct`
    omit 06e0e44c26ce [SPARK-48855][K8S][TESTS] Make `ExecutorPodsAllocatorSuite` independent from default allocation batch size
    omit 708bce39db44 [SPARK-48857][SQL] Restrict charsets in CSVOptions
    omit cd2b60dde23d [SPARK-48791][CORE] Fix perf regression caused by the accumulators registration overhead using CopyOnWriteArrayList
    omit a39f70d0c8e8 [SPARK-48843] Prevent infinite loop with BindParameters
    omit 35fbedb54dd3 [SPARK-48854][DOCS] Add missing options in CSV documentation
    omit f2dd0b3338a6 [SPARK-48848][PYTHON][DOCS] Set the upper bound version of `sphinxcontrib-*` in `dev/requirements.txt` with `sphinx==4.5.0`
    omit b13fc16ebfe0 [SPARK-48807][SQL] Binary Support for CSV datasource
    omit a3c1e306a4b3 [SPARK-48772][SS][SQL] State Data Source Change Feed Reader Mode
    omit 7600c97ecb33 [SPARK-36680][SQL] Supports Dynamic Table Options for Spark SQL
    omit f76bc3a843f4 [SPARK-48822][DOCS] Add examples section header to `format_number` docstring
    omit b5f3e1e8ddc3 [SPARK-48817][SQL] Eagerly execute union multi commands together
    omit f73884328cb8 [SPARK-44728][PYTHON][DOCS] Fix the incorrect naming and missing params in func docs in `builtin.py`
    omit 4c99c4df7f9c [SPARK-48716] Add jobGroupId to SparkListenerSQLExecutionStart
    omit d824e9ee659c [SPARK-46625] CTE with Identifier clause as reference
    omit fdbacdf0fb21 [SPARK-48760][SQL][DOCS][FOLLOWUP] Add `CLUSTER BY` to doc `sql-ref-syntax-ddl-alter-table.md`
    omit 43b671865542 [SPARK-48831][CONNECT] Make default column name of `cast` compatible with Spark Classic
    omit 6edfd66bdf4a [MINOR][DOCS] Fix some typos in docs
    omit 18f2450ded5f [SPARK-48743][SQL][SS] MergingSessionIterator should better handle when getStruct returns null
    omit f0cc86d04aeb [SPARK-48840][INFRA] Remove unnecessary existence check for `./dev/free_disk_space_container`
    omit 65daff55f556 [SPARK-48826][BUILD] Upgrade `fasterxml.jackson` to 2.17.2
    omit 5adbc03e2809 [SPARK-48816][SQL] Shorthand for interval converters in UnivocityParser
    omit 82fd7bbca09b [SPARK-48771][SQL] Speed up `LogicalPlanIntegrity.validateExprIdUniqueness` for large query plans
    omit aa3c369c36ff [SPARK-48804][SQL] Add classIsLoadable & OutputCommitter.isAssignableFrom check for output committer class configrations
    omit 75709f547da0 [SPARK-48623][CORE] Structured logging migrations [Part 2]
    omit 97ce5b8964a4 [SPARK-48803][SQL] Throw internal error in Orc(De)serializer to align with ParquetWriteSupport
    omit ba4fb5a50ab6 [SPARK-48837][ML] In CountVectorizer, only read binary parameter once per transform, not once per row
    omit 5b007863432f [SPARK-48801][BUILD][K8S] Upgrade `kubernetes-client` to 6.13.1
    omit d16a88df9ce7 [SPARK-48742][SS] Virtual Column Family for RocksDB
    omit dab708e562ac [SPARK-48307][SQL][FOLLOWUP] Eliminate the use of mutable.ArrayBuffer
    omit 3c3b1129fb6b [MINOR][PYTHON] Eliminating warnings for panda
    omit b062d4436f2b [SPARK-48798][PYTHON] Introduce `spark.profile.render` for SparkSession-based profiling
    omit c4085f1f6f58 [SPARK-48776] Fix timestamp formatting for json, xml and csv
    omit 2c54aa534d32 [SPARK-48343][SQL] Introduction of SQL Scripting interpreter
    omit cd1d687ac0e6 [SPARK-48825][DOCS] Unify the 'See Also' section formatting across PySpark docstrings
    omit 30055f7059b5 [SPARK-48818][PYTHON] Simplify `percentile` functions
    omit 0b0bf4f424c0 [SPARK-48810][CONNECT] Session stop() API should be idempotent and not fail if the session is already closed by the server
    omit 872420dbc7ae [SPARK-48802][SS][FOLLOWUP] FileStreamSource maxCachedFiles set to 0 causes batch with no files to be processed
    omit 2606f388a282 [SPARK-48800][CONNECT][SS] Deflake ClientStreamingQuerySuite
    omit 73126b17515a [SPARK-48720][SQL] Align the command `ALTER TABLE ... UNSET TBLPROPERTIES ...` in v1 and v2
    omit 489e32535aad [SPARK-48177][BUILD][FOLLOWUP] Update parquet version in `sql-data-sources-parquet.md` doc
    omit f1eca903f5c2 [SPARK-48719][SQL] Fix the calculation bug of `RegrSlope` & `RegrIntercept` when the first parameter is null
    omit 6161632a9580 [SPARK-48792][SQL] Fix regression  for INSERT with partial column list to a table with char/varchar
    omit 4c1066d0654d [SPARK-48815][CONNECT] Update environment when stoping connect session
    omit 9f22fa4d2acf [SPARK-48767][SQL] Fix some error prompts when `variant` type data is invalid
    omit a2f8001b59df [SPARK-48783][DOCS] Update the table-valued function docs
    omit 310f8ea2456d [SPARK-48806][SQL] Pass actual exception when url_decode fails
    omit c73c412b172e [SPARK-48808][SQL] Fix NPE when connecting thriftserver through Hive 1.2.1 and the result schema is empty
    omit 54b75582506d [SPARK-48805][SQL][ML][SS][AVRO][EXAMPLES] Replace calls to bridged APIs based on `SparkSession#sqlContext` with `SparkSession` API
    omit d5dc22356d23 [SPARK-48280][SQL] Improve collation testing surface area using expression walking
    omit bf25f0ac8002 [SPARK-48715][SQL] Integrate UTF8String validation into collation-aware string function implementations
    omit f4386745a6b7 [SPARK-48799][SS] Refactor versioning for operator metadata read/write and callers
    omit 8ace648a8357 [SPARK-47046][BUILD][TESTS] Upgrade `mysql-connector-j` to 9.0.0
    omit 906079202a93 [SPARK-48780][SQL] Make errors in NamedParametersSupport generic to handle functions and procedures
    omit b93bfcd95e16 [SPARK-48787][BUILD] Upgrade Kafka to 3.7.1
    omit 2e210d9390d7 [SPARK-48790][TESTING] Use checkDatasetUnorderly in DeprecatedDatasetAggregatorSuite
    omit bb59267abecb [MINOR][TESTS] Replace `getResource` with `getWorkspaceFilePath` to enable `HiveUDFSuite` to run successfully in the IDE
    omit d410e0631fbf [SPARK-48282][SQL][FOLLOWUP] Fix FindInSet code generation
    omit b8cc91cef909 [SPARK-48760][SQL] Introduce ALTER TABLE ... CLUSTER BY SQL syntax to change clustering columns
    omit 955190705308 [SPARK-48774][SQL] Use SparkSession in SQLImplicits
    omit 15f216774ee5 [SPARK-48785][DOCS] Add a simple Python data source example in the user guide
    omit 6d226c1e2aba [SPARK-48714][PYTHON] Implement `DataFrame.mergeInto` in PySpark
    omit 0aa32e42f97c [SPARK-48710][PYTHON] Use NumPy 2.0 compatible types
    omit fea930ab8d6e [SPARK-48770][SS] Change to read operator metadata once on driver to check if we can find info for numColsPrefixKey used for session window agg queries
    omit ee0d30686c4f [SPARK-48589][SQL][SS] Add option snapshotStartBatchId and snapshotPartitionId to state data source
    omit db9e1ac92bea [SPARK-48177][BUILD] Upgrade `Apache Parquet` to 1.14.1
    omit 4ee37edf063a [SPARK-48764][PYTHON] Filtering out IPython-related frames from user stack
    omit 8a5f4e0a1cdb [SPARK-48759][SQL] Add migration doc for CREATE TABLE AS SELECT behavior change behavior change since Spark 3.4
    omit 930422389352 [SPARK-44718][FOLLOWUP][DOCS] Avoid using ConfigEntry in spark.sql.columnVector.offheap.enabled's doc field
    omit 5ac7c9bdb6ca [SPARK-48766][PYTHON] Document the behavior difference of `extraction` between `element_at` and `try_element_at`
    omit 5c29d8d505a9 [SPARK-48768][PYTHON][CONNECT] Should not cache `explain`
    omit 703b07625aa4 [SPARK-48697][SQL] Add collation aware string filters
    omit 48eb4d53d651 [SPARK-48737] Perf improvement during analysis - Create exception only when it is necessary for default expression resolving
    omit cd1f1af86924 [SPARK-48747][SQL] Add code point iterator to UTF8String
    omit bc16b24c7a32 [SPARK-48765][DEPLOY] Enhance default value evaluation for SPARK_IDENT_STRING
    omit 399980edaa81 [MINOR][DOCS] Fix the type hints of `functions.first(..., ignorenulls)` and `functions.last(..., ignorenulls)`
    omit f70ce135ba1e [SPARK-48638][INFRA][FOLLOW-UP] Add graphviz into CI to run the related tests
    omit a84a6a49f98d [SPARK-48749][SQL] Simplify UnaryPositive and eliminate its Catalyst Rules with RuntimeReplaceable
    omit 0487d7857ecf [SPARK-48748][SQL] Cache numChars in UTF8String
    omit f49418b79c88 [SPARK-48751][INFRA][PYTHON][TESTS] Re-balance `pyspark-pandas-connect` tests on GA
    omit 6bfeb0942482 [SPARK-48757][CORE] Make `IndexShuffleBlockResolver` have explicit constructors
    omit 4e57f0629ece [SPARK-48307][SQL][FOLLOWUP] not-inlined CTE references sibling should not fail
    omit 80277eefae75 [SPARK-48745][INFRA][PYTHON][TESTS] Remove unnecessary installation `miniconda` from `build_and_test.yml`
    omit fc98ccdf8f7e [SPARK-48746][PYTHON][SS][TESTS] Avoid using global temp view in foreachBatch test case
    omit 69f3a9b969f3 [SPARK-48586][SS][FOLLOWUP] RocksDB and RocksDBFileManager code style changes
    omit a2c4be0a834f [SPARK-47233][CONNECT][SS][FOLLOW-UP] Add eventually for terminated event assert
    omit 9141aa4bb2af [SPARK-48744][CORE] Log entry should be constructed only once
    omit 2eeebef81a11 [SPARK-46957][CORE][FOLLOW-UP] Use Collections.emptyMap for Java compatibility
    omit 3bf7de099132 [SPARK-48668][SQL] Support ALTER NAMESPACE ... UNSET PROPERTIES in v2
    omit 8cd095f297c8 [SPARK-48738][SQL] Correct since version for built-in func alias `random`, `position`, `mod`, `cardinality`, `current_schema`, `user`, `session_user`,`char_length`,`character_length`
    omit 6304484a14a8 [SPARK-48282][SQL] Alter string search logic for UTF8_BINARY_LCASE collation (StringReplace, FindInSet)
    omit 2c9eb1c1f00f [SPARK-48682][SQL] Use ICU in InitCap expression for UTF8_BINARY strings
    omit 1cdd5fa68715 [SPARK-48736][PYTHON] Support infra fro additional includes for Python UDFs
    omit df13ca05c475 [SPARK-48735][SQL] Performance Improvement for BIN function
    omit baf461b5cb44 [SPARK-48708][CORE] Remove three unnecessary type registrations from `KryoSerializer`
    omit 40ad8290e997 [SPARK-48586][SS] Remove lock acquisition in doMaintenance() by making a deep copy of file mappings in RocksDBFileManager in load()
    omit b5a55e46e9c1 [SPARK-46957][CORE] Decommission migrated shuffle files should be able to cleanup from executor
    omit b11608c96f8a [SPARK-48428][SQL] Fix IllegalStateException in NestedColumnAliasing
    omit d89aad3ada57 [SPARK-47927][SQL][FOLLOWUP] fix ScalaUDF output nullability
    omit 7bf911946099 [SPARK-48734][PYTHON][TESTS] Separate local cluster test from test_artifact to test_artifact_localcluster
    omit c788d1255aa0 [SPARK-48733][PYTHON][TESTS] Do not test SET command in Python UDTF test
    omit 5b53c6ca74e9 [MINOR][PYTHON][DOCS] Fix indents in function API references
    omit 642c4bb24d2e [SPARK-48639][CONNECT][PYTHON] Add Origin to RelationCommon
    omit 59439051dc7c [MINOR][PYTHON][TESTS] Remove duplicate schema checking
    omit ff2d17777f9d [MINOR][DOCS] Make pivot doctest deterministic
    omit 58d1a892faf8 [SPARK-48555][PYTHON][FOLLOW-UP] Simplify the support of `Any` parameters
    omit b154623d9fac [MINOR][TESTS] Always remove spark.master in ReusedConnectTestCase
    omit 2e315722c2ea Revert "[SPARK-48639][CONNECT][PYTHON] Add Origin to Relation.RelationCommon"
    omit 48f39b898c36 [SPARK-48729][SQL] Add a UserDefinedFunction interface to represent a SQL function
    omit 7c7c1965878b [SPARK-48712][SQL] Perf Improvement for encode with empty values or UTF-8 charset
    omit ea0cd0151a26 [SPARK-39627][SQL][FOLLOWUP] Cleanup deprecated api usage related to `JdbcDialect.compileAggregate`
    omit ec7dde792a85 [SPARK-42610][CONNECT][FOLLOWUP] Add some test cases for `SQLImplicits.newSequenceEncoder`
    omit 14272e8b9c22 [SPARK-48721][SQL][DOCS] Fix the doc of `decode` function in SQL API page
    omit b5f76b130194 [SPARK-48723][INFRA] Run `git cherry-pick --abort` if backporting is denied by committer
    omit 906af781492a [SPARK-48578][SQL][FOLLOWUP] add UTF8 string validation related functions
    omit 1581264a5d77 [SPARK-48717][PYTHON][SS] Catch ForeachBatch py4j InterruptedException better in classic PySpark
    omit 7f5f96cad224 [SPARK-48691][BUILD] Upgrade scalatest related dependencies to the 3.2.19 series
    omit a50b30d7a02b [SPARK-48687][SS] Add change to perform state schema validation and update on driver in planning phase for stateful queries
    omit a474b88aa2eb [SPARK-48724][SQL][TESTS] Fix incorrect conf settings of `ignoreCorruptFiles` related tests case in `ParquetQuerySuite`
    omit e23d69b0f1da [SPARK-48709][SQL] Fix varchar type resolution mismatch for DataSourceV2 CTAS
    omit 313479c45421 [SPARK-48713][SQL] Add index range check for UnsafeRow.pointTo when baseObject is byte array
    omit 4cf54508fd23 [SPARK-48699][SQL][FOLLOWUP] Refine collation API
    omit 0fc5b0b9383b [SPARK-47353][SQL] Enable collation support for the Mode expression
    omit 7a1608bbc3f1 [SPARK-48670][SQL] Providing suggestion as part of error message when invalid collation name is given
    omit ec0ee863562e [SPARK-48699][SQL] Refine collation API
    omit bb21861a9580 [SPARK-48059][CORE][FOLLOWUP] Fix bug for SparkLogger
    omit ee3a612b28c1 [SPARK-48705][PYTHON] Explicitly use worker_main when it starts with pyspark
    omit 07cbba64e3ea [SPARK-48706][PYTHON] Python UDF in higher order functions should not throw internal error
    omit 169346caa9b2 [SPARK-48578][SQL][FOLLOWUP] Fix `dev/scalastyle` error for `ExpressionImplUtilsSuite`
    omit c459afbba406 [SPARK-48573][SQL] Upgrade ICU version
    omit ebacb9163f26 [SPARK-48718][SQL] Handle and fix the case when deserializer in cogroup is resolved during application of DeduplicateRelation rule
    omit 9d4abaf19fa4 [SPARK-48638][CONNECT] Add ExecutionInfo support for DataFrame
    omit 59289083828e [SPARK-48466][SQL][FOLLOWUP] Fix missing pattern match in EmptyRelationExec
    omit 068be4b10255 [SPARK-48578][SQL] add UTF8 string validation related functions
    omit 8c4ca7e61228 [SPARK-48693][SQL] Simplify and unify toString of Invoke and StaticInvoke
    omit 51f11033f86e [SPARK-48686][SQL] Improve performance of ParserUtils.unescapeSQLString
    omit b49479b43e1f [SPARK-48704][INFRA] Update `build_sparkr_window.yml` to use `windows-2022`
    omit d47f34f3182c [SPARK-48629] Migrate the residual code to structured logging framework
    omit 5112e5887714 [SPARK-48692][BUILD] Upgrade `rocksdbjni` to 9.2.1
    omit 2ac2710b46be [SPARK-48702][INFRA] Fix `Python CodeGen check`
    omit fb5697d6873c [SPARK-48658][SQL] Encode/Decode functions report coding errors instead of mojibake for  unmappable characters
    omit 8e02a6493ef5 [SPARK-48695][PYTHON] `TimestampNTZType.fromInternal` not use the deprecated methods
    omit 09cb5921b7d8 [SPARK-48639][CONNECT][PYTHON] Add Origin to Relation.RelationCommon
    omit e459674127e7 [SPARK-48683][SQL] Fix schema evolution with `df.mergeInto` losing `when` clauses
    omit 8b16196ad89a [SPARK-48680][SQL][DOCS] Add missing Java APIs and language-specific documentations for char/varchar
    omit a7dc020bc0e5 [SPARK-48681][SQL] Use ICU in Lower/Upper expressions for UTF8_BINARY strings
    omit 4663b84eed19 [SPARK-47681][FOLLOWUP] Fix schema_of_variant for float inputs
    omit 31fa9d839af0 [SQL][TEST][FOLLOWUP] Re-run collation benchmark (NonASCII)
    omit 88cc1530a166 [SPARK-48650][PYTHON] Display correct call site from IPython Notebook
    omit e972daec30eb [SPARK-48688][SQL] Return reasonable error when calling SQL to_avro and from_avro functions but Avro is not loaded by default
    omit 4b37eb8169c9 [SPARK-48678][CORE] Performance optimizations for SparkConf.get(ConfigEntry)
    omit 84d278ca99e3 [MINOR] Fix some typos in `error-states.json`
    omit c8d75c1f9f61 [SPARK-48620][PYTHON][FOLLOW-UP] Correct the error message for `CalendarIntervalType`
    omit b1677a425408 [SPARK-48545][SQL] Create to_avro and from_avro SQL functions to match DataFrame equivalents
    omit 7e5a46106bca [SPARK-48655][SQL] SPJ: Add tests for shuffle skipping for aggregate queries
    omit 941421144da6 [SPARK-48490][CORE][TESTS][FOLLOWUP] Add some UT for the Windows paths scene
    omit 0bc38acc615a [SPARK-48675][SQL] Fix cache table with collated column
    omit f0563ef64c7f [SPARK-47172][CORE] Add support for AES-GCM for RPC encryption
    omit 32861e0d4fb1 [SPARK-48684][INFRA] Print related JIRA summary before proceeding merge
    omit 97d3addd2129 Revert "Revert "[SPARK-48554][INFRA] Use R 4.4.0 in `windows` R GitHub Action Window job""
    omit b5d0d0721f37 [SPARK-48662][SQL] Fix StructsToXml expression with collations
    omit fdabe08294e6 [SPARK-48490][CORE][FOLLOWUP] Properly process escape sequences
    omit cd8bf110b7f5 [SPARK-48659][SQL][TESTS] Unify v1 and v2 ALTER TABLE .. SET TBLPROPERTIES tests
    omit 3469ec6b4196 [SPARK-48656][CORE] Do a length check and throw COLLECTION_SIZE_LIMIT_EXCEEDED error in `CartesianRDD.getPartitions`
    omit b99bb00ea11b [SPARK-48630][INFRA] Make `merge_spark_pr` keep the format of revert PR
    omit 62bad5301f5f [SPARK-48672][DOC] Update Jakarta Servlet reference in security page
    omit f077759460c9 [SPARK-48631][CORE][TEST] Fix test "error during accessing host local dirs for executors"
    omit 67c71874bcb2 [SPARK-48661][BUILD] Upgrade `RoaringBitmap` to 1.1.0
    omit d8099a201d24 [SPARK-48479][SQL][FOLLOWUP] Consolidate createOrReplaceTableColType and colDefinitionType in parser
    omit 6eb797808622 [SQL][TEST] Re-run collation benchmark
    omit e68b8ca9bf4d [SPARK-48677][BUILD] Upgrade `scalafmt` to 3.8.2
    omit b0b02b231aef [SPARK-48653][PYTHON] Fix invalid Python data source error class references
    omit 904d4ddc31bb [SPARK-48635][SQL] Assign classes to join type errors and as-of join error
    omit 714699ba73c6 [SPARK-47911][SQL][FOLLOWUP] Enable binary format tests in ThriftServerQueryTestSuite
    omit 955349f6d970 [SPARK-48620][PYTHON] Fix internal raw data leak in `YearMonthIntervalType` and `CalendarIntervalType`
    omit 692d8692ef08 [SPARK-48591][PYTHON] Add a helper function to simplify `Column.py`
    omit 0d9f8a107f57 [SPARK-48479][SQL] Support creating scalar and table SQL UDFs in parser
    omit 9eadb2cbcc90 [SPARK-48634][PYTHON][CONNECT][FOLLOW-UP] Do not make a request if threadpool is not initialized
    omit 248fd4c67101 [SPARK-48342][FOLLOWUP][SQL] Remove unnecessary import in AstBuilder
    omit 5d6e9dd6b121 [SPARK-47986][CONNECT][FOLLOW-UP] Unable to create a new session when the default session is closed by the server
    omit 54587638685b [SPARK-48541][CORE] Add a new exit code for executors killed by TaskReaper
    omit 484e7ac9acef [SPARK-48472][SQL] Enable reflect expressions with collated strings
    omit 3ac31b1b6eaf [SPARK-48574][SQL] Fix support for StructTypes with collations
    omit 2fe0692e94cc [SPARK-48466][SQL] Create dedicated node for EmptyRelation in AQE
    omit b0e2cb575390 [SPARK-48623][CORE] Structured Logging Migrations
    omit d067fc6c1635 Revert "[SPARK-48567][SS] StreamingQuery.lastProgress should return the actual StreamingQueryProgress"
    omit 1e868b2a7a17 Revert "[SPARK-48554][INFRA] Use R 4.4.0 in `windows` R GitHub Action Window job"
    omit b77caf776368 [SPARK-48651][DOC] Configuring different JDK for Spark on YARN
    omit 878dd6abce76 [SPARK-48601][SQL] Give a more user friendly error message when setting a null value for JDBC Option
    omit 5e28e9532978 [SPARK-48649][SQL] Add "ignoreInvalidPartitionPaths" and "spark.sql.files.ignoreInvalidPartitionPaths" configs to allow ignoring invalid partition paths
    omit 6ee7c257b9fb [SPARK-48634][PYTHON][CONNECT] Avoid statically initialize threadpool at ExecutePlanResponseReattachableIterator
    omit 47ffe4094d9d [SPARK-48646][PYTHON] Refine Python data source API docstring and type hints
    omit 80bba4463eba [SPARK-48459][CONNECT][PYTHON] Implement DataFrameQueryContext in Spark Connect
    omit 58701d811d95 [SPARK-47148][SQL][FOLLOWUP] Use broadcast hint to make test more stable
    omit 9898e9d3e4a5 [SPARK-48342][SQL] Introduction of SQL Scripting Parser
    omit a3feffdff9cd [SPARK-48585][SQL] Make `built-in` JdbcDialect's method `classifyException` throw out the `original` exception
    omit c5809b6f6d15 [SPARK-48647][PYTHON][CONNECT] Refine the error message for `YearMonthIntervalType` in `df.collect`
    omit 05c87e51a5e5 [SPARK-48644][SQL] Do a length check and throw COLLECTION_SIZE_LIMIT_EXCEEDED error in Hex.hex
    omit 738acd16847b [SPARK-48648][PYTHON][CONNECT] Make SparkConnectClient.tags properly threadlocal
    omit e265c6041aa2 [SPARK-47910][CORE] close stream when DiskBlockObjectWriter closeResources to avoid memory leak
    omit f0b7cfa56cb9 [SPARK-48497][PYTHON][DOCS] Add an example for Python data source writer in user guide
    omit 042804ad545c [SPARK-48567][SS] StreamingQuery.lastProgress should return the actual StreamingQueryProgress
    omit d8a24b7a9432 [SPARK-48645][BUILD] Upgrade Maven to 3.9.8
    omit 00a96bb42ec6 [SPARK-48642][CORE] False SparkOutOfMemoryError caused by killing task on spilling
    omit 0864bbeac399 [SPARK-48566][PYTHON] Fix bug where partition indices are incorrect when UDTF analyze() uses both select and partitionColumns
    omit 66d8a29dd7b4 [SPARK-47577][SPARK-47579] Correct misleading usage of log key TASK_ID
    omit 8fdd85f09779 [SPARK-48603][TEST] Update *ParquetReadSchemaSuite to cover type widen capability
    omit 9ef092f19aaa [SPARK-48641][BUILD] Upgrade `curator` to 5.7.0
    omit d3455dfea8a7 [SPARK-48572][SQL] Fix DateSub, DateAdd, WindowTime, TimeWindow and SessionWindow expressions
    omit e00d26fcf24f [SPARK-48600][SQL] Fix FrameLessOffsetWindowFunction expressions implicit casting
    omit d3da240ee302 [SPARK-48610][SQL] refactor: use auxiliary idMap instead of OP_ID_TAG
    omit 90d302aeb788 [SPARK-48557][SQL] Support scalar subquery with group-by on column equal to constant
    omit 0c16624c77ad [SPARK-48627][SQL] Perf improvement for binary to to HEX_DISCRETE string
    omit 71475f7084b9 [SPARK-48577][SQL] Invalid UTF-8 byte sequence replacement
    omit 0a4b112d87a3 [SPARK-48633][BUILD] Upgrade `scalacheck` to 1.18.0
    omit 42cd961fa97d [SPARK-48587][VARIANT] Avoid storage amplification when accessing a sub-Variant
    omit 257a7883f215 [SPARK-48615][SQL] Perf improvement for parsing hex string
    omit 33a9c5d7f478 [MINOR][PYTHON][DOCS] Fix pyspark.sql.functions.reduce docstring typo
    omit 8f662fc2d0ea [SPARK-48555][SQL][PYTHON][CONNECT] Support using Columns as parameters for several functions in pyspark/scala
    omit 9881e0a3fec1 [SPARK-47777] fix python streaming data source connect test
    omit c09039ad2a95 [SPARK-48597][SQL] Introduce a marker for isStreaming property in text representation of logical plan
    omit 347f9c64645b [SPARK-48302][PYTHON] Preserve nulls in map columns in PyArrow Tables
    omit 0775ea732a38 [SPARK-48611][CORE] Log TID for input split in HadoopRDD and NewHadoopRDD
    omit 8ee8abaa599f [SPARK-48621][SQL] Fix Like simplification in Optimizer for collated strings
    omit aa4bfb05a0c3 Revert "[SPARK-48591][PYTHON] Simplify the if-else branches with `F.lit`"
    omit 2d2bedf4aa16 [SPARK-48056][CONNECT][FOLLOW-UP] Scala Client re-execute plan if a SESSION_NOT_FOUND error is raised and no partial response was received
    omit dd8b05f25fdc [SPARK-42252][CORE] Add `spark.shuffle.localDisk.file.output.buffer` and deprecate `spark.shuffle.unsafe.file.output.buffer`
    omit 878de0014a37 [SPARK-48626][CORE] Change the scope of object LogKeys as private in Spark
    omit 38318863f641 [SPARK-48625][BUILD] Upgrade `mssql-jdbc` to 12.6.2.jre11
    omit 157b1e3ba5d5 [SPARK-48612][SQL][SS] Cleanup deprecated api usage related to commons-pool2
    omit 75fff90d2618 [SPARK-45685][SQL][FOLLOWUP] Add handling for `Stream` where `LazyList.force` is called
    omit 0b214f166a92 [MINOR][DOCS][TESTS] Update repo name and link from `parquet-mr` to `parquet-java`
    omit 70bdcc97910e [MINOR][DOCS] Fix metrics info of shuffle service
    omit be154a371df0 [SPARK-48622][SQL] get SQLConf once when resolving column names
    omit 08e741b92b8f [SPARK-48604][SQL] Replace deprecated `new ArrowType.Decimal(precision, scale)` method call
    omit bdcb79f23da3 [SPARK-48543][SS] Track state row validation failures using explicit error class
    omit b8c7aee12f02 [SPARK-48609][BUILD] Upgrade `scala-xml` to 2.3.0
    omit 78fd4e3301ff [SPARK-48584][SQL][FOLLOWUP] Improve the unescapePathName
    omit ea2bca74923e [SPARK-48602][SQL] Make csv generator support different output style with spark.sql.binaryOutputStyle
    omit fd045c9887fe [SPARK-48583][SQL][TESTS] Replace deprecated classes and methods of `commons-io` called in Spark
    omit 39885489465d [SPARK-48593][PYTHON][CONNECT] Fix the string representation of lambda function
    omit c059c8402df6 [SPARK-48421][SQL] SPJ: Add documentation
    omit 0bbd049a9ade [SPARK-48591][PYTHON] Simplify the if-else branches with `F.lit`
    omit d1d29c9840fe [SPARK-48598][PYTHON][CONNECT] Propagate cached schema in dataframe operations
    omit 2d0b12299211 [SPARK-48594][PYTHON][CONNECT] Rename `parent` field to `child` in `ColumnAlias`
    omit b5e1b7988031 [SPARK-48596][SQL] Perf improvement for calculating hex string for long
    omit a3625a98e78c [SPARK-48595][CORE] Cleanup deprecated api usage related to `commons-compress`
    omit da81d8ecb802 [SPARK-48584][SQL] Perf improvement for unescapePathName
    omit 8870efce19f2 [SPARK-48581][BUILD] Upgrade dropwizard metrics to 4.2.26
    omit 334816af5f86 [SPARK-48411][SS][PYTHON] Add E2E test for DropDuplicateWithinWatermark
    omit 72df3cb1a43b [SPARK-48582][BUILD] Upgrade `braces` from 3.0.2 to 3.0.3 in ui-test
    omit 82a84ede6a47 [SPARK-46937][SQL] Revert "[] Improve concurrency performance for FunctionRegistry"
    omit 61078366b672 [SPARK-48576][SQL][FOLLOWUP] Rename UTF8_BINARY_LCASE to UTF8_LCASE
    omit aad6771aac3d [SPARK-48576][SQL] Rename UTF8_BINARY_LCASE to UTF8_LCASE
    omit 224ba162b5d6 [SPARK-48556][SQL] Fix incorrect error message pointing to UNSUPPORTED_GROUPING_EXPRESSION
    omit 583ab0500c79 [SPARK-47415][SQL] Add collation support for Levenshtein expression
    omit df4156aa3217 [SPARK-48372][SPARK-45716][PYTHON][FOLLOW-UP] Remove unused helper method
    omit 452c1b64b625 [SPARK-48551][SQL] Perf improvement for escapePathName
    omit 53d65fd12dd9 [SPARK-48565][UI] Fix thread dump display in UI
    omit 1e4750e63403 [SPARK-47500][PYTHON][CONNECT][FOLLOWUP] Restore error message for `DataFrame.select(None)`
    omit 3fe6abde125b [SPARK-48563][BUILD] Upgrade `pickle` to 1.5
    omit 5a2f374a208f [SPARK-48544][SQL] Reduce memory pressure of empty TreeNode BitSets
    omit ec6db63ca6ac [SPARK-48569][SS][CONNECT] Handle edge cases in query.name
    omit 3857a9d36d4a [SPARK-48410][SQL] Fix InitCap expression for UTF8_BINARY_LCASE & ICU collations
    omit 61fd936c8939 [SPARK-48403][SQL] Fix Lower & Upper expressions for UTF8_BINARY_LCASE & ICU collations
    omit 190166908ba0 [SPARK-48564][PYTHON][CONNECT] Propagate cached schema in set operations
    omit d9394eee5ebb [SPARK-48560][SS][PYTHON] Make StreamingQueryListener.spark settable
    omit 24bce72c9065 [SPARK-48012][SQL] SPJ: Support Transfrom Expressions for One Side Shuffle
    omit 201df0d7ac81 [MINOR][PYTHON][TESTS] Move a test out of parity tests
    omit 8911d59005e8 [SPARK-46393][SQL][FOLLOWUP] Classify exceptions in JDBCTableCatalog.loadTable and Fix UT
    omit d81b1e3d358c [SPARK-48559][SQL] Fetch globalTempDatabase name directly without invoking initialization of GlobalaTempViewManager
    omit 87b0f5995383 [SPARK-48561][PS][CONNECT] Throw `PandasNotImplementedError` for unsupported plotting functions
    omit b7d9c317aa2e Revert "[SPARK-46393][SQL][FOLLOWUP] Classify exceptions in JDBCTableCatalog.loadTable"
    omit 94912920b0e9 [SPARK-48548][BUILD] Add LICENSE/NOTICE for spark-core with shaded dependencies
    omit 82b4ad2af648 [SPARK-46393][SQL][FOLLOWUP] Classify exceptions in JDBCTableCatalog.loadTable
    omit 56b9b134041b [SPARK-48552][SQL] multi-line CSV schema inference should also throw FAILED_READ_FILE
    omit 9de0a2eca9ce [SPARK-42944][FOLLOWUP][SS][CONNECT] Reenable ApplyInPandasWithState tests
    omit cf3051b18131 [MINOR][PYTHON] add export class and fix doc typo for python streaming data source
    omit 3c4cb407b5ca [SPARK-47952][CORE][CONNECT] Support retrieving the real SparkConnectService GRPC address and port programmatically when running on Yarn
    omit edb9236ea688 [SPARK-48504][PYTHON][CONNECT][FOLLOW-UP] Code clean up
    omit ce1b08f6e30b [SPARK-48553][PYTHON][CONNECT] Cache more properties
    omit 0f21df0b29cc [SPARK-48286] Fix analysis of column with exists default expression - Add user facing error
    omit 84fa0527834b [SPARK-48283][SQL] Modify string comparison for UTF8_BINARY_LCASE
    omit b5a4b3200362 [SPARK-48435][SQL] UNICODE collation should not support binary equality
    omit 3878b57e6e88 [SPARK-48526][SS] Allow passing custom sink to testStream()
    omit 9f4007f3d89e [SPARK-48546][SQL] Fix ExpressionEncoder after replacing NullPointerExceptions with proper error classes in AssertNotNull expression
    omit 7cba1ab4d6ac [SPARK-48554][INFRA] Use R 4.4.0 in `windows` R GitHub Action Window job
    omit 65db87697949 [SPARK-48513][SS] Add error class for state schema compatibility and minor refactoring
    omit ab00533221e2 [SPARK-47933][PYTHON][TESTS][FOLLOW-UP] Enable doctest `pyspark.sql.connect.column`
    omit 8cb78a7811f3 [SPARK-48550][PS] Directly use the parent Window class
    omit f4434c36cc4f [SPARK-48540][CORE] Avoid ivy output loading settings to stdout
    omit b3700ac09861 [SPARK-48539][BUILD][TESTS] Upgrade docker-java to 3.3.6
    omit 966c3d9ef1ed [SPARK-47552][CORE][FOLLOWUP] Set spark.hadoop.fs.s3a.connection.establish.timeout to numeric
    omit 31ce2db6d208 [SPARK-48538][SQL] Avoid HMS memory leak casued by bonecp
    omit d5c33c6bfb57 [SPARK-48307][SQL][FOLLOWUP] Allow outer references in un-referenced CTE relations
    omit 490a4b3b1fdf [SPARK-48498][SQL] Always do char padding in predicates
    omit 34ac7de89711 [SPARK-48536][PYTHON][CONNECT] Cache user specified schema in applyInPandas and applyInArrow
    omit 88b8dc29e100 [SPARK-46937][SQL][FOLLOWUP] Properly check registered function replacement
    omit c4f720dfb419 [SPARK-48535][SS] Update config docs to indicate possibility of data loss/corruption issue if skip nulls for stream-stream joins config is enabled
    omit db527ac346f2 Revert "[SPARK-48505][CORE] Simplify the implementation of `Utils#isG1GC`"
    omit adbfd17318bf [SPARK-48533][CONNECT][PYTHON][TESTS] Add test for cached schema
    omit 4075ce677120 [SPARK-48374][PYTHON][TESTS][FOLLOW-UP] Explicitly enable ANSI mode for non-ANSI build
    omit a17ab572cfda [MINOR][DOCS] Fix a typo in core-migration-guide.md
    omit 33aa467f7582 [SPARK-48523][DOCS] Add `grpc_max_message_size ` description to `client-connection-string.md`
    omit e47ce476b9ac [SPARK-48485][CONNECT][SS] Support interruptTag and interruptAll in streaming queries
    omit 8a0927c07a14 [SPARK-48307][SQL] InlineCTE should keep not-inlined relations in the original WithCTE node
    omit c7caac9b10ca [SPARK-47972][SQL][FOLLOWUP] Restrict CAST expression for collations
    omit 651f68782ab7 [SPARK-48531][INFRA] Fix `Black` target version to Python 3.9
    omit 8b88f5ae10cc [SPARK-48522][BUILD] Update Stream Library to 2.9.8 and attach its NOTICE
    omit f4afa2215a1a [SPARK-48506][CORE] Compression codec short names are case insensitive except for event logging
    omit d273fdf37bc2 [SPARK-48519][BUILD] Upgrade jetty to 11.0.21
    omit 90ee29992522 [SPARK-48518][CORE] Make LZF compression be able to run in parallel
    omit 02c645607f43 [SPARK-48512][PYTHON][TESTS] Refactor Python tests
    omit 8cebb9b56cc3 [SPARK-46937][SQL] Improve concurrency performance for FunctionRegistry
    omit abbe301d7645 [SPARK-48505][CORE] Simplify the implementation of `Utils#isG1GC`
    omit c852c4f72acb [SPARK-48318][SQL] Enable hash join support for all collations (complex types)
    omit 6475ddfed7f4 [SPARK-48514][BUILD][K8S] Upgrade `kubernetes-client` to 6.13.0
    omit 560c08332b35 [SPARK-48482][PYTHON] dropDuplicates and dropDuplicatesWIthinWatermark should accept variable length args
    omit 6272c0511d23 [SPARK-48508][CONNECT][PYTHON] Cache user specified schema in `DataFrame.{to, mapInPandas, mapInArrow}`
    omit cfb79d9d44c4 [SPARK-47933][PYTHON][FOLLOWUP] Correct the error message
    omit e4e8bb5936d3 [SPARK-47972][SQL] Restrict CAST expression for collations
    omit f9542d008402 [SPARK-48413][SQL] ALTER COLUMN with collation
    omit 5baaa615ef85 [SPARK-47977] DateTimeUtils.timestampDiff and DateTimeUtils.timestampAdd should not throw INTERNAL_ERROR exception
    omit 5d71ef0716f7 [SPARK-48503][SQL] Fix invalid scalar subqueries with group-by on non-equivalent columns that were incorrectly allowed
    omit 7e8b60b5ae7d [SPARK-48507][INFRA] Use Hadoop 3.3.6 winutils in `build_sparkr_window`
    omit 67ad55c46347 [SPARK-48504][PYTHON][CONNECT] Parent Window class for Spark Connect and Spark Classic
    omit 8d9d9c243ed1 Revert "[SPARK-45891] Describe shredding scheme for Variant"
    omit 6b12b317e42d [SPARK-48447][SS] Check state store provider class before invoking the constructor
    omit bf2fc24967ba [SPARK-48383][SS] Throw better error for mismatched partitions in startOffset option in Kafka
    omit 168230a81287 [SPARK-48488][CORE] Fix methods `log[info|warning|error]` in `SparkSubmit`
    omit 32832af093ab [SPARK-48204][INFRA][FOLLOW] fix release scripts for the "finalize" step
    omit 9270931221d4 [SPARK-48433][BUILD] Upgrade `checkstyle` to 10.17.0
    omit 8d534c048866 [SPARK-48487][INFRA] Update License & Notice according to the dependency changes
    omit 41b14605ebaf [SPARK-48496][CORE] Use static regex Pattern instances in JavaUtils.timeStringAs and JavaUtils.byteStringAs
    omit ced788a4a927 [SPARK-48494][BUILD] Update `airlift:aircompressor` to 0.27
    omit 5e8f73443d2b [SPARK-48489][SQL] Throw a better, user-facing error when reading illegal schema from text Data Source
    omit 26b7aca162c9 [SPARK-45891] Describe shredding scheme for Variant
    omit e208b77f8d79 [SPARK-48374][PYTHON] Support additional PyArrow Table column types
    omit bc187013da82 [SPARK-48220][PYTHON] Allow passing PyArrow Table to createDataFrame()
    omit 8cf3195aed8a [SPARK-48392][CORE][FOLLOWUP] Add `--load-spark-defaults` flag to decide whether to load `spark-defaults.conf`
    omit 1cecdc7596e0 [SPARK-48481][SQL][SS] Do not apply OptimizeOneRowPlan against streaming Dataset
    omit 114164bd3f6b [SPARK-48490][CORE] Unescapes any literals for message of MessageWithContext
    omit 3cd35f8cb646 [SPARK-48391][CORE] Using addAll instead of add function in fromAccumulatorInfos method of TaskMetrics Class
    omit 96365c86962b [SPARK-48465][SQL] Avoid no-op empty relation propagation
    omit 844821c82da5 [SPARK-47578][R] Migrate RPackageUtils with variables to structured logging framework
    omit f083e61925e9 [SPARK-48430][SQL] Fix map value extraction when map contains collated strings
    omit 747437c80aa8 [SPARK-48476][SQL] fix NPE error message for null delmiter csv
    omit 67d11b1992aa [SPARK-48484][SQL] Fix: V2Write use the same TaskAttemptId for different task attempts
    omit 4360ec733d24 [SPARK-48172][SQL][FOLLOWUP] Fix escaping issues in JDBCDialects
    omit 80addbbddab7 [SPARK-48474][CORE] Fix the class name of the log in `SparkSubmitArguments` & `SparkSubmit`
    omit f64646c96f37 [SPARK-48467][BUILD] Upgrade Maven to 3.9.7
    omit c00742bbb651 [SPARK-47716][SQL] Avoid view name conflict in SQLQueryTestSuite semantic sort test case
    omit 3e27543128c8 [SPARK-48419][SQL] Foldable propagation replace foldable column shoul…
    omit ce7a889ad9fa [SPARK-48461][SQL] Replace NullPointerExceptions with error class in AssertNotNull expression
    omit 9e35b0067756 [SPARK-48446][SS][DOCS] Update SS doc of dropDuplicates to use the right syntax
    omit bb1f026da0e3 [SPARK-48475][PYTHON] Optimize _get_jvm_function in PySpark
    omit 6b4f97e1411c [SPARK-48468] Add LogicalQueryStage interface in catalyst
    omit e1f5a7c856ab [SPARK-48477][SQL][TESTS] Use withSQLConf in tests: Refactor CollationSuite, CoalesceShufflePartitionsSuite, SQLExecutionSuite
    omit a7da9b6b8aed [SPARK-48008][1/2] Support UDAFs in Spark Connect
    omit f68d761c9b21 [SPARK-48292][CORE] Revert [SPARK-39195][SQL] Spark OutputCommitCoordinator should abort stage when committed file not consistent with task status
    omit 69afd4be9c93 [SPARK-47361][SQL] Derby: Calculate suitable precision and scale for DECIMAL type
    omit df15c8d7744b [SPARK-48471][CORE] Improve documentation and usage guide for history server
    omit b477ef4fa992 [SPARK-47260][SQL] Assign name to error class _LEGACY_ERROR_TEMP_3250
    omit 910c3733bfdd Revert "[SPARK-48415][PYTHON] Refactor TypeName to support parameterized datatypes"
    omit 772d7d3aba17 [SPARK-47579][CORE][PART4] Migrate logInfo with variables to structured logging framework
    omit 51d4f64077b2 [SPARK-48464][SQL][TESTS] Use withSQLConf in tests: Refactor SQLConfSuite and StatisticsSuite
    omit 1f83f4d89eb7 [MINOR][DOCS] Fix usage doc for mod
    omit 47c55f4c9a44 [SPARK-48454][PS] Directly use the parent dataframe class
    omit 935f092b9d24 [SPARK-48442][PYTHON][TESTS] Add parenthesis to awaitTermination
    omit a3b8420e5eec [SPARK-48431][SQL] Do not forward predicates on collated columns to file readers
    omit cf47293b5fc7 [SPARK-48273][SQL][FOLLOWUP] Explicitly create non-Hive table in identifier-clause.sql
    omit dc6b493dd1f4 [SPARK-48462][SQL][TESTS] Use withSQLConf in tests: Refactor HiveQuerySuite and HiveTableScanSuite
    omit 0461745f1616 [SPARK-48281][SQL] Alter string search logic for UTF8_BINARY_LCASE collation (StringInStr, SubstringIndex)
    omit 49204b10af58 [SPARK-48444][SQL][TESTS] Use withSQLConf in tests: Refactor SQLQuerySuite
    omit ed436139743a [MINOR][CORE][TESTS] Make SparkLoggerSuiteBase code more concise
    omit f5d9b8098815 [MINOR][PS] Fallback code clean up
    omit cfbed998530e Revert "[SPARK-48322][SPARK-42965][SQL][CONNECT][PYTHON] Drop internal metadata in `DataFrame.schema`"
    omit 8bbbde7cb3c3 [SPARK-48426][SQL][DOCS] Add documentation for SQL operator precedence
    omit eac413afa0e8 [SPARK-48392][CORE] Also load `spark-defaults.conf` when provided `--properties-file`
    omit e6236af3d08c [SPARK-48000][SQL] Enable hash join support for all collations (StringType)
    omit a86bca131028 [SPARK-48445][SQL] Don't inline UDFs with expensive children
    omit 56c65d58729d [SPARK-23015][WINDOWS] Fix bug in Windows where starting multiple Spark instances within the same second causes a failure
    omit 99ffcaa13b36 [SPARK-48436][SQL][TESTS] Use `c.m.c.j.Driver` instead of `c.m.j.Driver` in `MySQLNamespaceSuite`
    omit 279f5bece4cb [SPARK-48322][SPARK-42965][SQL][CONNECT][PYTHON] Drop internal metadata in `DataFrame.schema`
    omit e2a588c91adc [SPARK-48438][PS][CONNECT] Directly use the parent column class
    omit 249390017ef4 [SPARK-48221][SQL] Alter string search logic for UTF8_BINARY_LCASE collation (Contains, StartsWith, EndsWith, StringLocate)
    omit e9a3ed857954 [SPARK-48159][SQL] Extending support for collated strings on datetime expressions
    omit 731a2cfcffae [SPARK-48273][SQL] Fix late rewrite of PlanWithUnresolvedIdentifier
    omit 7fe1b93884aa [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers
    omit a78ef738af02 [SPARK-48168][SQL][FOLLOWUP] Match expression strings of shift operators & functions with user inputs
    omit f164e4ae53ca [SPARK-48425][INFRA][FOLLOWUP] Do not copy the base spark folder
    omit df5ef91f44ba [SPARK-48425][INFRA][FOLLOWUP] copy spark connect release tarballs
    omit af1ac1edc2a9 [SPARK-41049][SQL][FOLLOW-UP] Mark map related expressions as stateful expressions
    omit a88cc1ad9319 [SPARK-48420][BUILD] Upgrade netty to `4.1.110.Final`
    omit d863503e8737 [SPARK-48434][PYTHON][CONNECT] Make `printSchema` use the cached schema
    omit fc1435d14d09 [SPARK-48415][PYTHON] Refactor `TypeName` to support parameterized datatypes
    omit de8d96892f92 [SPARK-48370][CONNECT][FOLLOW-UP] Use JDK's Cleaner instead
    omit cc3bf36c9f22 [SPARK-48432][SQL] Avoid unboxing integers in UnivocityParser
    omit b52645652eff [SPARK-48168][SQL][FOLLOWUP] Fix bitwise shifting operator's precedence
    omit 48a4bdb9eacb [SPARK-48427][BUILD] Upgrade `scala-parser-combinators` to 2.4
    omit 49da3a43f7ab [SPARK-48340][PYTHON][FOLLOWUP] Support TimestampNTZ infer schema miss prefer_timestamp_ntz
    omit 90c163753426 [MINOR][INFRA] Make pure Python build compatible with other setuptools versions
    omit 8f678a402e90 [SPARK-48425][PYTHON][BUILD] Replaces pyspark-connect to pyspark_connect for its output name
    omit 4ef5ec92ef70 [SPARK-48424][INFRA] Make dev/is-changed.py to return true it it fails
    omit 541158fe0352 [SPARK-47579][CORE][PART3][FOLLOWUP] Fix KubernetesSuite
    omit 468aa842c643 [SPARK-48320][CORE][DOCS] Add structured logging guide to the scala and java doc
    omit 416d7f24fc35 [SPARK-48239][INFRA][FOLLOWUP] install the missing `jq` library
    omit cff7014b337b [SPARK-47579][CORE][PART3] Spark core: Migrate logInfo with variables to structured logging framework
    omit 6cd1ccc56321 [SPARK-48394][CORE] Cleanup mapIdToMapIndex on mapoutput unregister
    omit 1a536f01ead3 [SPARK-48407][SQL][DOCS] Teradata: Document Type Conversion rules between Spark SQL and teradata
    omit 7d96334902f2 [SPARK-48325][CORE] Always specify messages in ExecutorRunner.killProcess
    omit 3cb30c2366b2 [SPARK-47579][SQL][FOLLOWUP] Restore the `--help` print format of spark sql shell
    omit 7ae939ae12a6 [SPARK-48168][SQL] Add bitwise shifting operators support
    omit bd95040c3170 [SPARK-48412][PYTHON] Refactor data type json parse
    omit b15b6cf1f537 [SPARK-48409][BUILD][TESTS] Upgrade MySQL & Postgres & Mariadb docker image version
    omit ef43bbbc1163 [SPARK-48384][BUILD] Exclude `io.netty:netty-tcnative-boringssl-static` from `zookeeper`
    omit 3346afd4b250 [SPARK-46090][SQL][FOLLOWUP] Add DeveloperApi import
    omit a29c9653f3d4 [SPARK-46090][SQL] Support plan fragment level SQL configs in AQE
    omit f42ed6c76004 [SPARK-48406][BUILD] Upgrade commons-cli to 1.8.0
    omit 3b9b52dff614 [SPARK-48405][BUILD] Upgrade `commons-compress` to 1.26.2
    omit 6afa6cc3c16e [SPARK-48399][SQL] Teradata: ByteType should map to BYTEINT instead of BYTE(binary)
    omit 2516fd8439df [SPARK-45009][SQL][FOLLOW UP] Add error class and tests for decorrelation of predicate subqueries in join condition which reference both join child
    omit 4a471cceebed [MINOR][TESTS] Add a helper function for `spark.table` in dsl
    omit 14d3f447360b [SPARK-48395][PYTHON] Fix `StructType.treeString` for parameterized types
    omit e8f58a9c4a64 [SPARK-48370][SPARK-48258][CONNECT][PYTHON][FOLLOW-UP] Refactor local and eager required fields in CheckpointCommand
    omit a393d6cdf00c [SPARK-48370][CONNECT] Checkpoint and localCheckpoint in Scala Spark Connect client
    omit 5df9a0866ae6 [SPARK-48386][TESTS] Replace JVM assert with JUnit Assert in tests
    omit a48365dd98c9 [SPARK-48387][SQL] Postgres: Map TimestampType to TIMESTAMP WITH TIME ZONE
    omit d96ab44c8251 [SPARK-48393][PYTHON] Move a group of constants to `pyspark.util`
    omit 06aafb1eeaf3 [SPARK-48258][PYTHON][CONNECT][FOLLOW-UP] Bind relation ID to the plan instead of DataFrame
    omit 6be3560f3c89 [SPARK-48364][SQL] Add AbstractMapType type casting and fix RaiseError parameter map to work with collated strings
    omit e04d3d7c430a [SPARK-48215][SQL] Extending support for collated strings on date_format expression
    omit 9fd85d9acc5a Revert "[SPARK-48379][INFRA] Cancel previous builds on a PR when new commit is pushed"
    omit 1188887c31c1 [SPARK-48389][INFRA] Remove obsolete workflow cancel_duplicate_workflow_runs
    omit 93d83c347042 [SPARK-48379][INFRA] Cancel previous builds on a PR when new commit is pushed
    omit 366a052fe106 [SPARK-47920][DOCS][SS][PYTHON] Add doc for python streaming data source API
    omit e55875b0bbe0 [SPARK-48372][SPARK-45716][PYTHON] Implement `StructType.treeString`
    omit bf7f664296c5 [SPARK-47515][SPARK-47406][SQL][FOLLOWUP] Add legacy config spark.sql.legacy.mysql.timestampNTZMapping.enabled
    omit f4958ba9587c [SPARK-48367][CONNECT][FOLLOWUP] Replace keywords that identify `lint-scala` detection results
    omit a886121aee45 [MINOR][TESTS] Fix `DslLogicalPlan.as`
    omit e702b32656bc [SPARK-48314][SS] Don't double cache files for FileStreamSource using Trigger.AvailableNow
    omit a1e27a38868b [SPARK-48341][CONNECT] Allow plugins to use QueryTest in their tests
    omit 80c0f1165417 [SPARK-48381][K8S][DOCS] Update `YuniKorn` docs with v1.5.1
    omit febdbf56fb22 [SPARK-48031] Grandfather legacy views to SCHEMA BINDING
    omit 6b3a88195e30 [SPARK-48329][SQL] Enable `spark.sql.sources.v2.bucketing.pushPartValues.enabled` by default
    omit df50d4b309b5 [SPARK-48336][PS][CONNECT] Implement `ps.sql` in Spark Connect
    omit 6e6e7a00f662 [SPARK-48369][SQL][PYTHON][CONNECT] Add function `timestamp_add`
    omit 664c8c19dae7 [SPARK-48365][DOCS] DB2: Document Mapping Spark SQL Data Types to DB2
    omit 255e7408d95b [SPARK-48367][CONNECT] Fix lint-scala for scalafmt to detect files to format properly
    omit 6213fa661ffe [SPARK-48300][SQL] Codegen Support for `from_xml`
    omit 0e0134a9a48d [MINOR][DOCS] correct the doc error in configuration page (fix rest to reset)
    omit 87c345d8d4ea [MINOR][TESTS] Rename test_union to test_eqnullsafe at ColumnTestsMixin
    omit 2a1bdc3eda8a [SPARK-48337][SQL] Fix precision loss for JDBC TIME values
    omit 839efb1d72f6 [SPARK-48363][SQL] Cleanup some redundant codes in `from_xml`
    omit 0a2eb7c8f7bb [SPARK-48340][PYTHON] Support TimestampNTZ infer schema miss prefer_timestamp_ntz
    omit 7d6bb7474faa [SPARK-48258][PYTHON][CONNECT] Checkpoint and localCheckpoint in Spark Connect
    omit 0393ab432389 [SPARK-48330][SS][PYTHON] Fix the python streaming data source timeout issue for large trigger interval
    omit f5ffb74f170e [SPARK-48328][BUILD] Upgrade `Arrow` to 16.1.0
    omit 4fc2910f92d1 [SPARK-48238][BUILD][YARN] Replace YARN AmIpFilter with a forked implementation
    omit 6fcdaab27ae9 [SPARK-48332][BUILD][TESTS] Upgrade `jdbc` related test dependencies
    omit 3a888f725315 [SPARK-48323][SQL] DB2: Map BooleanType to BOOLEAN instead of CHAR(1)
    omit 2eea28b5efd4 [SPARK-48242][BUILD] Upgrade extra-enforcer-rules to 1.8.0
    omit 617ac1aec748 [SPARK-48031] Decompose viewSchemaMode config, add SHOW CREATE TABLE support
    omit 6a17b794338b [SPARK-48305][SQL] Add collation support for CurrentLike expressions
    omit d2edefac59a7 [MINOR][PYTHON][TESTS] Test `test_mixed_udf_and_sql` with parent `Colum` class
    omit fa8aa571ad18 [SPARK-48335][PYTHON][CONNECT] Make `_parse_datatype_string` compatible with Spark Connect
    omit 52ca921113b4 [MINOR][PYTHON][TESTS] Remove unnecessary hack imports
    omit 88eb9ebe8524 [SPARK-48333][PYTHON][CONNECT][TESTS] Test `test_sorting_functions_with_column` with same `Column`
    omit 74b42fd1ce7d [SPARK-44924][SS] Add config for FileStreamSource cached files
    omit 6f6b4860268d [SPARK-48175][SQL][PYTHON] Store collation information in metadata and not in type for SER/DE
    omit 51623785c38c [SPARK-47818][CONNECT][FOLLOW-UP] Introduce plan cache in SparkConnectPlanner to improve performance of Analyze requests
    omit 3edd6c7e1d50 [SPARK-48312][SQL] Improve Alias.removeNonInheritableMetadata performance
    omit 15fb4787354a [SPARK-48321][CONNECT][TESTS] Avoid using deprecated methods in dsl
    omit 1ea156169ce8 [SPARK-48319][PYTHON][CONNECT][TESTS] Test `assert_true` and `raise_error` with the same error class as Spark Classic
    omit 5643cfb71d34 [SPARK-48303][CORE] Reorganize `LogKeys`
    omit e07f1af03edf [SPARK-48317][PYTHON][CONNECT][TESTS] Enable `test_udtf_with_analyze_using_archive` and `test_udtf_with_analyze_using_file`
    omit 74a1a76e811a [MINOR][PYTHON][TESTS] Call `test_apply_schema_to_dict_and_rows` in `test_apply_schema_to_row`
    omit 889820c1ff39 [SPARK-41625][PYTHON][CONNECT][TESTS][FOLLOW-UP] Enable `DataFrameObservationParityTests.test_observe_str`
    omit 714fc8cd872d [SPARK-48316][PS][CONNECT][TESTS] Fix comments for SparkFrameMethodsParityTests.test_coalesce and test_repartition
    omit 403619a3974c [SPARK-48306][SQL] Improve UDT in error message
    omit b0e535217bf8 [SPARK-48301][SQL][FOLLOWUP] Update the error message
    omit 05e1706e5aa6 [SPARK-48310][PYTHON][CONNECT] Cached properties must return copies
    omit 153053fe6c3d [SPARK-48268][CORE] Add a configuration for SparkContext.setCheckpointDir
    omit e9d4152a319a [SPARK-48031][SQL][FOLLOW-UP] Use ANSI-enabled cast in view lookup test
    omit 59f88c372522 [SPARK-48294][SQL] Handle lowercase in nestedTypeMissingElementTypeError
    omit 283b2ff42221 [SPARK-48291][CORE][FOLLOWUP] Rename Java *LoggerSuite* as *SparkLoggerSuite*
    omit 57948c865e06 [SPARK-48308][CORE] Unify getting data schema without partition columns in FileSourceStrategy
    omit 3d3d18f14ba2 [SPARK-48301][SQL] Rename `CREATE_FUNC_WITH_IF_NOT_EXISTS_AND_REPLACE` to `CREATE_ROUTINE_WITH_IF_NOT_EXISTS_AND_REPLACE`
    omit 4be0828e6e6a [SPARK-48288] Add source data type for connector cast expression
    omit fa83d0f8fce7 [SPARK-48296][SQL] Codegen Support for `to_xml`
    omit 3bd845ea930a [SPARK-48297][SQL] Fix a regression TRANSFORM clause with char/varchar
    omit b53d78e94f6e [SPARK-48036][DOCS][FOLLOWUP] Update sql-ref-ansi-compliance.md
    omit 0ba8ddc9ce5b [SPARK-48293][SS] Add test for when ForeachBatchUserFuncException wraps interrupted exception due to query stop
    omit 97717363abae [SPARK-48264][BUILD] Upgrade `datasketches-java` to 6.0.0
    omit 9130f78fb12e [SPARK-47607] Add documentation for Structured logging framework
    omit 726f2c95d4dc [SPARK-48299][BUILD] Upgrade `scala-maven-plugin` to 4.9.1
    omit dec910ba3c36 [SPARK-48214][INFRA] Ban import `org.slf4j.Logger` & `org.slf4j.LoggerFactory`
    omit d0f4533f4e79 [SPARK-48287][PS][CONNECT] Apply the builtin `timestamp_diff` method
    omit 5e8322150a05 [SPARK-48219][CORE] StreamReader Charset fix with UTF8
    omit 9f64219ae325 [SPARK-48295][PS] Turn on `compute.ops_on_diff_frames` by default
    omit 4ffaa2e89a8a [SPARK-48289][DOCKER][TEST] Clean up Oracle JDBC tests by skipping redundant SYSTEM password reset
    omit ca3593288d57 [SPARK-48252][SQL] Update CommonExpressionRef when necessary
    omit a252cbd5ca13 [SPARK-48291][CORE] Rename Java Logger as SparkLogger
    omit a2d93d104a6c [SPARK-48256][BUILD] Add a rule to check file headers for the java side, and fix inconsistent files
    omit 6767053dacd9 [SPARK-48218][CORE] TransportClientFactory.createClient may NPE cause FetchFailedException
    omit 12820e11b094 [SPARK-48049][BUILD] Upgrade Scala to 2.13.14
    omit 973328cd376b [SPARK-48285][SQL][DOCS] Update docs for size function and sizeOfNull configuration
    omit 9e386b472981 [SPARK-48172][SQL] Fix escaping issues in JDBCDialects
    omit d0385c4a99c1 [SPARK-46759][AVRO][FOLLOWUP] Fix configuration name for spark.sql.avro.xz.level
    omit ad5fcae0b0ed [SPARK-48271][SQL] Turn match error in RowEncoder into UNSUPPORTED_DATA_TYPE_FOR_ENCODER
    omit 76329f9bb60e [SPARK-48279][BUILD] Upgrade ORC to 2.0.1
    omit e97ad0a44419 [SPARK-48278][PYTHON][CONNECT] Refine the string representation of `Cast`
    omit c03ebb467ec2 [SPARK-48272][SQL][PYTHON][CONNECT] Add function `timestamp_diff`
    omit da78949eee04 [SPARK-48269][DOCS][TESTS] DB2: Document Mapping Spark SQL Data Types from DB2 and add tests
    omit 5e87e9fbd6e6 [SPARK-48277] Improve error message for ErrorClassesJsonReader.getErrorMessage
    omit 8c0a7ba82c98 [SPARK-48160][SQL] Add collation support for XPATH expressions
    omit 723354039f1d [SPARK-48162][SQL] Add collation support for MISC expressions
    omit 7ec37e47d13e [SPARK-48233][SS][TESTS] Tests for streaming on columns with non-default collations
    omit b7cdff929eb9 [SPARK-48100][SQL] Fix issues in skipping nested structure fields not selected in schema
    omit d31161b27404 [SPARK-48276][PYTHON][CONNECT] Add the missing `__repr__` method for `SQLExpression`
    omit 7441bd0197e6 [SPARK-48274][BUILD] Upgrade GenJavadoc to `0.19`
    omit 4ff5ca81cffb Revert "[SPARK-48172][SQL] Fix escaping issues in JDBC Dialects"
    omit 3ae78c4c39a7 [SPARK-47599][MLLIB] MLLib: Migrate logWarn with variables to structured logging framework
    omit 42c1c8f7d264 [SPARK-48247][PYTHON] Use all values in a dict when inferring MapType schema
    omit 79aeae1a9aaa [SPARK-47579][CORE][PART2] Migrate logInfo with variables to structured logging framework
    omit 91da2caa409c [SPARK-48263] Collate function support for non UTF8_BINARY strings
    omit 97bf1ee9f6f7 [SPARK-47301][SQL][TESTS][FOLLOWUP] Remove workaround for ParquetIOSuite
    omit 47006a493f98 [SPARK-48172][SQL] Fix escaping issues in JDBC Dialects
    omit e5ad5e94a8c8 [SPARK-48155][SQL] AQEPropagateEmptyRelation for join should check if remain child is just BroadcastQueryStageExec
    omit 6766c39b458a [SPARK-46707][SQL][FOLLOWUP] Push down throwable predicate through aggregates
    omit c0982621f46b [SPARK-48267][SS] Regression e2e test with SPARK-47305
    omit e6c914f63079 [SPARK-48157][SQL] Add collation support for CSV expressions
    omit 9241b8e8c0df [SPARK-48229][SQL] Add collation support for inputFile expressions
    omit 7974811218c9 [SPARK-48265][SQL] Infer window group limit batch should do constant folding
    omit 0ea808880e22 [SPARK-48027][SQL][FOLLOWUP] Add comments for the other code branch
    omit e7a1efda2221 [SPARK-48266][CONNECT] Move package object `org.apache.spark.sql.connect.dsl` to test directory
    omit 78d2a86a927f [SPARK-48209][CORE] Common (java side): Migrate `error/warn/info` with variables to structured logging framework
    omit 4cc589afe5b5 [SPARK-41794][FOLLOWUP] Add `try_remainder` to python API references
    omit 28cf3db77932 [SPARK-48259][CONNECT][TESTS] Add 3 missing methods in dsl
    omit d9ff78e2e341 [SPARK-48260][SQL] Disable output committer coordination in one test of ParquetIOSuite
    omit a101c48dd965 [SPARK-44953][CORE] Log a warning when shuffle tracking is enabled along side another DA supported mechanism
    omit 8d8cc623085e [SPARK-41794][SQL] Add `try_remainder` function and re-enable column tests
    omit b14abb3a2ed0 [SPARK-48241][SQL] CSV parsing failure with char/varchar type columns
    omit 3456d4f69a86 [SPARK-47681][FOLLOWUP] Fix schema_of_variant(decimal)
    omit 42f2132d1fc9 [SPARK-48206][SQL][TESTS] Add tests for window rewrites with RewriteWithExpression
    omit 27048702830d [SPARK-48250][PYTHON][CONNECT][TESTS] Enable array inference tests at test_parity_types.py
    omit 722fac7cd54e [SPARK-48031][SQL] Support view schema evolution
    omit c4c9ccbdf562 [SPARK-48257][BUILD] Polish POM for Hive dependencies
    omit 224fd35d6fca Revert "[SPARK-48250][PYTHON][CONNECT][TESTS] Enable array inference tests at test_parity_types.py"
    omit 92b708b6b767 [SPARK-48254][BUILD] Enhance Guava version extraction rule in `dev/test-dependencies.sh`
    omit 2554e4716f16 [SPARK-48227][PYTHON][DOC][FOLLOWUP] Fix two typos
    omit b2140d0f25d8 [SPARK-48248][PYTHON] Fix nested array to respect legacy conf of inferArrayTypeFromFirstElement
    omit 13b0d1aab367 [SPARK-48250][PYTHON][CONNECT][TESTS] Enable array inference tests at test_parity_types.py
    omit acc37531deb9 [SPARK-47993][PYTHON][FOLLOW-UP] Update migration guide about Python 3.8 dropped
    omit cae2248bc13d [MINOR][PYTHON][TESTS] Move test `test_named_arguments_negative` to `test_arrow_python_udf`
    omit b5584221cfc2 [SPARK-48245][SQL] Fix typo in BadRecordException class doc
    omit 5b965f70c057 [SPARK-48239][INFRA] Update the release docker image to follow what we use in Github Action jobs
    omit 57b207774382 [SPARK-48240][DOCS] Replace `Local[..]` with `"Local[...]"` in the docs
    omit f699f556d8a0 [SPARK-48237][BUILD] Clean up `dev/pr-deps` at the end of `test-dependencies.sh` script
    omit d82458f15539 [SPARK-48205][SQL][FOLLOWUP] Add missing tags for the dataSource API
    omit 5b3b8a90638c [SPARK-48236][BUILD] Add `commons-lang:commons-lang:2.6` back to support legacy Hive UDF jars
    omit 726ef8aa66ea Revert "[SPARK-48230][BUILD] Remove unused `jodd-core`"
    omit 2225aa1dab0f [SPARK-48144][SQL] Fix `canPlanAsBroadcastHashJoin` to respect shuffle join hints
    omit a6632ffa16f6 [SPARK-48143][SQL] Use lightweight exceptions for control-flow between UnivocityParser and FailureSafeParser
    omit 5beaf85cd5ef [SPARK-47793][TEST][FOLLOWUP] Fix flaky test for Python data source exactly once
    omit c5b6ec734bd0 [SPARK-47441][YARN] Do not add log link for unmanaged AM in Spark UI
    omit 73bb619d45b2 [SPARK-48235][SQL] Directly pass join instead of all arguments to getBroadcastBuildSide and getShuffleHashJoinBuildSide
    omit 7ef0440ef221 [SPARK-48146][SQL] Fix aggregate function in With expression child assertion
    omit 259760a5c5e2 [SPARK-48228][PYTHON][CONNECT][FOLLOWUP] Also apply `_validate_pandas_udf` in MapInXXX
    omit 256a23883d90 [SPARK-48232][PYTHON][TESTS] Fix 'pyspark.sql.tests.connect.test_connect_session' in Python 3.12 build
    omit d8151186d794 [SPARK-48230][BUILD] Remove unused `jodd-core`
    omit 2df494fd4e4e [SPARK-48158][SQL] Add collation support for XML expressions
    omit 33cac4436e59 [SPARK-47847][CORE] Deprecate `spark.network.remoteReadNioBufferConversion`
    omit 8ccc8b92be50 [SPARK-48201][DOCS][PYTHON] Make some corrections in the docstring of pyspark DataStreamReader methods
    omit 9bb15db85e53 [SPARK-48228][PYTHON][CONNECT] Implement the missing function validation in ApplyInXXX
    omit b371e7dd8800 [SPARK-48224][SQL] Disallow map keys from being of variant type
    omit 2d609bfd37ae [SPARK-47018][BUILD][SQL] Bump built-in Hive to 2.3.10
    omit 1138b2a68b54 [MINOR][BUILD] Remove duplicate configuration of maven-compiler-plugin
    omit 32b2827b964b [SPARK-47834][SQL][CONNECT] Mark deprecated functions with `@deprecated` in `SQLImplicits`
    omit a41d0ae79b43 [SPARK-48176][SQL] Adjust name of FIELD_ALREADY_EXISTS error condition
    omit 9a2818820f11 [SPARK-48222][INFRA][DOCS] Sync Ruby Bundler to 2.4.22 and refresh Gem lock file
    omit 012d19d8e9b2 [SPARK-48227][PYTHON][DOC] Document the requirement of seed in protos
    omit 71f0eda71bc1 [SPARK-48180][SQL] Improve error when UDTF call with TABLE arg forgets parentheses around multiple PARTITION/ORDER BY exprs
    omit e704b9e56b0c [SPARK-48226][BUILD] Add `spark-ganglia-lgpl` to `lint-java` & `spark-ganglia-lgpl` and `jvm-profiler` to `sbt-checkstyle`
    omit b47d7853d92f [SPARK-48148][CORE] JSON objects should not be modified when read as STRING
    omit e1fb1d7e063a [SPARK-48216][TESTS] Remove overrides DockerJDBCIntegrationSuite.connectionTimeout to make related tests configurable
    omit 21333f8c1fc0 [SPARK-47409][SQL] Add support for collation for StringTrim type of functions/expressions (for UTF8_BINARY & LCASE)
    omit 3fd38d4c07f6 [SPARK-47803][FOLLOWUP] Check nulls when casting nested type to variant
    omit 207d675110e6 [SPARK-48211][SQL] DB2: Read SMALLINT as ShortType
    omit 9e62dbad6a75 [SPARK-48212][PYTHON][CONNECT][TESTS] Fully enable `PandasUDFParityTests.test_udf_wrong_arg`
    omit ecca1bf6453e [SPARK-47365][PYTHON] Add toArrow() DataFrame method to PySpark
    omit 027327d94b34 [SPARK-47986][CONNECT][PYTHON] Unable to create a new session when the default session is closed by the server
    omit 34ee0d8414b2 [SPARK-47421][SQL] Add collation support for URL expressions
    omit 045ec6a166c8 [SPARK-48208][SS] Skip providing memory usage metrics from RocksDB if bounded memory usage is enabled
    omit 91da4ac25148 [SPARK-47354][SQL] Add collation support for variant expressions
    omit a4ab82b8f340 [SPARK-48186][SQL] Add support for AbstractMapType
    omit 6cc3dc2ef4d2 [SPARK-48169][SPARK-48143][SQL] Revert BadRecordException optimizations
    omit 85a6e35d834e [SPARK-48182][SQL] SQL (java side): Migrate `error/warn/info` with variables to structured logging framework
    omit 5891b20ef492 [SPARK-47186][TESTS][FOLLOWUP] Correct the name of spark.test.docker.connectionTimeout
    omit 7e79e91dc8c5 [SPARK-48197][SQL] Avoid assert error for invalid lambda function
    omit 337f980f0073 [SPARK-48204][INFRA] Fix release script for Spark 4.0+
    omit 4fb6624bd2ce [SPARK-48205][PYTHON] Remove the private[sql] modifier for Python data sources
    omit 21548a8cc5c5 [SPARK-47545][CONNECT] Dataset `observe` support for the Scala client
    omit fbfcd402851e [SPARK-48203][INFRA] Spin off `pyspark` tests from `build_branch34.yml` Daily CI
    omit 70e5d2aa7a99 [SPARK-48202][INFRA] Spin off `pyspark` tests from `build_branch35.yml` Daily CI
    omit 9d79ab42b127 [SPARK-48200][INFRA] Split `build_python.yml` into per-version cron jobs
    omit e0c406eaef36 [SPARK-48198][BUILD] Upgrade jackson to 2.17.1
    omit 47afe77242ab [SPARK-48184][PYTHON][CONNECT] Always set the seed of `Dataframe.sample` in Client side
    omit 8d7081639ab4 [SPARK-48161][SQL] Add collation support for JSON expressions
    omit 8950add773e6 [SPARK-48188][SQL] Consistently use normalized plan for cache
    omit 003823b39d35 [SPARK-48191][SQL] Support UTF-32 for string encode and decode
    omit d7f69e7003a3 [SPARK-48190][PYTHON][PS][TESTS] Introduce a helper function to drop metadata
    omit bd896cac168a Revert "[SPARK-48163][CONNECT][TESTS] Disable `SparkConnectServiceSuite.SPARK-43923: commands send events - get_resources_command`"
    omit 1b966d2eb329 [SPARK-47965][SQL][FOLLOW-UP] Uses `null` as its default value for `OptionalConfigEntry`
    omit fe3ef20d6418 [SPARK-48193][INFRA] Make `maven-deploy-plugin` retry 3 times
    omit fe8b18b776f5 [SPARK-48185][SQL] Fix 'symbolic reference class is not accessible: class sun.util.calendar.ZoneInfo'
    omit f5401bab23c0 [MINOR][INFRA] Rename builds to have consistent names
    omit f693abc8de94 [SPARK-48192][INFRA] Enable TPC-DS tests in forked repository
    omit f3d9b819f3c0 [SPARK-48187][INFRA] Run `docs` only in PR builders and `build_non_ansi` Daily CI
    omit 52a7f634e913 [SPARK-48183][PYTHON][DOCS] Update error contribution guide to respect new error class file
    omit 5f883117203d [SPARK-47914][SQL] Do not display the splits parameter in Range
    omit 3b1ea0fde44e [MINOR][PYTHON][TESTS] Remove the doc in error message tests to allow other PyArrow versions in tests
    omit 6588554aa4cc [SPARK-48149][INFRA][FOLLOWUP] Use single quotation mark
    omit 84c5b919d998 [SPARK-48131][CORE][FOLLOWUP] Add a new configuration for the MDC key of Task Name
    omit 6bbf6b1eff2c [SPARK-48126][CORE] Make `spark.log.structuredLogging.enabled` effective
    omit 67ae23934b56 [SPARK-48045][PYTHON] Pandas API groupby with multi-agg-relabel ignores as_index=False
    omit a15adeb3a215 [SPARK-47240][CORE][PART1] Migrate logInfo with variables to structured logging framework
    omit 553e1b85c42a [SPARK-48152][BUILD] Make `spark-profiler` as a part of release and publish to maven central repo
    omit 5e49665ac39b [SPARK-47960][SS] Allow chaining other stateful operators after transformWithState operator
    omit 3d9d1f3dc05a [SPARK-48134][CORE] Spark core (java side): Migrate `error/warn/info` with variables to structured logging framework
    omit e24f8965e066 [SPARK-48037][CORE] Fix SortShuffleWriter lacks shuffle write related metrics resulting in potentially inaccurate data
    omit 26c50369edb2 [SPARK-48174][INFRA] Merge `connect` back to the original test pipeline
    omit 808186835077 [SPARK-48035][SQL][FOLLOWUP] Fix try_add/try_multiply being semantic equal to add/multiply
    omit 8f719adcf556 [SPARK-41547][CONNECT][TESTS] Re-eneable Spark Connect function tests with ANSI mode
    omit a3eebcf39687 [SPARK-48170][PYTHON][CONNECT][TESTS] Enable `ArrowPythonUDFParityTests.test_err_return_type`
    omit 925457cadd22 [SPARK-48169][SQL] Use lazy BadRecordException cause in all parsers and remove the old constructor, which was meant for the migration
    omit 9e0a87eb4cf2 [SPARK-48165][BUILD] Update `ap-loader` to 3.0-9
    omit 493493d6c5bb [SPARK-48173][SQL] CheckAnalysis should see the entire query plan
    omit 148f5335427c [SPARK-47297][SQL] Add collation support for format expressions
    omit c326f3c143ff [SPARK-48171][CORE] Clean up the use of deprecated constructors of `o.rocksdb.Logger`
    omit 326dbb447873 [SPARK-48143][SQL] Use lightweight exceptions for control-flow between UnivocityParser and FailureSafeParser
    omit 08c6bb9bf32f [SPARK-47267][SQL] Add collation support for hash expressions
    omit c4df12cc884c [SPARK-48113][CONNECT] Allow Plugins to integrate with Spark Connect
    omit 7f8ef96cea27 [SPARK-48166][SQL] Avoid using BadRecordException as user-facing error in VariantExpressionEvalUtils
    omit 98d4ab734de0 [SPARK-48058][SPARK-43727][PYTHON][CONNECT][TESTS][FOLLOWUP] Code clean up
    omit eee179135ed2 [SPARK-48090][SS][PYTHON][TESTS] Shorten the traceback in the test checking error message in UDF
    omit b5e39bedab14 [SPARK-48027][SQL] InjectRuntimeFilter for multi-level join should check child join type
    omit 56fe185c78a2 [SPARK-48163][CONNECT][TESTS] Disable `SparkConnectServiceSuite.SPARK-43923: commands send events - get_resources_command`
    omit 0f58b8e0c5eb [MINOR][CONNECT][TESTS] Improve test failure error message in StreamingParityTests
    omit 3b9f52f7768a [SPARK-48154][PYTHON][CONNECT][TESTS] Enable `PandasUDFGroupedAggParityTests.test_manual`
    omit 05b22ebb3060 [SPARK-48141][TEST] Update the Oracle docker image version used for test and integration to use Oracle Database 23ai Free
    omit 8cf602a3f587 [SPARK-48150][SQL] try_parse_json output should be declared as nullable
    omit 0862f69f7be8 [SPARK-48147][SS][CONNECT] Remove client side listeners when local Spark session is deleted
    omit 0907a15b2d15 [SPARK-48153][INFRA] Run `build` job of `build_and_test.yml` only if needed
    omit f918d1179642 [SPARK-48151][INFRA] `build_and_test.yml` should use `Volcano` 1.7.0 for `branch-3.4/3.5`
    omit 2ef7246b9c5b [SPARK-48142][PYTHON][CONNECT][TESTS] Enable `CogroupedApplyInPandasTests.test_wrong_args`
    omit 7290000d51d7 [SPARK-48035][SQL] Fix try_add/try_multiply being semantic equal to add/multiply
    omit c94fe37ba6d0 [MINOR][SQL][DOCS] Correct comments for UnresolvedRelation
    omit 9dc5599b01b1 [SPARK-48105][SS] Fix the race condition between state store unloading and snapshotting
    omit 13c7bdcf89c9 [SPARK-48112][CONNECT] Expose session in SparkConnectPlanner to plugins
    omit d67752a8f3d7 [SPARK-47681][SQL][FOLLOWUP] Fix variant decimal handling
    omit 4c6884291e8b [SPARK-48149][INFRA] Serialize `build_python.yml` to run a single Python version per cron schedule
    omit de8ba8589c21 [SPARK-48145][CORE] Remove logDebug and logTrace with MDC in JAVA structured logging framework
    omit 526e4141457d [SPARK-45220][FOLLOWUP][DOCS][TESTS] Make a `dataframe.join` doctest deterministic
    omit 7c728b2c2d6c [SPARK-48137][INFRA] Run `yarn` test only in PR builders and Daily CIs
    omit 8294c5962feb [SPARK-48138][CONNECT][TESTS] Disable a flaky `SparkSessionE2ESuite.interrupt tag` test
    omit 8094535973f1 [SPARK-48054][INFRA][FOLLOW-UP] Rename SPARK_SKIP_JVM_REQUIRED_TESTS to SPARK_SKIP_CONNECT_COMPAT_TESTS
    omit d09f174be5e9 [SPARK-48136][INFRA][CONNECT] Always upload Spark Connect log files in scheduled build for Spark Connect
    omit 4e69857195a6 Revert "[SPARK-47777][PYTHON][SS][TESTS] Add spark connect test for python streaming data source"
    omit 8b2251734519 [SPARK-48135][INFRA] Run `buf` and `ui` only in PR builders and Java 21 Daily CI
    omit 32ba5c1db62c [SPARK-48133][INFRA] Run `sparkr` only in PR builders and Daily CIs
    omit bf2e25459fe4 [SPARK-48019][SQL][FOLLOWUP] Use primitive arrays over object arrays when nulls exist
    omit a0f62393d69a [SPARK-48132][INFRA] Run `k8s-integration-tests` only in PR builder and Daily CIs
    omit b924e689942d [SPARK-48124][CORE] Disable structured logging for Connect-Repl by default
    omit 8443672b1ab1 [SPARK-48131][CORE] Unify MDC key `mdc.taskName` and `task_name`
    omit 9a45da21dd1c [SPARK-48129][PYTHON] Provide a constant table schema in PySpark for querying structured logs
    omit 356aca5af5b8 [SPARK-46009][SQL][FOLLOWUP] Remove unused golden file
    omit 96f65c950064 [SPARK-48128][SQL] For BitwiseCount / bit_count expression, fix codegen syntax error for boolean type inputs
    omit ca8c269a1503 [SPARK-48124][CORE] Disable structured logging for Interpreters by default
    omit 3b8c0049a5b5 [SPARK-48116][INFRA][FOLLOW-UP] Deduplicate pyspark.pandas skipping logic
    omit e4453b480f98 [SPARK-48123][CORE] Provide a constant table schema for querying structured logs
    omit 111529e0dea6 [SPARK-47578][CORE] Migrate logWarning with variables to structured logging framework
    omit 454fcbb277b9 [SPARK-48116][INFRA][FOLLOWUP] Fix `if` statement to check repository
    omit 2be447f89ea8 Revert "[SPARK-48116][INFRA][FOLLOWUP] Simplify the build with fixing the if condition"
    omit 2cb6ea721fe0 [SPARK-48116][INFRA][FOLLOWUP] Simplify the build with fixing the if condition
    omit c0ef52760de9 [SPARK-48127][INFRA] Fix `dev/scalastyle` to check `hadoop-cloud` and `jvm-profiler` modules
    omit 1904dee475d7 [SPARK-48116][INFRA] Run `pyspark-pandas*` only in PR builder and Daily Python CIs
    omit be998221fc93 [SPARK-48056][PYTHON][CONNECT][FOLLOW-UP] Use `assertEqual` instead of `assertEquals` for Python 3.12
    omit 5d1f976f85fe [SPARK-47969][PYTHON][TESTS][FOLLOWUP] Make Test `test_creation_index` deterministic
    omit 7f08df4af95d [SPARK-47097][CONNECT][TESTS][FOLLOWUP] Increase timeout to `1 minute` for `interrupt tag` test
    omit 5c01f196afc3 [SPARK-48059][CORE] Implement the structured log framework on the java side
    omit c3a462ce2966 [SPARK-48121][K8S] Promote `KubernetesDriverConf` to `DeveloperApi`
    omit b42d235c2930 [SPARK-48114][CORE] Precompile template regex to avoid unnecessary work
    omit 85902880d709 [SPARK-48119][K8S] Promote `KubernetesDriverSpec` to `DeveloperApi`
    omit d6ca2c5c3c4b [SPARK-48118][SQL] Support `SPARK_SQL_LEGACY_CREATE_HIVE_TABLE` env variable
    omit aa00b00c18e6 [SPARK-48115][INFRA] Remove `Python 3.11` from `build_python.yml`
    omit f450272a9aac [SPARK-43046][FOLLOWUP][SS][CONNECT] Remove not used line in deduplicateWithinWatermark
    omit cd789acb5e51 [SPARK-48088][PYTHON][CONNECT][TESTS] Prepare backward compatibility test 4.0 <> above
    omit aaf39955eb3d [MINOR][INFRA] Clean up the worker heatup that are unnecessary
    omit 2d346fbb9c5c [SPARK-48111][INFRA] Disable Docker integration test and TPC-DS in commit builder
    omit defda8663c05 [SPARK-48110][INFRA] Remove all Maven compilation build
    omit 8d70f4ba5396 [SPARK-48107][PYTHON] Exclude tests from Python distribution
    omit ddc1f6b2a466 [SPARK-48106][INFRA] Use `Python 3.11` in `pyspark` tests of `build_and_test.yml`
    omit f044748efeac [SPARK-48103][K8S] Promote `KubernetesDriverBuilder` to `DeveloperApi`
    omit 63837020ed29 [SPARK-48109][INFRA] Enable `k8s-integration-tests` only for `kubernetes` module change
    omit df633091b714 [SPARK-48054][PYTHON][CONNECT][INFRA] Backward compatibility test for Spark Connect
    omit 111df27d21ee [SPARK-48108][INFRA] Skip `tpcds-1g` and `docker-integration-tests` tests from `RocksDB UI-Backend` job
    omit 7b472e30db99 [SPARK-48104][INFRA] Run `publish_snapshot.yml` once per day
    omit c6696cdcd611 [SPARK-47671][CORE] Enable structured logging in log4j2.properties.template and update docs
    omit d9d79a54a3cd [SPARK-48102][SS] Track duration for acquiring source/sink metrics while reporting streaming query progress
    omit 8d9e7c9c6623 [SPARK-48099][INFRA] Run `maven-build` test only on `Java 21 on MacOS14 (Apple Silicon)`
    omit 7f6a1399a56b [SPARK-48098][INFRA] Enable `NOLINT_ON_COMPILE` for all except `lint` job
    omit f156db2e3852 [SPARK-48048][CONNECT][SS] Added client side listener support for Scala
    omit 5ec62a760bcd [SPARK-48065][SQL] SPJ: allowJoinKeysSubsetOfPartitionKeys is too strict
    omit 1321dd604480 [SPARK-48097][INFRA] Limit GHA job execution time to up to 3 hours in `build_and_test.yml`
    omit 53d4cdb4eefa [SPARK-48096][INFRA] Run `build_maven_java21_macos14.yml` every two days
    omit 48df28f6b311 [SPARK-48095][INFRA] Run `build_non_ansi.yml` once per day
    omit ffa4d198cec6 [SPARK-48067][SQL] Fix variant default columns
    omit bf1300835503 [SPARK-48079][BUILD] Upgrade maven-install/deploy-plugin to 3.1.2
    omit b99a64b0fd1c [SPARK-48081] Fix ClassCastException in NTile.checkInputDataTypes() when argument is non-foldable or of wrong type
    omit 5bbbc6c25bb7 [SPARK-48072][SQL][TESTS] Improve SQLQuerySuite test output - use `===` instead of `sameElements` for Arrays
    omit 2f31d05d75e1 [SPARK-48056][CONNECT][PYTHON] Re-execute plan if a SESSION_NOT_FOUND error is raised and no partial response was received
    omit ae5da1814573 [SPARK-48075][SS] Add type checking for PySpark avro functions
    omit 66e2a350fc55 [SPARK-48080][K8S] Promote `*MainAppResource` and `NonJVMResource` to `DeveloperApi`
    omit 9e12bd7f2b40 [SPARK-48064][SQL] Update error messages for routine related error classes
    omit 2b76ef43afb9 [SPARK-47683][PYTHON][BUILD][FOLLOW-UP] Exclude `lib/py4j*zip` in `pyspark-connect` package
    omit 2fbfb21896bb [SPARK-48058][SPARK-43727][PYTHON][CONNECT] UserDefinedFunction.returnType` parse the DDL string
    omit 4b16238784e0 [SPARK-48078][K8S] Promote `o.a.s.d.k8s.Constants` to `DeveloperApi`
    omit 918c50afa007 [SPARK-46894][PYTHON][FOLLOW-UP] Includes `error-conditions.json` into PyPI package
    omit a42eef9e029a [SPARK-48077][K8S] Promote `KubernetesClientUtils` to `DeveloperApi`
    omit 0fc7c4a29c46 [SPARK-45891][SQL][FOLLOW-UP] Added length check to the is_variant_null expression
    omit 04f3a938895c [SPARK-48076][K8S] Promote `KubernetesVolumeUtils` to `DeveloperApi`
    omit e521d3c1f357 [MINOR] Fix the grammar of some comments on renaming error classes
    omit f86a51921f73 [SPARK-48062][PYTHON][SS][TESTS] Add pyspark test for SimpleDataSourceStreamingReader
    omit fd57c3493af7 [SPARK-47911][SQL] Introduces a universal BinaryFormatter to make binary output consistent
    omit 69ea082fc69a [SPARK-47934][CORE] Ensure trailing slashes in `HistoryServer` URL redirections
    omit 5ac803079b30 [SPARK-48074][CORE] Improve the readability of JSON loggings
    omit 35767bb09fe1 [SPARK-48070][SQL][TESTS] Support `AdaptiveQueryExecSuite.runAdaptiveAndVerifyResult` to skip check results
    omit ad63eef20617 [SPARK-46009][SQL][FOLLOWUP] Remove unused PERCENTILE_CONT and PERCENTILE_DISC in g4
    omit 26c871f18030 [SPARK-48068][PYTHON] `mypy` should have `--python-executable` parameter
    omit ff401dde5034 [SPARK-48069][INFRA] Handle `PEP-632` by checking `ModuleNotFoundError` on `setuptools` in Python 3.12
    omit 65cf5b18648a [SPARK-48016][SQL][TESTS][FOLLOWUP] Update Java 21 golden file
    omit 02206cd66dbf [SPARK-48047][SQL] Reduce memory pressure of empty TreeNode tags
    omit 991763c2cdf8 [SPARK-46894][PYTHON] Move PySpark error conditions into standalone JSON file
    omit c71d02ab7c80 [SPARK-48028][TESTS] Regenerate benchmark results after turning ANSI on
    omit f3cc8f930383 [SPARK-48063][CORE] Enable `spark.stage.ignoreDecommissionFetchFailure` by default
    omit faab553cac70 [SPARK-48060][SS][TESTS] Fix `StreamingQueryHashPartitionVerifySuite` to update golden files correctly
    omit dab20b31388b [SPARK-48057][PYTHON][CONNECT][TESTS] Enable `GroupedApplyInPandasTests.test_grouped_with_empty_partition`
    omit 9caa6f7f8b8e [SPARK-48061][SQL][TESTS] Parameterize max limits of `spark.sql.test.randomDataGenerator`
    omit 0329479acb67 [SPARK-47359][SQL] Support TRANSLATE function to work with collated strings
    omit c8c249204178 [SPARK-47793][SS][PYTHON] Implement SimpleDataSourceStreamReader for python streaming data source
    omit 3aea6c258bf3 [SPARK-48003][SQL] Add collation support for hll sketch aggregate
    omit 94763438943e [SPARK-48050][SS] Log logical plan at query start
    omit 332570f42203 [SPARK-48052][PYTHON][CONNECT] Recover `pyspark-connect` CI by parent classes
    omit 12a507464f10 [SPARK-47566][SQL] Support SubstringIndex function to work with collated strings
    omit 9e8c4aa3f43a [SPARK-46122][SQL] Set `spark.sql.legacy.createHiveTableByDefault` to `false` by default
    omit ed5aa56f1200 [SPARK-48055][PYTHON][CONNECT][TESTS] Enable `PandasUDFScalarParityTests.{test_vectorized_udf_empty_partition, test_vectorized_udf_struct_with_empty_partition}`
    omit e0af82497607 [SPARK-48053][PYTHON][CONNECT] SparkSession.createDataFrame should warn for unsupported options
    omit 87b20b166c41 [SPARK-47585][SQL] SQL core: Migrate logInfo with variables to structured logging framework
    omit e6217c111fbd [SPARK-48030][SQL] SPJ: cache rowOrdering and structType for InternalRowComparableWrapper
    omit da92293f9ce0 [SPARK-48033][SQL] Fix `RuntimeReplaceable` expressions being used in default columns
    omit fe05eb8fa3b2 [SPARK-47741] Added stack overflow handling in parser
    omit 3fbcb26d8e99 [SPARK-48016][SQL] Fix a bug in try_divide function when with decimals
    omit c9ed9dfccb72 [SPARK-48042][SQL] Use a timestamp formatter with timezone at class level instead of making copies at method level
    omit c35a21e5984f [SPARK-48044][PYTHON][CONNECT] Cache `DataFrame.isStreaming`
    omit f781d153a5e4 [SPARK-48046][K8S] Remove `clock` parameter from `DriverServiceFeatureStep`
    omit d42c10d9411d [SPARK-47693][TESTS][FOLLOWUP] Reduce CollationBenchmarks time
    omit ccb0eb699f7c [SPARK-48038][K8S] Promote driverServiceName to KubernetesDriverConf
    omit d913d1b2662c [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation
    omit 3f15ad40640c [SPARK-47994][SQL] Fix bug with CASE WHEN column filter push down in SQLServer
    omit c74f584481d9 [SPARK-48039][PYTHON][CONNECT] Update the error class for `group.apply`
    omit 7b1147a05a6c [SPARK-47567][SQL] Support LOCATE function to work with collated strings
    omit 0e52b59b540f [SPARK-47939][SQL] Implement a new Analyzer rule to move ParameterizedQuery inside ExplainCommand and DescribeQueryCommand
    omit eaed585c55e5 [SPARK-48002][PYTHON][SS][TESTS] Adds sleep before event testing after query termination
    omit ff0751a56f01 [MINOR][DOCS] Remove space in the middle of configuration name in Arrow-optimized Python UDF page
    omit e1445e3f1cf5 [SPARK-48036][DOCS] Update `sql-ref-ansi-compliance.md` and `sql-ref-identifier.md`
    omit 9a42610d5ad8 [SPARK-48029][INFRA] Update the packages name removed in building the spark docker image
    omit 59d5946cfd37 [SPARK-48034][TESTS] NullPointerException in MapStatusesSerDeserBenchmark
    omit 8f1634e833ce [SPARK-48032][BUILD] Upgrade `commons-codec` to 1.17.0
    omit 3d62dd72a58f [SPARK-47730][K8S] Support `APP_ID` and `EXECUTOR_ID` placeholders in labels
    omit 8c446f35dc03 [SPARK-47292][SS] safeMapToJValue should consider null typed values
    omit 506b2d5eb8d9 [SPARK-48014][SQL] Change the makeFromJava error in EvaluatePython to a user-facing error
    omit 023f07d845c3 [SPARK-47933][CONNECT][PYTHON][FOLLOW-UP] Remove `pyspark.sql.classic` reference in `pyspark.ml.stat`
    omit 0bf39459e435 [SPARK-48025][SQL][TESTS] Fix org.apache.spark.sql.execution.benchmark.DateTimeBenchmark
    omit 657d9d0df5d6 [SPARK-48024][PYTHON][CONNECT][TESTS] Enable `UDFParityTests.test_udf_timestamp_ntz`
    omit a80579bbcf74 [SPARK-48002][PYTHON][SS] Add test for observed metrics in PySpark StreamingQueryListener
    omit 1f9e09ce2148 [SPARK-48021][ML][BUILD][FOLLOWUP] add `--add-modules=jdk.incubator.vector` to maven compile args
    omit 8b8ea60bd4f2 [SPARK-47927][SQL] Fix nullability attribute in UDF decoder
    omit 76ce6b00e036 [SPARK-48019] Fix incorrect behavior in ColumnVector/ColumnarArray with dictionary and nulls
    omit 64d321926bbc [SPARK-48021][ML][BUILD] Add `--add-modules=jdk.incubator.vector` to `JavaModuleOptions`
    omit 356830ada6c6 [SPARK-48020][INFRA][PYTHON] Pin 'pandas==2.2.2'
    omit b623601910a3 [SPARK-47408][SQL] Fix mathExpressions that use StringType
    omit d5712cea88cd [SPARK-48018][SS] Fix null groupId causing missing param error when throwing KafkaException.couldNotReadOffsetRange
    omit 2b2a33cc35a8 [SPARK-48011][CORE] Store LogKey name as a value to avoid generating new string instances
    omit 6098bd944f66 [SPARK-48010][SQL] Avoid repeated calls to conf.resolver in resolveExpression
    omit 9cf6dc873ff3 [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms
    omit 675f5f0b599b [SPARK-47818][CONNECT][FOLLOW-UP] Introduce plan cache in SparkConnectPlanner to improve performance of Analyze requests
    omit 78b19d5af08e [SPARK-48005][PS][CONNECT][TESTS] Enable `DefaultIndexParityTests.test_index_distributed_sequence_cleanup`
    omit beda1a4615c7 [SPARK-47440][SQL][FOLLOWUP] Reenable predicate pushdown for syntax with boolean comparison in MsSqlServer
    omit 733e53a4ff03 [SPARK-47968][SQL] MsSQLServer: Map datatimeoffset to TimestampType
    omit 07b84dd57e38 [SPARK-47476][SQL] Support REPLACE function to work with collated strings
    omit 4ee528f9b29f [SPARK-48007][BUILD][TESTS] Upgrade `mssql.jdbc` to `12.6.1.jre11`
    omit afe310d617e5 [SPARK-47351][SQL] Add collation support for StringToMap & Mask string expressions
    omit 6c40214f3d93 [SPARK-47350][SQL] Add collation support for SplitPart string expression
    omit b8b6d17ad8e4 [SPARK-48004][SQL] Add WriteFilesExecBase trait for v1 write
    omit 95d6c615c081 [SPARK-47355][SQL] Use wildcard imports in CollationTypeCasts
    omit e04ac56e645f [SPARK-45225][SQL][FOLLOW-UP] XML: Fix nested XSD file path resolution
    omit 3451e66fe71d [SPARK-47993][PYTHON] Drop Python 3.8
    omit 2e5825fb32c0 [SPARK-47858][PYTHON][FOLLOWUP] Excluding Python magic methods from error context target
    omit a7150074f0a5 [SPARK-48001][CORE] Remove unused `private implicit def arrayToArrayWritable` from `SparkContext`
    omit 7d04d0f043d2 [SPARK-47986][CONNECT][PYTHON] Unable to create a new session when the default session is closed by the server
    omit b0e03a193531 [SPARK-47999][SS] Improve logging around snapshot creation and adding/removing entries from state cache map in HDFS backed state store provider
    omit 033ca3e7dd5b [SPARK-47922][SQL] Implement the try_parse_json expression
    omit ea4b7a242910 [SPARK-47991][SQL][TEST] Arrange the test cases for window frames and window functions
    omit 79357c8ccd22 [SPARK-47933][CONNECT][PYTHON][FOLLOW-UP] Avoid referencing _to_seq in `pyspark-connect`
    omit d540786d9cea [SPARK-47597][STREAMING] Streaming: Migrate logInfo with variables to structured logging framework
    omit 994775a624f3 [SPARK-47955][SQL] Improve `DeduplicateRelations` performance
    omit e1d021214c61 [SPARK-45425][DOCS][FOLLOWUP] Add a migration guide for TINYINT type mapping change
    omit 287d02073929 [SPARK-47989][SQL] MsSQLServer: Fix the scope of spark.sql.legacy.mssqlserver.numericMapping.enabled
    omit de5c512e0179 [SPARK-47987][PYTHON][CONNECT][TESTS] Enable `ArrowParityTests.test_createDataFrame_empty_partition`
    omit 5810554ce0fa [SPARK-47990][BUILD] Upgrade `zstd-jni` to 1.5.6-3
    omit 24b0c7560718 [SPARK-47985][PYTHON] Simplify functions with `lit`
    omit a84cffd8b3da [SPARK-47982][BUILD] Update some code style's plugins to latest version
    omit 5f730c84abd7 [SPARK-47984][ML][SQL] Change `MetricsAggregate/V2Aggregator#serialize/deserialize` to call `SparkSerDeUtils#serialize/deserialize`
    omit 7090bc1f43fd [SPARK-47981][BUILD] Upgrade `Arrow` to 16.0.0
    omit a066d0c17853 [SPARK-47983][SQL] Demote spark.sql.pyspark.legacy.inferArrayTypeFromFirstElement.enabled to internal
    omit 775bc54fcd0d [SPARK-47580][SQL] SQL catalyst: eliminate unnamed variables in error logs
    omit 08caa567fb29 [SPARK-47980][SQL][TESTS] Reactivate test 'Empty float/double array columns raise EOFException'
    omit dab4a044b647 [SPARK-47973][CORE] Log call site in SparkContext.stop() and later in SparkContext.assertNotStopped()
    omit b4624bf4be28 [SPARK-47414][SQL] Lowercase collation support for regexp expressions
    omit c6aaa18e6cfd Revert "[SPARK-45302][PYTHON] Remove PID communication between Pythonworkers when no demon is used"
    omit ea37c860a1a8 [SPARK-47962][PYTHON][DOCS] PySpark Dataframe doc test improvement
    omit d23389252a7d [SPARK-47967][SQL] Make `JdbcUtils.makeGetter` handle reading time type as NTZ correctly
    omit 0fcced63be99 [SPARK-47979][SQL][TESTS] Use Hive tables explicitly for Hive table capability tests
    omit 5a1559a7ef03 [SPARK-47921][CONNECT] Fix ExecuteJobTag creation in ExecuteHolder
    omit 62dd64a5d13d [SPARK-47583][CORE] SQL core: Migrate logError with variables to structured logging framework
    omit 03d4ea6a707c [SPARK-47974][BUILD] Remove `install_scala` from `build/mvn`
    omit 09ed09cb18e7 [SPARK-47958][TESTS] Change LocalSchedulerBackend to notify scheduler of executor on start
    omit 0042b676c232 [SPARK-47965][CORE] Avoid orNull in TypedConfigBuilder and OptionalConfigEntry
    omit 0f1b6446a25a [SPARK-47971][PYTHON][CONNECT][TESTS] Reenable `PandasUDFGroupedAggParityTests.test_grouped_with_empty_partition`
    omit c44493db1dd7 [SPARK-47764][CORE][SQL] Cleanup shuffle dependencies based on ShuffleCleanupMode
    omit 461ffa1a681b [SPARK-47692][SQL] Fix default StringType meaning in implicit casting
    omit 890f78d03020 [SPARK-47418][SQL] Add hand-crafted implementations for lowercase unicode-aware contains, startsWith and endsWith and optimize UTF8_BINARY_LCASE
    omit 6f01982094f6 [SPARK-47964][PYTHON][CONNECT] Hide SQLContext and HiveContext in pyspark-connect
    omit fd695be19d3f [SPARK-47903][PYTHON][FOLLOW-UP] Removed changes relating to try_parse_json
    omit cb1e1f5cd49a [SPARK-47969][PYTHON][TESTS] Make `test_creation_index` deterministic
    omit 3cf0c83d29aa [SPARK-47771][PYTHON][DOCS][TESTS][FOLLOWUP] Make `max_by, min_by` doctests deterministic
    omit aa4a84bd8474 [SPARK-47933][PYTHON][CONNECT][FOLLOW-UP] Add a check of `__name__` at `_with_origin`
    omit b3c11ef8e090 Revert "Revert "[SPARK-45302][PYTHON] Remove PID communication between Python workers when no demon is used""
    omit e8f529bb89a6 Revert "[SPARK-45302][PYTHON] Remove PID communication between Python workers when no demon is used"
    omit 390fb7429029 [SPARK-47941][SS][CONNECT] Propagate ForeachBatch worker initialization errors to users for PySpark
    omit c88fabfee41d [SPARK-47604][CORE] Resource managers: Migrate logInfo with variables to structured logging framework
    omit e50737be366a [SPARK-47864][FOLLOWUP][PYTHON][DOCS] Fix minor typo: "MLLib" -> "MLlib"
    omit 66613ba042c4 [SPARK-47956][SQL] Sanity check for unresolved LCA reference
    omit 2b01755f2791 [SPARK-47948][PYTHON] Upgrade the minimum `Pandas` version to 2.0.0
    omit 9c4f12ca04ac [SPARK-47949][SQL][DOCKER][TESTS] MsSQLServer: Bump up mssql docker image version to 2022-CU12-GDR1-ubuntu-22.04
    omit cf5fc0c720ee [MINOR][DOCS] Fix type hint of 3 functions
    omit 33fa77cb4868 [MINOR][DOCS] Add `docs/_generated/` to .gitignore
    omit ca916258b991 [SPARK-47953][DOCS] MsSQLServer: Document Mapping Spark SQL Data Types to Microsoft SQL Server
    omit 9d9f292ac941 [SPARK-47873][SQL] Write collated strings to Hive metastore using the regular string type
    omit b9f2270f5b0b [SPARK-47352][SQL] Fix Upper, Lower, InitCap collation awareness
    omit 61ac3421651c [SPARK-47805][SS] Implementing TTL for MapState
    omit e74221e6525e [SPARK-47945][SQL] MsSQLServer: Document Mapping Spark SQL Data Types from Microsoft SQL Server and add tests
    omit d1298e73a8d5 [SPARK-47931][SQL] Remove unused and leaked threadlocal/session sessionHive
    omit 885e98ecbe64 [SPARK-47412][SQL] Add Collation Support for LPad/RPad
    omit eba6364dbb36 [SPARK-47633][SQL] Include right-side plan output in `LateralJoin#allAttributes` for more consistent canonicalization
    omit 876c2cf34a35 [SPARK-44170][BUILD][FOLLOWUP] Align JUnit5 dependency's version and clean up exclusions
    omit 9d715ba49171 [SPARK-47938][SQL] MsSQLServer: Cannot find data type BYTE error
    omit 3c7905e00d2e [SPARK-47600][CORE] MLLib: Migrate logInfo with variables to structured logging framework
    omit e01ac581f46a [SPARK-47933][PYTHON] Parent Column class for Spark Connect and Spark Classic
    omit a97e72cfa7d4 [SPARK-47937][PYTHON][DOCS] Fix docstring of `hll_sketch_agg`
    omit e4fb7dd98219 [MINOR] Remove unnecessary `imports`
    omit b335dd366fb1 [SPARK-47909][CONNECT][PYTHON][TESTS][FOLLOW-UP] Move `pyspark.classic` references
    omit 9e7ee7601d38 [SPARK-47903][PYTHON] Add support for remaining scalar types in the PySpark Variant library
    omit ac9a12ef6e06 [SPARK-47942][K8S][DOCS] Drop K8s v1.26 Support
    omit fc0c8553ea05 [SPARK-47904][SQL] Preserve case in Avro schema when using enableStableIdentifiersForUnionType
    omit f2d0cf23018f [SPARK-47907][SQL] Put bang under a config
    omit 86563169eef8 [SPARK-47940][BUILD][TESTS] Upgrade `guava` dependency to `33.1.0-jre` in Docker IT
    omit 676d47ffe091 [SPARK-47935][INFRA][PYTHON] Pin `pandas==2.0.3` for `pypy3.8`
    omit 256fc51508e4 [SPARK-47411][SQL] Support StringInstr & FindInSet functions to work with collated strings
    omit 61dc9d991373 [SPARK-47928][SQL][TEST] Speed up test "Add jar support Ivy URI in SQL"
    omit b20356ef55a2 [SPARK-47900] Fix check for implicit (UTF8_BINARY) collation
    omit 2fb31dea1c53 [SPARK-47930][BUILD] Upgrade RoaringBitmap to 1.0.6
    omit 79a1fa4b84dd [SPARK-47890][CONNECT][PYTHON] Add variant functions to Scala and Python
    omit e1432ef6405a [SPARK-47413][SQL] - add support to substr/left/right for collations
    omit 2d0b56c3eac6 [SPARK-47932][SQL][TESTS] Avoid using legacy commons-lang
    omit 458f70bd5213 [SPARK-47902][SQL] Making Compute Current Time* expressions foldable
    omit 9f34b8eca2f3 [SPARK-47845][SQL][PYTHON][CONNECT] Support Column type in split function for scala and python
    omit 2d9d444b122d [MINOR][DOCS] Change `SPARK_ANSI_SQL_MODE`in PlanStabilitySuite documentation
    omit 393a84fb074a [SPARK-47909][PYTHON][CONNECT] Parent DataFrame class for Spark Connect and Spark Classic
    omit adf02d38061b [SPARK-47925][SQL][TESTS] Mark `BloomFilterAggregateQuerySuite` as `ExtendedSQLTest`
    omit 2613516110a4 [SPARK-47924][CORE] Add a DEBUG log to `DiskStore.moveFileToBlock`
    omit 3fcc0f7ac142 [SPARK-47923][R] Upgrade the minimum version of `arrow` R package to 10.0.0
    omit 8aa8ad6be7b3 [SPARK-47915][BUILD][K8S] Upgrade `kubernetes-client` to 6.12.1
    omit 0d553d06fe2f [SPARK-47906][PYTHON][DOCS] Fix docstring and type hint of `hll_union_agg`
    omit 2bf43460b923 [SPARK-47833][SQL][CORE] Supply caller stackstrace for checkAndGlobPathIfNecessary AnalysisException
    omit 06b12fc31709 [SPARK-47901][BUILD] Upgrade common-text 1.12.0
    omit 8aa2dad46b79 [SPARK-47596][DSTREAMS] Streaming: Migrate logWarn with variables to structured logging framework
    omit fe47edece059 [SPARK-47883][SQL] Make `CollectTailExec.doExecute` lazy with RowQueue
    omit 76fb196a8ab7 [SPARK-46935][DOCS] Consolidate error documentation
    omit bb5ded8f1bb0 [SPARK-47371][SQL][FOLLOWUP] XML: Stop ignoring CDATA within row tags
    omit e2e17fe6e038 [SPARK-47816][CONNECT][DOCS][FOLLOWUP] refine the description
    omit d09230b6601c [SPARK-44461][FOLLOWUP][SS][CONNECT] Remove unneeded TODOs
    omit 074ddc282567 [SPARK-47898][SQL] Port HIVE-12270: Add DBTokenStore support to HS2 delegation token
    omit 9f9cc87c1a19 [MINOR][TESTS] Replace CONFIG_DIM1 with CONFIG_DIM2 in timestamp tests
    omit 6232085227ee [SPARK-47887][CONNECT] Remove unused import `spark/connect/common.proto` from `spark/connect/relations.proto`
    omit 51ca47da6c1a [SPARK-47893][BUILD] Upgrade ASM to 9.7
    omit fd3f70c74ba6 [SPARK-47858][SPARK-47852][PYTHON][SQL][FOLLOWUP] Fix Python Liner
    omit b5bb75ca240a [SPARK-47895][SQL] group by all should be idempotent
    omit 52a2e63dd714 [SPARK-47850][SQL] Support `spark.sql.hive.convertInsertingUnpartitionedTable`
    omit bcafb43825ac [SPARK-47767][SQL] Show offset value in TakeOrderedAndProjectExec
    omit 216379df3543 [SPARK-47858][SPARK-47852][PYTHON][SQL] Refactoring the structure for DataFrame error context
    omit 564b2384c40d [SPARK-47864][PYTHON][DOCS] Enhance "Installation" page to cover all installable options
    omit eb8688c2b6ce [SPARK-47896][BUILD] Upgrade netty to `4.1.109.Final`
    omit 9718573ce748 [SPARK-47591][SQL] Hive-thriftserver: Migrate logInfo with variables to structured logging framework
    omit 47d783bc6489 [SPARK-47882][SQL] createTableColumnTypes need to be mapped to database types instead of using directly
    omit 24b4581fa818 [SPARK-47894][CORE][WEBUI] Add `Environment` page to Master UI
    omit 60e75e6e0275 [SPARK-47810][SQL] Replace equivalent expression to <=> in join condition
    omit c4b0c260bb13 [SPARK-47839][SQL] Fix aggregate bug in RewriteWithExpression
    omit 61a7901a2c83 [SPARK-47846][SQL] Add support for Variant type in from_json expression
    omit 37afc3fff8c6 [SPARK-47803][FOLLOWUP] Fix cast binary/decimal to variant
    omit 21d8bbdc59d6 [SPARK-47867][FOLLOWUP] Fix variant parsing in JacksonParser
    omit 2e81f9a3b561 [SPARK-46810][DOCS][FOLLOWUP] Make some reference file links clickable
    omit a7f8ccef122a [SPARK-47891][PYTHON][DOCS] Improve docstring of mapInPandas
    omit 4957a40d6e6b [SPARK-47584][SQL] SQL core: Migrate logWarn with variables to structured logging framework
    omit 898838a239d3 [SPARK-47627][SQL] Add SQL MERGE syntax to enable schema evolution
    omit f99b1b1dd14d [SPARK-47360][SQL] Collation support: Overlay, FormatString, Length, BitLength, OctetLength, SoundEx, Luhncheck
    omit be6eef97a81c [SPARK-47726][DOC] Document push-based shuffle metrics
    omit 1b65b2aebf4e [SPARK-47416][SQL] Add new functions to CollationBenchmark
    omit 19833d92f325 [SPARK-47886][SQL][DOCS][TESTS] Postgres: Add tests and doc for Postgres special numeric values
    omit e2e6710fc8ea [SPARK-47765][SQL][FOLLOWUP] Disable SET COLLATION when collations are disabled
    omit 7dc88db0c8b6 [SPARK-47885][PYTHON][CONNECT] Make pyspark.resource compatible with pyspark-connect
    omit 5da7b6878c20 [SPARK-47763][CONNECT][TESTS] Enable local-cluster tests with pyspark-connect package
    omit 1b53f122d56c [SPARK-47863][SQL] Fix startsWith & endsWith collation-aware implementation for ICU
    omit e327c1220abf [SPARK-47884][INFRA] Switch ANSI SQL CI job to NON-ANSI SQL CI job
    omit 6fb2f7c3772a [SPARK-44444][SQL] Use ANSI SQL mode by default
    omit 4e754f778fdc [SPARK-47822][SQL] Prohibit Hash Expressions from hashing the Variant Data Type
    omit 76882d0c13db [SPARK-47821][SQL] Implement is_variant_null expression
    omit 4913c344756d [SPARK-47867][SQL] Support variant in JSON scan
    omit 8c2c0fb43a9d [SPARK-46812][CONNECT][PYTHON][FOLLOW-UP] Add pyspark.pyspark.sql.connect.resource into PyPi packaging
    omit f77495909b29 [SPARK-47588][CORE] Hive module: Migrate logInfo with variables to structured logging framework
    omit 2054ab0fb03f [SPARK-47880][SQL][DOCS] Oracle: Document Mapping Spark SQL Data Types to Oracle
    omit 268856da31c1 [SPARK-47879][SQL] Oracle: Use VARCHAR2 instead of VARCHAR for VarcharType mapping
    omit ab6338e09aa0 [SPARK-47838][BUILD] Upgrade `rocksdbjni` to 8.11.4
    omit 6c827c10dc15 [SPARK-47876][PYTHON][DOCS] Improve docstring of mapInArrow
    omit f9ebe1b3d24b [SPARK-46375][DOCS] Add user guide for Python data source API
    omit 86837d3155b1 [SPARK-47877][SS][CONNECT] Speed up test_parity_listener
    omit 5321353b24db [SPARK-47875][CORE] Remove `spark.deploy.recoverySerializer`
    omit 57c7db2c4c1d [SPARK-47760][SPARK-47763][CONNECT][TESTS] Reeanble Avro and Protobuf function doctests
    omit 119271d49356 [SPARK-47868][CONNECT] Fix recursion limit error in SparkConnectPlanner and SparkSession
    omit f7440f384191 [SPARK-47590][SQL] Hive-thriftserver: Migrate logWarn with variables to structured logging framework
    omit 6919febfcc87 [SPARK-47594] Connector module: Migrate logInfo with variables to structured logging framework
    omit 9a1fc112677f [SPARK-47871][SQL] Oracle: Map TimestampType to TIMESTAMP WITH LOCAL TIME ZONE
    omit ee2673f2e948 [SPARK-47417][SQL] Collation support: Ascii, Chr, Base64, UnBase64, Decode, StringDecode, Encode, ToBinary, FormatNumber, Sentences
    omit 9b1b2b30d591 [SPARK-47081][CONNECT][FOLLOW] Unflake Progress Execution
    omit 4dad2170b05c [SPARK-47356][SQL] Add support for ConcatWs & Elt (all collations)
    omit fa2e9c7275aa [SPARK-47739][SQL] Register logical avro type
    omit b7a729bfd19c [SPARK-46574][BUILD] Upgrade maven plugin to latest version
    omit a1fc6d57b27d [SPARK-47818][CONNECT] Introduce plan cache in SparkConnectPlanner to improve performance of Analyze requests
    omit 6762d1f8beaf [SPARK-47420][SQL] Fix test output
    omit 348b1bcff75c [SPARK-47769][SQL] Add schema_of_variant_agg expression
    omit fa3ef03a0734 [SPARK-47463][SQL] Use V2Predicate to wrap expression with return type of boolean
    omit 21cc89b68ba5 [SPARK-47862][PYTHON][CONNECT] Fix generation of proto files
    omit 3ff339362b75 [SPARK-47861][BUILD] Upgrade `slf4j` to 2.0.13
    omit f21719346eb0 [SPARK-47840][SS] Disable foldable propagation across Streaming Aggregate/Join nodes
    omit 51d3efcead5b [SPARK-47233][CONNECT][SS][2/2] Client & Server logic for Client side streaming query listener
    omit e815012f26ab [SPARK-47371][SQL] XML: Ignore row tags found in CDATA
    omit 61f8ec6c5175 [SPARK-47866][SQL][TESTS] Use explicit GC in `PythonForeachWriterSuite`
    omit c5b8e60e0d59 [SPARK-46810][DOCS] Align error class terminology with SQL standard
    omit be080703688f [SPARK-47673][SS] Implementing TTL for ListState
    omit f10ad3d56896 [SPARK-47804] Add Dataframe cache debug log
    omit 6d1b3668db42 [SPARK-47855][CONNECT] Add `spark.sql.execution.arrow.pyspark.fallback.enabled` in the unsupported list
    omit becbca6752a5 [SPARK-47860][BUILD][K8S] Upgrade `kubernetes-client` to 6.12.0
    omit 61264f77fd68 [SPARK-47603][KUBERNETES][YARN] Resource managers: Migrate logWarn with variables to structured logging framework
    omit 83fe9b16ab5a [SPARK-47694][CONNECT] Make max message size configurable on the client side
    omit ba673d74973a [SPARK-47856][SQL] Document Mapping Spark SQL Data Types from Oracle and add tests
    omit f3a6ca9e2c47 [SPARK-47357][SQL] Add support for Upper, Lower, InitCap (all collations)
    omit b8354bbe53c0 [SPARK-47851][CONNECT][DOCS] Document pyspark-connect package
    omit e6b7950f553c [SPARK-47788][SS] Ensure the same hash partitioning for streaming stateful ops
    omit f86dc2b7e5c3 [SPARK-47848][SS] Fix thread safety issue around access to loadedMaps in close function for hdfs store provider
    omit dd4c7fc6ae91 [SPARK-47803][SQL] Support cast to variant
    omit 73aa4059cd5c [SPARK-47849][PYTHON][CONNECT] Change release script to release pyspark-connect
    omit a7cff5c7a383 [SPARK-47757][SPARK-47756][CONNECT][PYTHON][TESTS] Make testing Spark Connect server having pyspark.core
    omit 1096801ead29 [SPARK-47828][CONNECT][PYTHON] DataFrameWriterV2.overwrite` fails with invalid plan
    omit e7d0ba783f43 [SPARK-47826][SQL][PYTHON][FOLLOW-UP] Do not use list[...] that is invalid type syntax for Python 3.8
    omit 85c4f053f25a [SPARK-47841][BUILD] Upgrade `postgresql` to 42.7.3
    omit b53c6f9d1225 [SPARK-47253][CORE] Allow LiveEventBus to stop without the completely draining of event queue
    omit e7fc4003b246 [SPARK-47812][CONNECT] Support Serialization of SparkSession for ForEachBatch worker
    omit f0ea0968d946 [MINOR][PYTHON] Enable parity test `test_different_group_key_cardinality`
    omit de00ac8a05ae [SPARK-47765][SQL] Add SET COLLATION to parser rules
    omit 2bc9573e94f2 [SPARK-47819][CONNECT] Use asynchronous callback for execution cleanup
    omit daf260f74e12 [SPARK-47831][PS][CONNECT][TESTS] Run Pandas API on Spark for pyspark-connect package
    omit 49d2214a7660 [SPARK-47816][CONNECT][DOCS] Document the lazy evaluation of views in `spark.{sql, table}`
    omit ae93b46b8c2b [SPARK-47733][SS] Add custom metrics for transformWithState operator part of query progress
    omit 27987536be38 [SPARK-47800][SQL] Create new method for identifier to tableIdentifier conversion
    omit 4e46852d0880 [SPARK-47827][PYTHON] Missing warnings for deprecated features
    omit 1ee3496f4836 [SPARK-47792][CORE] Make the value of MDC can support `null` & cannot be `MessageWithContext`
    omit 9ffdbc65029a [SPARK-47784][SS] Merge TTLMode and TimeoutMode into a single TimeMode
    omit de2079163270 [SPARK-47795][K8S][DOCS] Supplement the doc of job schedule for K8S
    omit ffc378c1544b [SPARK-47813][SQL] Replace getArrayDimension with updateExtraColumnMeta
    omit 4510c0624209 [SPARK-47541][SQL][FOLLOWUP] Fix `AnsiTypeCoercion` to handle ArrayType
    omit f086c2327c36 [SPARK-47174][CONNECT][SS][1/2] Server side SparkConnectListenerBusListener for Client side streaming query listener
    omit a0ccdf27e5ff [SPARK-47824][PS] Fix nondeterminism in pyspark.pandas.series.asof
    omit aa568354725c [SPARK-47811][PYTHON][CONNECT][TESTS] Run ML tests for pyspark-connect package
    omit ed95b887829a [SPARK-47814][DSTREAM] Move `KinesisTestUtils` & `WriteInputFormatTestDataGenerator` from `main` to `test`
    omit c3cc38bae6e7 [SPARK-47318][CORE] Adds HKDF round to AuthEngine key derivation to follow standard KEX practices
    omit 3d2b7fea7fe0 [SPARK-47777][PYTHON][SS][TESTS] Add spark connect test for python streaming data source
    omit 5765c0150659 [SPARK-47820][INFRA] Run `ANSI` SQL CI twice per day
    omit f98d4e644e1b [SPARK-47799][BUILD] Add `-g` to javac compile parameters when using SBT package jar
    omit d0e7645a5a34 [SPARK-47797][K8S] Skip deleting pod from k8s if the pod does not exists
    omit 7c639a1579de [SPARK-47817][PYTHON][PS][BUILD] Update `pandas` to 2.2.2
    omit 31036274fc1c [SPARK-47410][SQL] Refactor UTF8String and CollationFactory
    omit 6e371e1df50e [SPARK-47617][SQL] Add TPC-DS testing infrastructure for collations
    omit dedddf510af9 [SPARK-47809][SQL] `checkExceptionInExpression` should check error for each codegen mode
    omit 6fdf9c9df545 [SPARK-47807][PYTHON][ML] Make pyspark.ml compatible with pyspark-connect
    omit c303b042966b [SPARK-47808][PYTHON][ML][TESTS] Make pyspark.ml.connect tests running without optional dependencies
    omit a8b919f924db [SPARK-47736][SQL] Add support for AbstractArrayType
    omit a54e53bc0c3c [SPARK-46812][CONNECT][FOLLOW-UP] Make `handleCreateResourceProfileCommand` private
    omit eaf6b518f67c [SPARK-47587][SQL] Hive module: Migrate logWarn with variables to structured logging framework
    omit 75d43dd05757 [SPARK-47601][GRAPHX] Graphx: Migrate logs with variables to structured logging framework
    omit 3da52fb4490e [SPARK-47798][SQL] Enrich the error message for the reading failures of decimal values
    omit aca3d1025e2d [MINOR][DOCS] Make the link of spark properties with YARN more accurate
    omit d0605bf3bf7b [SPARK-47001][SQL] Pushdown verification in optimizer
    omit 6c8e4cfd6f3f [MINOR][DOCS] Clarify relation between grouping API and `spark.sql.execution.arrow.maxRecordsPerBatch`
    omit a2b7050e0fc5 [SPARK-47704][SQL] JSON parsing fails with "java.lang.ClassCastException" when spark.sql.json.enablePartialResults is enabled
    omit 86ae0d2bc198 [SPARK-47274][PYTHON][SQL] Provide more useful context for PySpark DataFrame API errors
    omit b637bb92a2f8 [SPARK-47802][SQL] Revert (*) from meaning struct(*) back to meaning *
    omit 8da9a70f701a [SPARK-47725][INFRA][FOLLOW-UP] Do not run scheduled job in forked repository
    omit 697596bf9357 [MINOR][PYTHON][TESTS] Enable `test_udf_cache` parity test
    omit f6d5ad3ec75b [SPARK-47366][SQL][PYTHON] Add VariantVal for PySpark
    omit a100e11936bc [SPARK-41811][PYTHON][CONNECT] Implement `SQLStringFormatter` with `WithRelations`
    omit 52fd8c01cc8b [SPARK-47595][STREAMING] Streaming: Migrate logError with variables to structured logging framework
    omit 10a308774b71 [SPARK-47781][SPARK-47791][SPARK-47798][DOCS][FOLLOWUP] Update the decimal mapping remarks with JDBC data sources
    omit 627f6082edca [SPARK-47693][SQL] Add optimization for lowercase comparison of UTF8String used in UTF8_BINARY_LCASE collation
    omit d817c9a60f51 [SPARK-47775][SQL] Support remaining scalar types in the variant spec
    omit 4cb364e6f615 [SPARK-47680][SQL] Add variant_explode expression
    omit 3082319bb1d5 [SPARK-47791][SQL] Truncate exceed decimals with scale first instead of precision from JDBC datasource
    omit 0d7c07047a62 [SPARK-47790][BUILD] Upgrade `commons-io` to 2.16.1
    omit 7996b031f7f2 [SPARK-47706][BUILD] Bump json4s 4.0.7
    omit b53ec00421b1 [SPARK-47781][SQL] Handle negative scale decimals for JDBC data sources
    omit 520f3b1c192b [SPARK-47593][CORE] Connector module: Migrate logWarn with variables to structured logging framework
    omit 2f7cdc9cdaf4 [SPARK-47776][SS] Disallow binary inequality collation be used in key schema of stateful operator
    omit bebd78e13160 [SPARK-47761][SQL] Oracle: Support reading AnsiIntervalTypes
    omit e8e7dcce4ac7 [SPARK-47786] SELECT DISTINCT (*) should not become SELECT DISTINCT struct(*) (revert to previous behavior)
    omit ec509b49dcaa [SPARK-47586][SQL] Hive module: Migrate logError with variables to structured logging framework
    omit d1e1a1907412 [SPARK-47787][BUILD] Upgrade `commons-compress` to 1.26.1
    omit 0eb58a06a7d4 [SPARK-47785][BUILD][TESTS] Upgrade `bouncycastle` to 1.78
    omit a68337892246 [SPARK-47783] Add some missing SQLSTATEs an clean up the YY000 to use…
    omit 07b1346db477 [SPARK-47581][CORE][FOLLOWUP] Fix GA failure
    omit 2793397140af [SPARK-47782][BUILD] Remove redundant json4s-jackson definition in sql/api POM
    omit 1586652b6421 [MINOR][PS][TESTS] Make `test_compare` deterministic
    omit 070461cc673c [SPARK-47779][PS][TESTS] Add a helper function to sort PS Frame/Series
    omit 4a3fd8f7e69e [SPARK-47081][CONNECT][FOLLOW] Improving the usability of the Progress Handler
    omit 992b69dbc327 [SPARK-47774][INFRA] Remove redundant rules from `MimaExcludes`
    omit 83f3a34df7a0 [SPARK-47772][PYTHON][TESTS] Fix the doctest of `mode` function
    omit 149ac0f8893b [SPARK-47581][CORE] SQL catalyst: Migrate logWarning with variables to structured logging framework
    omit 702180614900 [SPARK-47754][SQL] Postgres: Support reading multidimensional arrays
    omit edba7b9f9119 [SPARK-47771][PYTHON][TESTS] Make `max_by, min_by` doctests deterministic
    omit 319edfdc5cd6 [SPARK-47682][SQL] Support cast from variant
    omit 08c49637795f [SPARK-47770][INFRA] Fix `GenerateMIMAIgnore.isPackagePrivateModule` to return `false` instead of failing
    omit fea1994fe756 [MINOR][TESTS] Make `check_invalid_args` reusable in parity test
    omit f8e652e88320 [SPARK-47589][SQL] Hive-Thriftserver: Migrate logError with variables to structured logging framework
    omit 04bf981781ba [SPARK-47737][PYTHON] Bump PyArrow to 10.0.0
    omit b1998bd4145d [SPARK-47725][INFRA] Set up the CI for pyspark-connect package
    omit abb7b046d50d [SPARK-47504][SQL] Resolve AbstractDataType simpleStrings for StringTypeCollated
    omit 134a13928965 [SPARK-47681][SQL] Add schema_of_variant expression
    omit 60806c63d97b [SPARK-47746] Implement ordinal-based range encoding in the RocksDBStateEncoder
    omit 29d077fbbd54 [SPARK-47748][BUILD] Upgrade `zstd-jni` to 1.5.6-2
    omit f94d95d75886 [SPARK-47762][PYTHON][CONNECT] Add pyspark.sql.connect.protobuf into setup.py
    omit 211afd4ab526 [MINOR][PYTHON][CONNECT][TESTS] Enable `MapInPandasParityTests.test_dataframes_with_incompatible_types`
    omit f0d8f82f8c45 [SPARK-47750][DOCS][SQL] Postgres: Document Mapping Spark SQL Data Types to PostgreSQL
    omit eb8e99721714 [SPARK-47657][SQL] Implement collation filter push down support per file source
    omit 3a39ac231fe4 [SPARK-47713][SQL][CONNECT] Fix a self-join failure
    omit d55bb617a135 [SPARK-47558][SS] State TTL support for ValueState
    omit f576b8542f2e [SPARK-47541][SQL] Collated strings in complex types supporting operations reverse, array_join, concat, map
    omit ad2367c55aeb [MINOR][PYTHON][SS][TESTS] Drop the tables after being used at `test_streaming_foreach_batch`
    omit cc6c0eb1bee6 [MINOR][TESTS] Deduplicate test cases `test_parse_datatype_string`
    omit b299b2bc06a9 [SPARK-47299][PYTHON][DOCS] Use the same `versions.json` in the dropdown of different versions of PySpark documents
    omit 0c992b205946 [SPARK-47755][CONNECT] Pivot should fail when the number of distinct values is too large
    omit e92e8f5441a7 [SPARK-47744] Add support for negative-valued bytes in range encoder
    omit f7dff4aa0c8f [SPARK-47752][PS][CONNECT] Make pyspark.pandas compatible with pyspark-connect
    omit d7430124191a [SPARK-47753][PYTHON][CONNECT][TESTS] Make pyspark.testing compatible with pyspark-connect
    omit c11585ac296e [SPARK-47751][PYTHON][CONNECT] Make pyspark.worker_utils compatible with pyspark-connect
    omit 4d9dbb35aacb [SPARK-46722][CONNECT][SS][TESTS][FOLLOW-UP] Drop the tables after tests finished
    omit 644687b66e1a [SPARK-47709][BUILD] Upgrade tink to 1.13.0
    omit 60a3fbcab4e5 [SPARK-47727][PYTHON] Make SparkConf to root level to for both SparkSession and SparkContext
    omit d69df596e512 [SPARK-47738][BUILD] Upgrade Kafka to 3.7.0
    omit 7385f19c7539 [SPARK-47592][CORE] Connector module: Migrate logError with variables to structured logging framework
    omit 42dc815b8446 [SPARK-47743][CORE] Use milliseconds as the time unit in logging
    omit 11abc64a731d [SPARK-47094][SQL] SPJ : Dynamically rebalance number of buckets when they are not equal
    omit d1ace24f8fac [SPARK-47582][SQL] Migrate Catalyst logInfo with variables to structured logging framework
    omit 1efbf43160aa [SPARK-47310][SS] Add micro-benchmark for merge operations for multiple values in value portion of state store
    omit 18072b5357a5 [SPARK-47577][CORE][PART2] Migrate logError with variables to structured logging framework
    omit c34baebb36d4 [SPARK-47719][SQL] Change spark.sql.legacy.timeParserPolicy default to CORRECTED
    omit 6bd0ccf36dd2 [SPARK-47511][SQL][FOLLOWUP] Rename the config REPLACE_NULLIF_USING_WITH_EXPR to be more general
    omit 12d0367cd304 [SPARK-47724][PYTHON][TESTS][FOLLOW-UP] Make testing script to inherits SPARK_CONNECT_TESTING_REMOTE env
    omit 97e63ff035b5 [SPARK-47735][PYTHON][TESTS] Make pyspark.testing.connectutils compatible with pyspark-connect
    omit aeb082e06091 [SPARK-47081][CONNECT][TESTS][FOLLOW-UP] Skip the flaky doctests for now
    omit d5620cbe9a2e [SPARK-47289][SQL] Allow extensions to log extended information in explain plan
    omit 0107435cb39d [SPARK-47734][PYTHON][TESTS] Fix flaky DataFrame.writeStream doctest by stopping streaming query
    omit b9ca91dde94c [SPARK-47712][CONNECT] Allow connect plugins to create and process Datasets
    omit 404d58c74e16 [SPARK-47081][CONNECT][FOLLOW-UP] Add the `shell` module into PyPI package
    omit fb96b1a8d648 [SPARK-47723][CORE][TESTS] Introduce a tool that can sort alphabetically enumeration field in `LogEntry` automatically
    omit 240923c2750e [SPARK-46812][PYTHON][TESTS][FOLLOWUP] Check should_test_connect and pyarrow to skip tests
    omit 3fd0cd609df6 [SPARK-47598][CORE] MLLib: Migrate logError with variables to structured logging framework
    omit e3405c1aa93c [SPARK-47610][CONNECT][FOLLOWUP] Add -Dio.netty.tryReflectionSetAccessible=true for spark-connect-scala-client
    omit 25fc67fa114d [SPARK-47728][DOC] Document G1 Concurrent GC metrics
    omit 5ca3467b8653 [SPARK-47729][PYTHON][TESTS] Get the proper default port for pyspark-connect testcases
    omit 5f9f5db9f363 [SPARK-47689][SQL][FOLLOWUP] More accurate file path in TASK_WRITE_FAILED error
    omit 3b8aea39953c Revert "[SPARK-47708][CONNECT] Do not log gRPC exception to stderr in PySpark
    omit bffb02d14fa4 [SPARK-47565][PYTHON] PySpark worker pool crash resilience
    omit f6999df0c7f0 [SPARK-47081][CONNECT] Support Query Execution Progress
    omit 3f6ac60e9966 [SPARK-47577][CORE][PART1] Migrate logError with variables to structured logging framework
    omit d75c77562d27 [SPARK-46812][PYTHON][TESTS][FOLLOWUP] Skip `pandas`-required tests if pandas is not available
    omit d272a1b4367e [SPARK-47724][PYTHON][TESTS] Add an environment variable for testing remote pure Python library
    omit c25fd93199cc [SPARK-47705][INFRA][FOLLOWUP] Sort LogKey alphabetically and build a test to ensure it
    omit 678aeb7ef708 [SPARK-47683][PYTHON][BUILD] Decouple PySpark core API to pyspark.core package
    omit 447f8aff6c26 [SPARK-47720][CORE] Update `spark.speculation.multiplier` to 3 and `spark.speculation.quantile` to 0.9
    omit d87ac8ef49db [SPARK-47708][CONNECT] Do not log gRPC exception to stderr in PySpark
    omit e3aab8c19492 [SPARK-47210][SQL] Addition of implicit casting without indeterminate support
    omit fbe6b1dba2ab [SPARK-47721][DOC] Guidelines for the Structured Logging Framework
    omit 49eefc5d1d92 [SPARK-47722][SS] Wait until RocksDB background work finish before closing
    omit 6a0555c39b2d [SPARK-47700][SQL] Fix formatting of error messages with treeNode
    omit 7dec5eb14644 [SPARK-47705][INFRA] Sort LogKey alphabetically and build a test to ensure it
    omit 1515c567e4f0 [SPARK-47553][SS] Add Java support for transformWithState operator APIs
    omit a427a4586177 [SPARK-47710][SQL][DOCS] Postgres: Document Mapping Spark SQL Data Types from PostgreSQL
    omit efe437ef9f9e [SPARK-47711][BUILD][TESTS] Parameterize JDBC Driver versions for docker integration tests
    omit 1404d1f801c7 [SPARK-47715][BUILD][STS] Upgrade hive-service-rpc 4.0.0
    omit d487acd3008b [SPARK-47707][SQL] Special handling of JSON type for MySQL Connector/J 5.x
    omit d16183d044a6 [SPARK-47653][SS] Add support for negative numeric types and range scan key encoder
    omit 6e3a42515067 [MINOR][DOCS] replace `-formatted code with <code> tags inside configuration.md tables
    omit 62f90ec6d32f [SPARK-47452][INFRA][FOLLOWUP] Enforce to install `six` to `Python 3.10`
    omit 9a20794b252d [SPARK-47701][SQL][TESTS] Postgres: Add test for Composite and Range types
    omit 360a3f9023d0 [SPARK-45733][PYTHON][TESTS][FOLLOWUP] Skip `pyspark.sql.tests.connect.client.test_client` if not should_test_connect
    omit 344f640b2f35 [SPARK-47454][PYTHON][TESTS][FOLLOWUP] Skip `test_create_dataframe_from_pandas_with_day_time_interval` if pandas is not avaiable
    omit 2c2a2adc3275 [SPARK-47655][SS] Integrate timer with Initial State handling for state-v2
    omit 49b7b6b9fe6b [SPARK-47691][SQL] Postgres: Support multi dimensional array on the write side
    omit 55b5ff6f45fd [SPARK-47669][SQL][CONNECT][PYTHON] Add `Column.try_cast`
    omit 27fdf96842ea [SPARK-47697][INFRA] Add Scala style check for invalid MDC usage
    omit d11d9cf729ab [SPARK-47699][BUILD] Upgrade `gcs-connector` to 2.2.21 and add a note for 3.0.0
    omit e6144e4d75b7 [SPARK-47695][BUILD] Upgrade AWS SDK v2 to 2.24.6
    omit db0975cb2a1c [SPARK-47602][CORE] Resource managers: Migrate logError with variables to structured logging framework
    omit 8d4e9647c971 [SPARK-47684][SQL] Postgres: Map length unspecified bpchar to StringType
    omit ba98b7ae53f8 [SPARK-47689][SQL] Do not wrap query execution error during data writing
    omit b1b1fde42bd8 Revert "[SPARK-47684][SQL] Postgres: Map length unspecified bpchar to StringType"
    omit 03f4e45cd7e9 [SPARK-47685][SQL] Restore the support for `Stream` type in `Dataset#groupBy`
    omit a598f654066d [SPARK-47664][PYTHON][CONNECT][TESTS][FOLLOW-UP] Add more tests
    omit 22771a6a87e7 [SPARK-47684][SQL] Postgres: Map length unspecified bpchar to StringType
    omit e2e6e09a24db [SPARK-47634][SQL] Add legacy support for disabling map key normalization
    omit eb9b12692601 [SPARK-47663][CORE][TESTS] add end to end test for task limiting according to different cpu and gpu configurations
    omit 00162b82fe3c [SPARK-47686][SQL][TESTS] Use `=!=` instead of `!==` in `JoinHintSuite`
    omit c3da26051d0c [SPARK-47665][SQL] Use SMALLINT to Write ShortType to MYSQL
    omit e9833495e13a [SPARK-47679][SQL] Use `HiveConf.getConfVars` or Hive conf names directly
    omit 1fd308951668 [SPARK-47551][SQL] Add variant_get expression
    omit 43cedc0b3dea [SPARK-46812][CONNECT][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile
    omit 58f44e8a00f3 [SPARK-47643][SS][PYTHON] Add pyspark test for python streaming source
    omit 128f74b055d3 [SPARK-47676][BUILD] Clean up the removed `VersionsSuite` references
    omit 72fe58d716d4 [SPARK-47675][K8S][TESTS] Use AWS SDK v2 `2.23.19` in K8s IT
    omit e86c499b008c [SPARK-47674][CORE] Enable `spark.metrics.appStatusSource.enabled` by default
    omit 14811974338e [SPARK-47659][CORE][TESTS] Improve `*LoggingSuite*`
    omit e9f204ae9306 [SPARK-46840][SQL][TESTS] Add `CollationBenchmark`
    omit 72c619e4f3ba [SPARK-47666][SQL] Fix NPE when reading mysql bit array as LongType
    omit 968cba200a5a [SPARK-47664][PYTHON][CONNECT] Validate the column name with cached schema
    omit cf02b1a60d9d [SPARK-47569][SQL] Disallow comparing variant
    omit 535734e3901f [SPARK-47431][SQL][FOLLOWUP] Do not access SQLConf in Literal.apply
    omit a5ca5867298f [SPARK-47654][INFRA] Structured logging framework: support log concatenation
    omit 4c1405d10b8f [SPARK-47645][BUILD][CORE][SQL][YARN] Make Spark build with `-release` instead of `-target`
    omit 18b582ca806c [SPARK-47662][SQL][DOCS] Add User Document for Mapping Spark SQL Data Types to MySQL
    omit ef4c27bac533 [SPARK-47658][BUILD] Upgrade `tink` to 1.12.0
    omit 1bc812b07921 [SPARK-47656][BUILD] Upgrade commons-io to 2.16.0
    omit 11d76c96554c [SPARK-47576][INFRA] Implement logInfo API in structured logging framework
    omit db14be8ab5f7 [SPARK-47637][SQL] Use errorCapturingIdentifier in more places
    omit d182810abcd8 [SPARK-47575][INFRA] Implement logWarning API in structured logging framework
    omit 48def2b4a0b5 [SPARK-47648][SQL][TESTS] Use `checkError()` to check Exception in `[CSV|Json|Xml]Suite` and `[Csv|Json]FunctionsSuite`
    omit 12155e8e027c [SPARK-47647][SQL] Make MySQL data source able to read bit(n>1) as BinaryType like Postgres
    omit 5318846db1e3 [SPARK-47641][SQL] Improve the performance for `UnaryMinus` and `Abs`
    omit 5d5611e1788e [SPARK-47642][BUILD] Exclude dependencies related to `org.junit.jupiter` and `org.junit.platform` from `jmock-junit5`
    omit 63529bf97ad0 [SPARK-47543][CONNECT][PYTHON][TESTS][FOLLOW-UP] Skip the test if pandas and PyArrow are unavailable
    omit d709e20066be [SPARK-47646][SQL] Make try_to_number return NULL for malformed input
    omit 609bd4839e5d [SPARK-47572][SQL] Enforce Window partitionSpec is orderable
    omit 5259dcc667df [SPARK-47644][PYTHON][DOCS] Refine docstrings of try_*
    omit 874d033fc61b [SPARK-47574][INFRA] Introduce Structured Logging Framework
    omit a8b247e9a50a [SPARK-47629][INFRA] Add `common/variant` and `connector/kinesis-asl` to maven daily test module list
    omit 0b844e52b35b [SPARK-47568][SS] Fix race condition between maintenance thread and load/commit for snapshot files
    omit b3146721552f [SPARK-47511][SQL] Canonicalize With expressions by re-assigning IDs
    omit a57f94b03e30 [SPARK-47638][PS][CONNECT] Skip column name validation in PS
    omit 41cbd56ba725 [SPARK-47525][SQL] Support subquery correlation joining on map attributes
    omit fde7a7f3d06b [SPARK-47546][SQL][FOLLOWUP] Improve validation when reading Variant from Parquet using non-vectorized reader
    omit 6cf5f9508f38 [SPARK-47623][PYTHON][CONNECT][TESTS] Use `QuietTest` in parity tests
    omit 009b50e87c05 [SPARK-47631][SQL] Remove unused `SQLConf.parquetOutputCommitterClass` method
    omit 7f1eadcd9cbb [SPARK-47366][PYTHON] Add pyspark and dataframe parse_json aliases
    omit ca0001345d0b [SPARK-47635][K8S] Use Java `21` instead of `21-jre` image in K8s Dockerfile
    omit c832e2ac1d04 [SPARK-47492][SQL] Widen whitespace rules in lexer
    omit d8dc0c3e5e8a [SPARK-47632][BUILD] Ban `com.amazonaws:aws-java-sdk-bundle` dependency
    omit 1623b2d513d2 [SPARK-47630][BUILD] Upgrade `zstd-jni` to 1.5.6-1
    omit 2e4f2b0d307f [SPARK-47475][CORE][K8S] Support `spark.kubernetes.jars.avoidDownloadSchemes` for K8s Cluster Mode
    omit 32e73dd55416 [MINOR][CORE] Replace `get+getOrElse` with `getOrElse` with default value in `StreamingQueryException`
    omit 4b58a631fea9 [SPARK-47628][SQL] Fix Postgres bit array issue 'Cannot cast to boolean'
    omit 8c4d6764674f [SPARK-47559][SQL] Codegen Support for variant `parse_json`
    omit b594c4edb383 [SPARK-47621][PYTHON][DOCS] Refine docstring of `try_sum`, `try_avg`, `avg`, `sum`, `mean`
    omit 8a6e5bb7a292 [MINOR][PYTHON][TESTS] Remove redundant parity tests
    omit 299a6597950b [SPARK-47614][CORE][DOC] Update some outdated comments about `JavaModuleOptions`
    omit 4d72be3abdc4 [SPARK-47363][SS] Initial State without state reader implementation for State API v2
    omit 67bd52ce060e [SPARK-47619][PYTHON][DOCS] Refine docstring of `to_json/from_json`
    omit 0eb96ae6eb86 [SPARK-47620][PYTHON][CONNECT] Add a helper function to sort columns
    omit f2492ddbfe9d [SPARK-47546][SQL] Improve validation when reading Variant from Parquet
    omit 031864875674 [SPARK-47543][CONNECT][PYTHON] Inferring `dict` as `MapType` from Pandas DataFrame to allow DataFrame creation
    omit e3e7135af4df [SPARK-47107][SS][PYTHON] Implement partition reader for python streaming data source
    omit 1b3e5e71f7f9 [SPARK-47485][SQL][PYTHON][CONNECT] Create column with collations in dataframe API
    omit 8d1539f7bb23 [SPARK-47616][SQL] Add User Document for Mapping Spark SQL Data Types from MySQL
    omit d10dbaa31a44 [SPARK-47564][SQL] Always throw FAILED_READ_FILE error when fail to read files
    omit 87449c3f1d65 [SPARK-47563][SQL] Add map normalization on creation
    omit d326cb9c08d9 [SPARK-46575][SQL][FOLLOWUP] Correct @since annotation for HiveThriftServer2.startWithContext(SQLContext, exitonError)
    omit d57164a5cab9 [SPARK-47430][SQL] Support GROUP BY for MapType
    omit b540cc538614 [SPARK-47611][SQL] Cleanup dead code in MySQLDialect.getCatalystType
    omit a600c0ea3159 [SPARK-47491][CORE] Add `slf4j-api` jar to the class path first before the others of `jars` directory
    omit f9eb3f3c13bf [SPARK-46575][SQL][FOLLOWUP] Add back `HiveThriftServer2.startWithContext(SQLContext)` method for compatibility
    omit 5f392a219de2 [SPARK-47610][CORE] Always set `io.netty.tryReflectionSetAccessible=true`
    omit eef44f0e8bd7 [SPARK-42040][SQL] SPJ: Introduce a new API for V2 input partition to report partition statistics
    omit 9db527ec5adc [SPARK-47562][CONNECT] Factor literal handling out of `plan.py`
    omit 88b29c5076d4 [SPARK-47570][SS] Integrate range scan encoder changes with timer implementation
    omit 5ac39181fe87 [SPARK-47273][SS][PYTHON] implement Python data stream writer interface
    omit 96b301ce4e41 [SPARK-47498][TESTS][CORE] Refine some GPU fraction calculation tests
    omit 7d87a94dd77f [MINOR][CORE] When failed to canceling the job group, add a warning log
    omit fd4b8e89f3a0 [SPARK-47555][SQL] Show a warning message about SQLException if `JDBCTableCatalog.loadTable` fails
    omit e00eace41a63 [SPARK-47561][SQL] Fix analyzer rule order issues about Alias
    omit 8e20f8e3b440 [SPARK-47544][PYTHON] SparkSession builder method is incompatible with visual studio code intellisense
    omit 89104b93d324 [SPARK-47557][SQL][TEST] Audit MySQL ENUM/SET Types
    omit ded8cdf8d945 [SPARK-47367][PYTHON][CONNECT][TESTS][FOLLOW-UP] Recover the test case for the number of partitions
    omit 6ee662c28ffb [SPARK-47431][SQL] Add session level default Collation
    omit 3e8343205c4d [SPARK-47469][SS][TESTS] Add `Trigger.AvailableNow` tests for `transformWithState` operator
    omit becbf8b94213 [SPARK-47560][PYTHON][CONNECT] Avoid RPC to validate column name with cached schema
    omit a127df47f7a2 [SPARK-47367][PYTHON][CONNECT] Support Python data sources with Spark Connect
    omit a3aa08697686 [SPARK-47358][SQL][COLLATION] Improve repeat expression support to return correct datatype
    omit ff38378d7e42 [SPARK-47372][SS] Add support for range scan based key state encoder for use with state store provider
    omit 31db27d193fb Revert "Revert "[SPARK-47450][INFRA][R] Use R 4.3.3 in `windows` R GitHub Action job""
    omit acf13d6515f7 [SPARK-47509][SQL] Block subquery expressions in lambda and higher-order functions
    omit 0054a12a013f Revert "[SPARK-47450][INFRA][R] Use R 4.3.3 in `windows` R GitHub Action job"
    omit 2a8bb5cdd3a5 [SPARK-47554][BUILD] Upgrade `sbt-assembly` to `2.2.0` and `sbt-protoc` to `1.0.7`
    omit 0340f805fdd2 [SPARK-47497][SQL][FOLLOWUP] Add a UT for `nested structure` for the function `to_csv`
    omit a5ea8848777b [SPARK-47549][BUILD] Remove Spark 3.0~3.2 `pyspark/version.py` workaround from release scripts
    omit 87cae7bc7870 [SPARK-47552][CORE] Set `spark.hadoop.fs.s3a.connection.establish.timeout` to 30s if missing
    omit 7b9b3cb9d82c [SPARK-47550][K8S][BUILD] Update `kubernetes-client` to 6.11.0
    omit 90d506573409 [SPARK-47548][BUILD] Remove unused `commons-beanutils` dependency
    omit d7db869de609 [SPARK-47256][SQL] Assign names to error classes _LEGACY_ERROR_TEMP_102[4-7]
    omit b2f6474848fc [SPARK-46743][SQL] Count bug after constant folding
    omit 1b55fd3ee107 [SPARK-47539][SQL][FOLLOWUP] Fix UT about `variant`
    omit b34178781fdd [SPARK-47539][SQL] Make the return value of method `castToString` be `Any => UTF8String`
    omit 4a1f2419e243 [SPARK-47451][SQL] Support to_json(variant).
    omit e22ddcbd852c [SPARK-47538][BUILD] Remove `commons-logging` dependency
    omit 824d67052c75 [SPARK-47537][SQL] Fix error data type mapping on MySQL Connector/J
    omit 804b2a416781 [SPARK-47506][SQL] Add support to all file source formats for collated data types
    omit d8d119a21e07 [SPARK-47536][BUILD] Upgrade `jmock-junit5` to 2.13.1
    omit b34a44f175aa [SPARK-47535][INFRA] Update `publish_snapshot.yml` to publish twice per day
    omit 99fb84b7ad27 [SPARK-47534][SQL] Move `o.a.s.variant` to `o.a.s.types.variant`
    omit 015719ee95b1 [SPARK-47516][INFRA] Move `remove unused installation package logic` from `each test job` to `create the docker image`
    omit 19e3de44ec1d [MINOR][PYTHON][DOCS] Fix a `pandas_udf` example
    omit 2ef15cabd4b4 [SPARK-44708][PYTHON] Migrate test_reset_index assert_eq to use assertDataFrameEqual
    omit 310fd6517887 [SPARK-47533][BUILD] Migrate scalafmt dialect to `scala213`
    omit 767a52d5db35 [SPARK-47528][SQL] Add UserDefinedType support to DataTypeUtils.canWrite
    omit ba175e3f6ae9 [SPARK-47526][BUILD] Upgrade `netty` to 4.1.108.Final and `netty-tcnative` to 2.0.65.Final
    omit 15ea9de88b8d [SPARK-47503][SQL] Make `makeDotNode` escape graph node name always
    omit 18bb6a3aea82 [SPARK-47497][SQL] Make `to_csv` support the output of `array/struct/map/binary` as pretty strings
    omit 063638b4d5db [SPARK-47531][BUILD] Upgrade `Arrow` to 15.0.2
    omit 11f5d3fa10b3 [SPARK-47530][BUILD][TESTS] Add `bcpkix-jdk18on` test dependencies to `hive` module for Hadoop 3.4.0
    omit 9d6b9f7305f2 [SPARK-47529][DOCS] Use hadoop 3.4.0 in some docs
    omit c29d132aeb5d [SPARK-47495][CORE] Fix primary resource jar added to spark.jars twice under k8s cluster mode
    omit b9335b90280a [SPARK-47510][INFRA] Fix `DSTREAM` label pattern in `labeler.yml`
    omit 39500a315166 [SPARK-47522][SQL][FOLLOWUP] Add float(p) values for MySQLIntegrationSuite
    omit 36126a5c1821 [SPARK-47522][SQL] Read MySQL FLOAT as FloatType to keep consistent with the write side
    omit 245669053a34 [SPARK-47521][CORE] Use `Utils.tryWithResource` during reading shuffle data from external storage
    omit 93f98c0a61dd [SPARK-47523][SQL] Replace deprecated `JsonParser#getCurrentName` with `JsonParser#currentName`
    omit d1be4fb61368 [SPARK-47517][INFRA][FOLLOWUP] Prevent `byteCountToDisplaySize` via Scalastyle
    omit 32dfdd305aec [SPARK-47517][CORE][SQL] Prefer Utils.bytesToString for size display
    omit d7be50f122ed [MINOR][DOCS] Fix typo in spark connect overview
    omit 227a50a1766a [SPARK-47440][SQL] Fix pushing unsupported syntax to MsSqlServer
    omit f41cd1706941 [SPARK-47515][SQL] Save TimestampNTZType as DATETIME in MySQL
    omit c486ed813ab5 [SPARK-47462][SQL][FOLLOWUP] Add migration guide for integral type mapping changes
    omit 105f008037e5 [MINOR][PYTHON][DOCS] Fix typos in `PandasCogroupedOps.applyInArrow`
    omit c94090e13139 [SPARK-47007][SQL] Add the `MapSort` expression
    omit 8cba15ed30ea [SPARK-47483][SQL] Add support for aggregation and join operations on arrays of collated strings
    omit ca44489f4585 [SPARK-47499][PYTHON][CONNECT][TESTS] Enable `test_help_command` test
    omit aea13fca5d57 [SPARK-47500][PYTHON][CONNECT] Factor column name handling out of `plan.py`
    omit 47bce8ececa8 [MINOR][SQL] Fix a typo in `DelegateSymlinkTextInputSplit` comment
    omit 0ef7b771b33d [SPARK-47501][SQL][FOLLOWUP] Rename convertDateToDate to convertJavaDateToDate
    omit 5042263f8668 [SPARK-47479][SQL] Optimize cannot write data to relations with multiple paths error log
    omit 9ba70d6ed302 [SPARK-47514][SQL][TESTS] Add a test coverage for createTable method (partitioned-table) in CatalogSuite
    omit ef3244c78f91 [SPARK-47513][TESTS] Regenerate benchmark results
    omit 7b575d137a1a Revert "[SPARK-46654][SQL][PYTHON] Make `to_csv` explicitly indicate that it does not support some types of data"
    omit 057acf986d78 [SPARK-47512][SS] Tag operation type used with RocksDB state store instance lock acquisition/release
    omit b4c09221b2e0 [SPARK-47502][INFRA] Make output the installation packages in `descending size`, add `titles`, and remove `unused` packages
    omit 6a27789ad7d5 [SPARK-47398][SQL] Extract a trait for InMemoryTableScanExec to allow for extending functionality
    omit 25ecde948beb [SPARK-47487][SQL] Simplify code in AnsiTypeCoercion
    omit 32f3d4dc0389 [SPARK-47501][SQL] Add convertDateToDate like the existing convertTimestampToTimestamp for JdbcDialect
    omit 2b0e841a3534 [SPARK-47496][SQL] Java SPI Support for dynamic JDBC dialect registering
    omit 3a1609af41e4 [SPARK-45393][BUILD][FOLLOWUP] Update IsolatedClientLoader fallback Hadoop version to 3.4.0
    omit 91df046e0825 [SPARK-41888][PYTHON][CONNECT][TESTS] Enable doctest for `DataFrame.observe`
    omit bffffee1bceb [SPARK-47474][CORE] Revert SPARK-47461 and add some comments
    omit 5d3845f2942a [SPARK-46920][YARN] Improve executor exit error message on YARN
    omit bb0867f54d43 [MINOR][CORE] Fix a comment typo `slf4j-to-jul` to `jul-to-slf4j`
    omit bb3e27581887 Revert "[SPARK-47007][SQL][PYTHON][R][CONNECT] Add the `map_sort` function"
    omit 11247d804cd3 [SPARK-47494][DOC] Add migration doc for the behavior change of Parquet timestamp inference since Spark 3.3
    omit b8e7d99d417a [SPARK-47490][SS] Fix RocksDB Logger constructor use to avoid deprecation warning
    omit f66274e92d1c [SPARK-47486][CONNECT] Remove unused private `ArrowDeserializers.getString` method
    omit 49b4c3bc9c09 [SPARK-45393][BUILD] Upgrade Hadoop to 3.4.0
    omit a34c8ceb19bd [SPARK-47462][SQL] Align mappings of other unsigned numeric types with TINYINT in MySQLDialect
    omit 3a3477b0f156 [SPARK-47443][SQL] Window Aggregate support for collations
    omit 76260807eef2 [SPARK-46990][SQL] Fix loading empty Avro files  emitted by event-hubs
    omit 8762e256d164 [SPARK-47296][SQL][COLLATION] Fail unsupported functions for non-binary collations
    omit a3c04ec11456 [SPARK-47478][SQL][TESTS] Improve test coverage for mysql bool synonyms
    omit df563904ccb1 [SPARK-41625][PYTHON][CONNECT][TESTS] Enable doctest `pyspark.sql.connect.dataframe.DataFrame.isStreaming`
    omit a5910a2dbb00 [SPARK-47480][PYTHON][CONNECT][TESTS] Enable doctest for `createDataFrame`
    omit ad8ac17dbdfa [SPARK-47473][SQL] Fix correctness issue of converting postgres INFINITY timestamps
    omit 85bf7615f85e [SPARK-47455][BUILD] Fix resource leak during the initialization of `scalaStyleOnCompileConfig` in `SparkBuild.scala`
    omit c3a04fa59ce1 [SPARK-47447][SQL] Allow reading Parquet TimestampLTZ as TimestampNTZ
    omit 747846bd3ef3 [SPARK-47007][SQL][PYTHON][R][CONNECT] Add the `map_sort` function
    omit 61d7b0f24fc9 [SPARK-47470][SQL][TESTS] Ignore `IntentionallyFaultyConnectionProvider` error in `CliSuite`
    omit bc378f4ff5e2 [SPARK-47330][SQL][TESTS] XML: Added XmlExpressionsSuite
    omit b9c0e9331935 [SPARK-47309][SQL] XML: Add schema inference tests for value tags
    omit c32d27850e2e [SPARK-47468][BUILD] Exclude `logback` dependency from SBT like Maven
    omit 46b46b7fd2d4 [SPARK-47454][PYTHON][CONNECT][TESTS][FOLLOWUP] Further split `pyspark.sql.tests.test_dataframe`
    omit f0061dbe856a [SPARK-47449][SS] Refactor and split list/timer unit tests
    omit 32ee2d7936a5 [SPARK-47464][INFRA] Update `labeler.yml` for module `common/sketch` and `common/variant`
    omit db531c6ee719 [SPARK-47461][CORE] Remove private function `totalRunningTasksPerResourceProfile` from `ExecutorAllocationManager`
    omit 90560dce85b0 [SPARK-47458][CORE] Fix the problem with calculating the maximum concurrent tasks for the barrier stage
    omit a6bffcc3e5f0 [SPARK-47457][SQL] Fix `IsolatedClientLoader.supportsHadoopShadedClient` to handle Hadoop 3.4+
    omit b6a836946311 [SPARK-47454][PYTHON][CONNECT][TESTS] Split `pyspark.sql.tests.test_dataframe`
    omit ef94f7094989 [SPARK-47452][INFRA] Use `Ubuntu 22.04` in `dev/infra/Dockerfile`
    omit 5e42ecc8163a [SPARK-47456][SQL] Support ORC Brotli codec
    omit 5f48931fcdf7 [SPARK-47453][SQL][DOCKER][BUILD][TESTS] Upgrade MySQL docker image version to 8.3.0
    omit 681b41f0808e [SPARK-47422][SQL] Support collated strings in array operations
    omit e01ed0da22f2 [SPARK-47345][SQL][TESTS][FOLLOW-UP] Rename JSON to XML within XmlFunctionsSuite
    omit 9f8147c2a8d2 [SPARK-47329][SS][DOCS] Add note to persist dataframe while using foreachbatch and stateful streaming query to prevent state from being re-loaded in each batch
    omit acf17fd67217 [SPARK-47450][INFRA][R] Use R 4.3.3 in `windows` R GitHub Action job
    omit cb20fcae951d [SPARK-47448][CORE] Enable `spark.shuffle.service.removeShuffle` by default
    omit 51e8634a5883 [SPARK-47380][CONNECT] Ensure on the server side that the SparkSession is the same
    omit a40940a0bc6d [SPARK-47446][CORE] Make `BlockManager` warn before `removeBlockInternal`
    omit ce93c9fd8671 [SPARK-47383][CORE] Support `spark.shutdown.timeout` config
    omit 8bd42cbdb6bf [SPARK-47435][SQL] Fix overflow issue of MySQL UNSIGNED TINYINT  caused by SPARK-45561
    omit 1aafe60b3e76 [SPARK-47442][CORE][TEST] Use port 0 to start worker servers in MasterSuite
    omit 4dc362dbc6c0 [SPARK-47438][BUILD] Upgrade jackson to 2.17.0
    omit 57424b92c5b5 [MINOR][DOCS] Add `Web UI` link to `Other Documents` section of index.md
    omit 9b466d329c3c [SPARK-47434][WEBUI] Fix `statistics` link in `StreamingQueryPage`
    omit d3f12df6e09e [SPARK-47437][PYTHON][CONNECT] Correct the error class for `DataFrame.sort*`
    omit 42c4dad62dcd [SPARK-43435][PYTHON][CONNECT][TESTS] Reenable doctest `pyspark.sql.connect.dataframe.DataFrame.writeStream`
    omit 08866c280f87 [SPARK-47439][PYTHON] Document Python Data Source API in API reference page
    omit 44a88edc995e [MINOR][TESTS] Collation - extending golden file coverage
    omit 3f171ce3f43b [SPARK-47436][PYTHON][DOCS] Fix docstring links and type hints in Python Data Source
    omit 310dd5294267 [SPARK-45891][DOCS] Update README with more details
    omit 2dba72100e03 [SPARK-47426][BUILD] Upgrade Guava used by the connect module to `33.1.0-jre`
    omit 714feb7b5aec [SPARK-47377][PYTHON][CONNECT][TESTS][FOLLOWUP] Factor out more tests from `SparkConnectSQLTestCase`
    omit 6719168b6ec7 [SPARK-47327][SQL] Move sort keys concurrency test to CollationFactorySuite
    omit 653ac5b729e2 [SPARK-47423][SQL] Collations - Set operation support for strings with collations
    omit add49b3c115f [SPARK-47346][PYTHON] Make daemon mode configurable when creating Python planner workers
    omit 7c81bdf1ed17 [SPARK-47345][SQL][TESTS] Xml functions suite
    omit e2c0471476ea [SPARK-47395] Add collate and collation to other APIs
    omit 6bf031796c8c [SPARK-44740][CONNECT][TESTS][FOLLOWUP] Deduplicate `test_metadata`
    omit 4437e6e21237 [SPARK-47419][CONNECT] Move `log4j2-defaults.properties` to `common/utils`
    omit b7aa9740249b [SPARK-47407][SQL] Support java.sql.Types.NULL map to NullType
    omit 4b1f8c3d779b [SPARK-47399][SQL] Disable generated columns on expressions with collations
    omit 19ac1fc13646 [SPARK-47406][SQL] Handle TIMESTAMP and DATETIME in MYSQLDialect
    omit 56cfc89e8f15 [SPARK-47234][BUILD] Upgrade Scala to 2.13.13
    omit 7eea705d3015 [SPARK-47366][SQL] Implement parse_json
    omit fe0aa1edff04 [SPARK-47402][BUILD] Upgrade `ZooKeeper` to 3.9.2
    omit 213399b61de5 [SPARK-47396][SQL] Add a general mapping for TIME WITHOUT TIME ZONE to TimestampNTZType
    omit 7b4ab4fa452d [SPARK-47387][SQL] Remove some unused error classes
    omit d41d5ecda8c1 [SPARK-47390][SQL][FOLLOWUP] Fix TIME_WITH_TIMEZONE mapping in PostgresDialect
    omit b98accd9d931 [SPARK-47401][K8S][DOCS] Update `YuniKorn` docs with v1.5
    omit 481597cd2d79 [SPARK-47400][BUILD] Upgrade `gcs-connector` to 2.2.20
    omit 205e826e7052 [SPARK-47384][BUILD] Upgrade RoaringBitmap to 1.0.5
    omit 168346f93303 [SPARK-47391][SQL] Remove the test case workaround for JDK 8
    omit 5ce150735bc5 [SPARK-47394][SQL] Support TIMESTAMP WITH TIME ZONE for H2Dialect
    omit e848b9665134 [SPARK-47388][SQL] Pass `messageParameters` by name to `require()`
    omit 4b073fd607f1 [SPARK-47208][CORE] Allow overriding base overhead memory
    omit 63b79c1eac01 [SPARK-47375][DOC][FOLLOWUP] Fix a mistake in JDBC's preferTimestampNTZ option doc
    omit 336801dead40 [SPARK-47390][SQL] PostgresDialect distinguishes TIMESTAMP from TIMESTAMP_TZ
    omit 49af659c9aec [SPARK-46988][CONNECT][FOLLOWUP] Protobuf `abbreviate` method code clean up
    omit 9986462811f1 [SPARK-47385] Fix tuple encoders with Option inputs
    omit e9802116b047 [SPARK-47374][CONNECT][DOCS] Fix connect-repl `usage prompt` & `docs link`
    omit 9efae37dc535 [MINOR][INFRA] Update the link text and ref in `notify_test_workflow.yml`
    omit 56416378f86d [SPARK-47377][PYTHON][CONNECT][TESTS] Factor out tests from `SparkConnectSQLTestCase`
    omit abfbd2718159 [SPARK-47375][DOC][FOLLOWUP] Correct the preferTimestampNTZ option description in JDBC doc
    omit bc0ba7dccdd5 [SPARK-41762][PYTHON][CONNECT][TESTS] Enable column name comparsion in `test_column_arithmetic_ops`
    omit 839bc9f9c264 [SPARK-46913][SS] Add support for processing/event time based timers with transformWithState operator
    omit ebe9f669e3fd [SPARK-47344] Extend INVALID_IDENTIFIER error beyond catching '-' in an unquoted identifier and fix "IS ! NULL" et al.
    omit b75325ccefa6 [SPARK-47378][PROTOBUF][TESTS] Make the related Protobuf UT run well in IDE
    omit 6ea983511b62 [SPARK-47375][SQL] Add guidelines for timestamp mapping in `JdbcDialect#getCatalystType`
    omit ccdc19d587c3 [SPARK-46654][SQL][PYTHON] Make `to_csv` explicitly indicate that it does not support some types of data
    omit e170252714e3 Revert "[SPARK-47342][SQL] Support TimestampNTZ for DB2 TIMESTAMP WITH TIME ZONE"
    omit 7d0f0832ae1a [SPARK-47373][SQL] Match FileSourceScanLike to get metadata instead of FileSourceScanExec
    omit a5d0c3778fd6 [SPARK-46761][SQL] Quoted strings in a JSON path should support ? characters
    omit c7795bb8cc82 [SPARK-47333][SQL] Use checkInputDataTypes to check the parameter types of the function `to_xml`
    omit 78531ebb8064 [SPARK-47340][SQL] Change "collate" in StringType typename to lowercase
    omit 625589f736fe [SPARK-47368][SQL] Remove inferTimestampNTZ config check in ParquetRowConverter
    omit 5d32e62436dc [MINOR][TESTS] Enable nullability check in `test_create_dataframe_from_arrays`
    omit 37d0c556fdd9 [SPARK-47348][INFRA] Align the version of the ubuntu host machine where the GitHub workflow runs on
    omit 806f8e0466b9 [SPARK-47349][SQL][TESTS] Refactor string function `startsWith` and `endsWith` tests
    omit 29e91d0e48a3 [SPARK-47272][SS] Add MapState implementation for State API v2.
    omit 621f2c88f3e5 [SPARK-47370][DOC] Add migration doc: TimestampNTZ type inference on Parquet files
    omit 8fcef1657a02 [SPARK-47364][CORE] Make `PluginEndpoint` warn when plugins reply for one-way message
    omit efc1aedce6d7 [SPARK-46331][SQL][FOLLOWUP] Time travel should support current datetime functions
    omit 6e5d1db9058d [SPARK-46919][BUILD][CONNECT] Upgrade `grpcio*` and `grpc-java` to 1.62.x
    omit e8bc176e6fd1 [SPARK-47342][SQL] Support TimestampNTZ for DB2 TIMESTAMP WITH TIME ZONE
    omit 72ecba538406 [SPARK-47335][BUILD] Upgrade `mvn-scalafmt` to `1.1.1684076452.9f83818` & `scalafmt` to `3.8.0`
    omit 6c30525e0d71 [SPARK-47343][SQL] Fix NPE when `sqlString` variable value is null string in execute immediate
    omit f8a49030fd68 [SPARK-47295][SQL] Added ICU StringSearch for the `startsWith` and `endsWith` functions
    omit fc61abd89cbf [SPARK-47169][SQL] Disable bucketing on collated columns
    omit 1f58f4c68e8d [SPARK-47254][SQL] Assign names to the error classes _LEGACY_ERROR_TEMP_325[1-9]
    omit 95aec974d8e7 [SPARK-47347][PYTHON][CONNECT][TESTS] Factor session-related tests out of `test_connect_basic`
    omit e1865811cac9 [MINOR] Minor English fixes
    omit e778ce689dcb [SPARK-47250][SS] Add additional validations and NERF changes for RocksDB state provider and use of column families
    omit f53dc08230b7 [SPARK-46962][SS][PYTHON] Add interface for python streaming data source API and implement python worker to run python streaming data source
    omit 09bdf2a9334f [SPARK-46043][SQL][FOLLOWUP] do not resolve v2 table provider with custom session catalog
    omit fe5998538596 [MINOR][DOCS] Remove the extra text on page sql-error-conditions-sqlstates
    omit 63c01c8036e8 [SPARK-47309][SQL][XML] Add schema inference unit tests
    omit f20b28e1005c [SPARK-47341][CONNECT] Replace commands with relations in a few tests in SparkConnectClientSuite
    omit c1b9f28682d8 [SPARK-47327][SQL] Fix thread safety issue in ICU Collator
    omit f40c693ad7fd [SPARK-47339][BUILD] Upgrade checkStyle to `10.14.0`
    omit f97da1638062 [SPARK-45245][CONNECT][TESTS][FOLLOW-UP] Remove unneeded Matchers trait in the test
    omit 610840e27e2e [SPARK-45827][SQL][FOLLOWUP] Fix for collation
    omit 10be03215775 [SPARK-47255][SQL] Assign names to the error classes _LEGACY_ERROR_TEMP_323[6-7] and _LEGACY_ERROR_TEMP_324[7-9]
    omit 4b0d1085a782 [SPARK-47313][SQL] Added scala.MatchError handling inside QueryExecution.toInternalError
    omit 427f74066d57 [SPARK-47328][SQL] Rename UCS_BASIC collation to UTF8_BINARY
    omit d5f42fc6f186 [SPARK-47337][SQL][DOCKER] Upgrade DB2 docker image version to 11.5.8.0
    omit f350b760ea11 [SPARK-47270][SQL] Dataset.isEmpty projects CommandResults locally
    omit 2a51242b1bdb [MINOR][DOCS][SQL] Fix doc comment for coalescePartitions.parallelismFirst on sql-performance-tuning page
    omit 66d6a23dc3b7 [SPARK-45827][SQL] Add variant singleton type for Java
    omit 72a95bcad7f1 [SPARK-47324][SQL] Add missing timestamp conversion for JDBC nested types
    omit afbebfbadc4b [SPARK-47331][SS] Serialization using case classes/primitives/POJO based on SQL encoder for Arbitrary State API v2
    omit 5ac560c76e60 [MINOR][DOCS][SQL] Fix doc comment for coalescePartitions.parallelismFirst
    omit f6df78154bac [SPARK-47035][SS][CONNECT][FOLLOWUP] Additional protobuf change to client side listener
    omit a4603f134fb0 [SPARK-47334][SQL] Make `withColumnRenamed` reuse the implementation of `withColumnsRenamed`
    omit f6a00f2dda93 [SPARK-47325][INFRA] Use the latest `buf-setup-action` in github workflow
    omit edb970b8a73e [SPARK-47200][SS] Make Foreach batch sink user function error handling backward compatible
    omit ae518ecb7068 [SPARK-47218][SQL] XML: Changed SchemaOfXml to fail on DROPMALFORMED mode
    omit 264e00ee12bb [MINOR][SQL][TEST] Moving tests to related suites
    omit 76b1c122cb7d [SPARK-44115][BUILD] Upgrade Apache ORC to 2.0.0
    omit 640ed4fad32f [SPARK-47322][PYTHON][CONNECT] Make `withColumnsRenamed` column names duplication handling consistent with `withColumnRenamed`
    omit 0a94e7c7b9a8 [SPARK-47302][SQL] Collate keyword as identifier
    omit 7a3b3c612d16 [SPARK-47316][SQL] Fix TimestampNTZ in Postgres Array
    omit 00de3ed5e0aa [MINOR][DOCS] Fix the doc for Integral types
    omit 50131054b818 [SPARK-47265][SQL][TESTS][COLUMN] Replace `createTable(..., schema: StructType, ...)` with `createTable(..., columns: Array, ...)` in UT
    omit c0a8cf9cca08 [SPARK-47079][PYTHON][DOCS][FOLLOWUP] Add `VariantType` to API references
    omit f659f8d1b019 [SPARK-47319][SQL] Improve missingInput calculation
    omit 54f15721649d [SPARK-47314][DOC] Remove the  wrong comment line of `ExternalSorter#writePartitionedMapOutput` method
    omit 703d1d8e4be2 [MINOR][INFRA] Make "y/n" consistent within merge script
    omit 9cac2bb608c6 [SPARK-47309][SQL][XML] Fix schema inference issues in XML
    omit 7a5bb5d83551 [SPARK-47078][DOCS][PYTHON] Documentation for SparkSession-based Profilers
    omit 501999a834ea [SPARK-47276][PYTHON][CONNECT] Introduce `spark.profile.clear` for SparkSession-based profiling
    omit 0f9164201ff0 [SPARK-45827][SQL] Move data type checks to CreatableRelationProvider
    omit 456d246badb8 [SPARK-47248][SQL][COLLATION] Improved string function support: contains
    omit 7a429aa84a5e [MINOR][DOCS][PYTHON] Fix documentation typo in takeSample method
    omit 3112762e88b9 [SPARK-47278][BUILD] Upgrade rocksdbjni to 8.11.3
    omit 620297ab6d8f [SPARK-47298][BUILD] Upgrade `mysql-connector-j` to `8.3.0` and `mariadb-java-client` to `2.7.12`
    omit 9b743ffb577a [SPARK-47300][SQL] `quoteIfNeeded` should quote identifier starts with digits
    omit 0ff22b0019f1 [SPARK-47301][SQL][TESTS] Fix flaky ParquetIOSuite
    omit 51f4cfa7560b [SPARK-47241][SQL] Fix rule order issues for ExtractGenerator
    omit 9f28c43a127d [SPARK-47315][SQL][TEST] Clean up tempView for `createTempView` UT
    omit 15d7755fd1cb [SPARK-47155][PYTHON] Fix Error Class Issue
    omit 8d6bd9bbd29d [SPARK-47305][SQL] Fix PruneFilters to tag the isStreaming flag of LocalRelation correctly when the plan has both batch and streaming
    omit 3ec79a9df06f [SPARK-47238][SQL] Reduce executor memory usage by making generated code in WSCG a broadcast variable
    omit e341e26330ba [SPARK-47311][SQL][PYTHON] Suppress Python exceptions where PySpark is not in the Python path
    omit 0c28df40e049 [SPARK-46988][CONNECT][TESTS] Add tests for `map` type in `ProtoUtils.abbreviate`
    omit c6538301c0a8 [SPARK-47070][SQL][FOLLOW-UP] Add a flag guarding a subquery in aggregate rewrite
    omit d9881bd59d05 [SPARK-47271][DOCS] Explain importance of statistics on SQL performance tuning page
    omit 88599ea6568f [SPARK-47303][CORE][TESTS] Restructure MasterSuite
    omit d705d868aaaa [SPARK-39771][CORE] Add a warning msg in `Dependency` when a too large number of shuffle blocks is to be created
    omit fe2174d8caaf [SPARK-47304][SQL][TESTS] Distribute tests from `DataFrameSuite` to more specific suites
    omit 5089140e2e6a [SPARK-47146][CORE][FOLLOWUP] Rename incorrect logger name
    omit 58761623dcdd [SPARK-47293][CORE] Build batchSchema with sparkSchema instead of append one by one
    omit d5f35ec97fc4 [SPARK-46835][SQL][COLLATIONS] Join support for non-binary collations
    omit d7cc708e1e0d [SPARK-47280][SQL] Remove timezone limitation for ORACLE TIMESTAMP WITH TIMEZONE
    omit e310e76e63fe [SPARK-47247][SQL] Use smaller target size when coalescing partitions with exploding joins
    omit 36299f1b2bff [SPARK-47285][SQL] AdaptiveSparkPlanExec should always use the context.session
    omit e0d79108cacc [SPARK-44746][PYTHON] Add more Python UDTF documentation for functions that accept input tables
    omit 474310347802 [SPARK-47277] PySpark util function assertDataFrameEqual should not support streaming DF
    omit c88f0310621b [SPARK-47251][PYTHON][FOLLOWUP] Use __name__ instead of string representation
    omit c417c757907e [SPARK-47033][SQL] Fix EXECUTE IMMEDIATE USING does not recognize session variable names
    omit 502fa8070c65 [SPARK-47168][SQL] Disable parquet filter pushdown when working with non default collated strings
    omit 6534a3398ae9 [SPARK-47102][SQL] Add the `COLLATION_ENABLED` config flag
    omit c9435b8b4864 [SPARK-47176][SQL][FOLLOW-UP] resolveExpressions should have three versions which is the same as resolveOperators
    omit dfb35bed522c [SPARK-47146][CORE] Possible thread leak when doing sort merge join
    omit 6e62a5690b81 [SPARK-47177][SQL] Cached SQL plan do not display final AQE plan in explain string
    omit 73aa14466683 [SPARK-47252][DOCS] Clarify that pivot may trigger an eager computation
    omit 35bced42474e [SPARK-47242][BUILD] Bump ap-loader 3.0(v8) to support for async-profiler 3.0
    omit db0e5c7bc464 [SPARK-47269][BUILD] Upgrade jetty to 11.0.20
    omit 6b5917beff30 [SPARK-46961][SS] Using ProcessorContext to store and retrieve handle
    omit 0b2eca28c307 [SPARK-47245][SQL] Improve error code for INVALID_PARTITION_COLUMN_DATA_TYPE
    omit ca7c60b49988 [SPARK-47268][SQL][COLLATIONS] Support for repartition with collations
    omit 7e7f3ff9e281 [SPARK-47070] Fix invalid aggregation after subquery rewrite
    omit 479954cf73a5 [SPARK-47131][SQL][COLLATION] String function support: contains, startswith, endswith
    omit b711efd6671a [SPARK-43258][SQL] Assign names to error _LEGACY_ERROR_TEMP_202[3,5]
    omit a1e57b32e495 [SPARK-47266][CONNECT] Make `ProtoUtils.abbreviate` return the same type as the input
    omit 69743ad98f34 [SPARK-46094][CORE][FOLLOW-UP] Mark `writing` as volatile at ExecutorJVMProfiler
    omit 5a71f2184207 [SPARK-46973][SQL] Skip V2 table lookup when a table is in V1 table cache
    omit 71767cfe3765 [SPARK-46834][SQL][COLLATIONS] Support for aggregates
    omit 433c9b064a3f [SPARK-47246][SQL] Replace `InternalRow.fromSeq` with `new GenericInternalRow` to save a collection conversion
    omit 3aa16e8193cf [MINOR] Update outdated comments for class `o.a.s.s.functions`
    omit 0ea9d9e71be7 [SPARK-47249][CONNECT] Fix bug where all connect executions are considered abandoned regardless of their actual status
    omit ee15cbd3ac3c [SPARK-47244][CONNECT] `SparkConnectPlanner` make internal functions private
    omit 2d934de64ebc [SPARK-47251][PYTHON] Block invalid types from the `args` argument for `sql` command
    omit e46d7d1c60ab [SPARK-47158][SQL] Assign proper name and `sqlState` to `_LEGACY_ERROR_TEMP_(2134|2231)`
    omit 1135f6b4c60c [SPARK-47237][BUILD] Upgrade xmlschema-core to 2.3.1
    omit 8cda48370c1e [SPARK-47227][FOLLOW][DOCS] Improve Spark Connect Documentation
    omit 0b9387e888da [SPARK-47216][DOCS] Refine layout of SQL performance tuning page
    omit e822775f1f07 [SPARK-47243][SS] Correct the package name of `StateMetadataSource.scala`
    omit 2981618d5aad [SPARK-47167][SQL] Add concrete class for JDBC anonymous relation
    omit a5a282234295 [SPARK-43255][SQL] Replace the error class _LEGACY_ERROR_TEMP_2020 by an internal error
    omit a993f9dcd56f [SPARK-42627][SPARK-26494][SQL] Support Oracle TIMESTAMP WITH LOCAL TIME ZONE
    omit 22f9a5a25304 [SPARK-47235][CORE][TESTS] Disable `deleteRecursivelyUsingUnixNative` in Apple Silicon test env
    omit 1cd7bab5c5c2 [SPARK-47236][CORE] Fix `deleteRecursivelyUsingJavaIO` to skip non-existing file input
    omit cc0ea60d6eee [SPARK-47218][SQL] XML: Ignore commented row tags in XML tokenizer
    omit 32102bcbec87 [SPARK-47211][CONNECT][PYTHON] Fix ignored PySpark Connect string collation
    omit 7447172ffd5c [SPARK-47200][SS] Error class for Foreach batch sink user function error
    omit 9ce43c85a5d2 [SPARK-47229][CORE][SQL][SS][YARN][CONNECT] Change the never changed `var` to `val`
    omit 70007c59177a [SPARK-47186][DOCKER][FOLLOWUP] Reduce test time for docker ITs
    omit 813934c69df6 [SPARK-47015][SQL] Disable partitioning on collated columns
    omit 28fd3de0fea0 [SPARK-41392][BUILD][TESTS] Add `bouncy-castle` test dependencies to `sql/core` module for Hadoop 3.4.0
    omit 919c19c008b8 [SPARK-47231][CORE][TESTS] FakeTask should reference its TaskMetrics to avoid TaskMetrics accumulators being GCed before stage completion
    omit a7825f6e8907 [SPARK-47227][DOCS] Improve documentation for Spark Connect
    omit 944a00db6f83 [SPARK-47224][PS][TESTS] Split `test_split_apply_basic` and `test_split_apply_adv`
    omit b0a027ca4f72 [SPARK-47221][SQL] Uses signatures from CsvParser to AbstractParser
    omit a8b6e3c906c9 [MINOR][PYTHON][DOCS] Clarify verifySchema at createDataFrame not working with pandas DataFrame with Arrow optimization
    omit 11e8ae42af92 [SPARK-47202][PYTHON][TESTS][FOLLOW-UP] Run the test only with Python 3.9+
    omit fd54d7107e45 [MINOR][DOCS] Remove extraneous whitespace from SQL tuning page
    omit f028d3eba314 [SPARK-47214][PYTHON] Create UDTF API for 'analyze' method to differentiate constant NULL arguments and other types of arguments
    omit 36df0a63a139 [SPARK-47135][SS] Implement error classes for Kafka data loss exceptions
    omit da2ae29c9cab [SPARK-47215][CORE][TESTS] Reduce the number of required threads in `MasterSuite`
    omit 2a93e46eb062 [SPARK-47207][CORE] Support `spark.driver.timeout` and `DriverTimeoutPlugin`
    omit ea2587f695cf [SPARK-47209][BUILD] Upgrade slf4j to 2.0.12
    omit 7e7ba4eaf071 [MINOR][SQL] Remove out-of-dated comment in `CollectLimitExec`
    omit c8d293dff595 [SPARK-47147][PYTHON][SQL] Fix PySpark collated string conversion error
    omit 5c4b0d1c0483 [SPARK-47145][SQL] Pass table identifier to row data source scan exec for V2 strategy
    omit 4cf90e3d468c [SPARK-47205][BUILD][TESTS] Upgrade docker-java to 3.3.5
    omit 7d2435197841 [SPARK-46077][SQL] Consider the type generated by TimestampNTZConverter in JdbcDialect.compileValue
    omit 34026d1f54ed [SPARK-42929][CONNECT][PYTHON][TEST] test barrier mode for mapInPandas/mapInArrow
    omit 721c2a41a54b [SPARK-47202][PYTHON][TESTS][FOLLOW-UP] Test timestamp with tzinfo in toPandas and createDataFrame with Arrow optimized
    omit 6eed08cb0c12 [SPARK-47199][PYTHON][TESTS] Add prefix into TemporaryDirectory to avoid flakiness
    omit 78bfa5c83d9c [SPARK-47203][DOCKER][TEST] Use gvenzl/oracle-free:23.3-slim to reduce disk usage for docker it
    omit ca7012d81b3e [SPARK-47192] Convert some _LEGACY_ERROR_TEMP_0035 errors
    omit 6d41866e6314 [SPARK-47191][SQL] Avoid unnecessary relation lookup when uncaching table/view
    omit 537a2d9d3ac8 [SPARK-46525][BUILD][TESTS][FOLLOWUP] Cleanup http client deps for spotify docker client
    omit bf8396eaef9f [SPARK-47202][PYTHON] Fix typo breaking datetimes with tzinfo
    omit 6385cf1bf155 [SPARK-47144][CONNECT][SQL][PYTHON] Fix Spark Connect collation error by adding collateId protobuf field
    omit f18d945af7b6 [SPARK-47063][SQL] CAST long to timestamp has different behavior for codegen vs interpreted
    omit 5d69388c8283 [SPARK-47201][PYTHON][CONNECT] sameSemantics` checks input types
    omit 999abb5671ce [SPARK-47186][DOCKER][TESTS] Add some timeouts options and logs to improve the debuggability for docker integration test
    omit 5b2da4865622 [SPARK-45527][SPARK-47185][SS][TESTS][FOLLOW-UP] Increase timeout more, and reduce the resource usage
    omit f0090c95ad4e [SPARK-43256][SQL] Remove error class _LEGACY_ERROR_TEMP_2021
    omit ba773fe9a286 [SPARK-47185][SS][TESTS] Increase timeout between actions in KafkaContinuousSourceSuite
    omit c773aed87d64 [SPARK-47189][SQL] Tweak column error names and text
    omit 9d2fd5e328fd [SPARK-47179][SQL] Improve error message from SparkThrowableSuite for better debuggability
    omit cdae4c6bb194 [SPARK-47183][PYTHON] Fix the error class for `sameSemantics`
    omit a778e6c79cc4 [SPARK-41873][PYTHON][CONNECT][TESTS] Enable `DataFrameParityTests.test_pandas_api`
    omit 1ca92181af8d [SPARK-47184][PYTHON][CONNECT][TESTS] Make `test_repartitionByRange_dataframe` reusable
    omit 209d0fcf22b1 [SPARK-47182][BUILD] Exclude `commons-(io|lang3)` transitive dependencies from `commons-compress` and `avro*`
    omit 7234e35130c2 [SPARK-47176][SQL] Have a ResolveAllExpressionsUpWithPruning helper function
    omit ccf6eb276de7 [MINOR][SS] Minor string representation improvement in AssertOnQuery
    omit 1a408033daf4 [SPARK-47181][CORE][TESTS] Fix `MasterSuite` to validate the number of registered workers
    omit 031b90b2ac0b [SPARK-47178][PYTHON][TESTS] Add a test case for createDataFrame with dataclasses
    omit 611ed06fd20b [SPARK-47166][INFRA] Improves the HINTs of merge_spark_pr.py
    omit afa9f9679bc0 [SPARK-45527][CORE][TESTS][FOLLOW-UP] Reduce the number of test cases in fraction resource calculation
    omit 1081bd12693f [SPARK-45527][CORE][TESTS][FOLLOW-UP] Reduce the number of test cases in fraction resource calculation
    omit 15c27fac9818 [SPARK-47175][SS][TESTS] Remove ZOOKEEPER-1844 comment from `KafkaTestUtils`
    omit cbe150532547 [SPARK-47165][SQL][TESTS] Pull docker image only when its' absent
    omit 866779d7cbd8 [SPARK-47164][SQL] Make Default Value From Wider Type Narrow Literal of v2 behave the same as v1
    omit 76c4fd56c5a5 [SPARK-45527][CORE][TESTS][FOLLOWUP] Reduce the number of threads from 1k to 100 in `TaskSchedulerImplSuite`
    omit f5d38001808a [SPARK-47079][PYTHON][SQL][CONNECT] Add Variant type info to PySpark
    omit e2dcb0274f13 [SPARK-47173][SS][UI] Fix a typo in streaming UI explanation
    omit 0ea318f3e23f [SPARK-46947][CORE] Delay memory manager initialization until Driver plugin is loaded
    omit a939a7d0fd9c [SPARK-47170][BUILD][CONNECT] Remove `jakarta.servlet-api` and `javax.servlet-api` dependency scope in `connect/server` module
    omit 298134fd5e98 [SPARK-47009][SQL] Enable create table support for collation
    omit fcc5dbc9b6c8 [SPARK-45599][CORE] Use object equality in OpenHashSet
    omit 242dd2e819b4 [SPARK-47163][BUILD] Fix `make-distribution.sh` to check `jackson-core-asl-1.9.13.jar` existence first
    omit 5c5b47c5ad51 [SPARK-47160][K8S] Update K8s `Dockerfile` to include `hive-jackson` directory if exists
    omit 1ca60d353792 [SPARK-47161][INFRA][R] Uses hash key properly for SparkR build on Windows
    omit 449de3597fae [SPARK-47159][INFRA] Set `OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES` in `MacOS` GitHub Action Job
    omit 0ff18e579c2f [SPARK-46802][PYTHON][TESTS][FOLLOWUP] Remove obsolete comment in run-tests-with-coverage
    omit 679b468854ab [MINOR][CONNECT][TESTS] Chain waitFor after destroyForcibly in SparkConnectServerUtils
    omit aa566ff3b9fd [SPARK-47120][SQL] Null comparison push down data filter from subquery produces in NPE in Parquet filter
    omit 759123409f0e [SPARK-47137][PYTHON][CONNECT][FOLLOW-UP] Uses assertEqual instead of assertEquals for Python 3.12 build
    omit e0facc32f439 [SPARK-47157][SQL] Refactor file listing with ScanFileListing interface
    omit 18b86068ff4c [SPARK-47154][SS][TESTS] Fix `kafka-0-10-sql` to use `ResetSystemProperties` if `KafkaTestUtils` is used
    omit dd3f81c3d610 [SPARK-47152][SQL][BUILD] Provide `CodeHaus Jackson` dependencies via a new optional directory
    omit ee312ecb40ea [SPARK-47151][PYTHON][PS][BUILD] Upgrade to `pandas` 2.2.1
    omit c2dbb6d04bc9 [SPARK-47099][SQL][FOLLOWUP] Regenerate `try_arithmetic.sql.out.java21`
    omit 772a445e412b [SPARK-47035][SS][CONNECT] Protocol for Client-Side Listener
    omit 28951ed6681f [SPARK-47118][BUILD][CORE][SQL][UI] Migrate from Jetty 10 to Jetty 11
    omit d20650bc8cf2 [SPARK-46975][PS] Support dedicated fallback methods
    omit 06c741a0061b [SPARK-47129][CONNECT][SQL] Make `ResolveRelations` cache connect plan properly
    omit 3baa60afe25c [SPARK-44914][BUILD] Upgrade Apache Ivy to 2.5.2
    omit c2079b9b13e4 [SPARK-47140][SPARK47139][INFRA][PYTHON] Upgrade Python verion and codecov action in Coverage job
    omit 09739294ba1d [SPARK-47143][CONNECT][TESTS] Improve `ArtifactSuite` to use unique `MavenCoordinate`s
    omit d466c0beabcf [SPARK-47142][K8S][TESTS] Use `spark.jars.ivy` instead `spark.driver.extraJavaOptions` in `DepsTestsSuite`
    omit a053b40ac0e9 [SPARK-47099][SQL][FOLLOW-UP] Uses ordinalNumber in UNEXPECTED_INPUT_TYPE
    omit 511839b6eac9 [SPARK-47137][PYTHON][CONNECT] Add getAll to spark.conf for feature parity with Scala
    omit b90514c37755 [SPARK-47130][CORE] Use listStatus to bypass block location info when cleaning driver logs
    omit 6ae0abb64289 [SPARK-43259][SQL][FOLLOWUP] Regenerate `sql-error-conditions.md` to recover `SparkThrowableSuite`
    omit 07226e6a8086 [SPARK-47123][CORE] JDBCRDD does not correctly handle errors in getQueryOutputSchema
    omit 241a95def31d Revert "[SPARK-47115][INFRA] Use larger memory for Maven builds"
    omit 9bc273ee0dad [SPARK-47136][CORE][TESTS] Fix `MavenUtilsSuite` to use `MavenUtils.resolveMavenCoordinates` properly
    omit 490467ff55bc Revert "[SPARK-47115][INFRA][FOLLOW-UP] Use larger runner for Maven build (macos-14-large)"
    omit 9debaeaa5a07 [SPARK-47069][PYTHON][CONNECT] Introduce `spark.profile.show/dump` for SparkSession-based profiling
    omit 6185e5cad7be [SPARK-47132][DOCS][PYTHON] Correct docstring for pyspark's dataframe.head
    omit 6de527e9ee94 [SPARK-43259][SQL] Assign a name to the error class _LEGACY_ERROR_TEMP_2024
    omit 94d0d96ac30a [SPARK-47127][INFRA][FOLLOWUP] Remove `3.5.1` from `SKIP_SPARK_RELEASE_VERSIONS`
    omit 78f7c30e140f [SPARK-42328][SQL] Remove _LEGACY_ERROR_TEMP_1175 from error classes
    omit df4d489b67a9 [SPARK-47127][INFRA] Update `SKIP_SPARK_RELEASE_VERSIONS` in Maven CIs
    omit a87015efb5cf [SPARK-47125][SQL] Return null if Univocity never triggers parsing
    omit 65fe9efa40a3 [SPARK-47101][SQL] Allow comma to be used in top-level column names and remove check nested type definition in `HiveExternalCatalog.verifyDataSchema`
    omit 9b6c84f0340f [SPARK-47124][R][INFRA] Skip scheduled SparkR on Windows in fork repositories by default
    omit b564082cc2ed [SPARK-47115][INFRA][FOLLOW-UP] Use larger runner for Maven build (macos-14-large)
    omit e081f06ea401 [SPARK-47121][CORE] Avoid RejectedExecutionExceptions during StandaloneSchedulerBackend shutdown
    omit 32415cf643d2 [SPARK-47122][INFRA] Pin `buf-setup-action` to `v1.29.0`
    omit d92e66643e8d [SPARK-47104][SQL] `TakeOrderedAndProjectExec` should initialize the unsafe projection
    omit 5e375c8ef323 [SPARK-47119][BUILD] Add `hive-jackson-provided` profile
    omit af28aebfa17c [SPARK-47036][SS] Cleanup RocksDB file tracking for previously uploaded files if files were deleted from local directory
    omit 9b53b8039980 [SPARK-47099][SQL] Use `ordinalNumber` to uniformly set the value of `paramIndex` for the error class `UNEXPECTED_INPUT_TYPE`
    omit f60ae4dbc1d3 [SPARK-46938][BUILD] Remove `javax-servlet-api` exclusion rule for SBT
    omit b93fd9c074bd [SPARK-47116][INFRA][R] Install proper Python version in SparkR Windows build to avoid warnings
    omit 0d938de112df [SPARK-47112][INFRA] Write logs into a file in SparkR Windows build
    omit b3e34629080b [SPARK-47113][CORE] Revert S3A endpoint fixup logic of SPARK-35878
    omit f4bb6528596b [SPARK-47115][INFRA] Use larger memory for Maven builds
    omit 011b843687d8 Revert "[SPARK-35878][CORE] Revert S3A endpoint fixup logic of SPARK-35878"
    omit 36f199d1e412 [SPARK-35878][CORE] Revert S3A endpoint fixup logic of SPARK-35878
    omit 0b907ed11e6e [SPARK-46928][SS] Add support for ListState in Arbitrary State API v2
    omit b96914b637ee [SPARK-47095][INFRA][FOLLOW-UP] Remove TTY specific workaround in Maven build
    omit bffa92c838d6 [SPARK-47052][SS] Separate state tracking variables from MicroBatchExecution/StreamExecution
    omit 6e6197073a57 [SPARK-47111][SQL][TESTS] Upgrade `PostgreSQL` JDBC driver to 42.7.2 and docker image to 16.2
    omit 434acaf48849 [SPARK-47109][BUILD] Upgrade `commons-compress` to 1.26.0
    omit 76575ee7481c [MINOR][SQL] Remove `toLowerCase(Locale.ROOT)` for `CATALOG_IMPLEMENTATION`
    omit 8ede494ad6c7 [SPARK-46906][SS] Add a check for stateful operator change for streaming
    omit eb71268a6a38 [SPARK-47108][CORE] Set `derby.connection.requireAuthentication` to `false` explicitly in CLIs
    omit 9d9675922543 [SPARK-45615][BUILD] Remove undated "Auto-application to `()` is deprecated" compile suppression rules
    omit 6a5649410d83 [SPARK-47098][INFRA] Migrate from AppVeyor to GitHub Actions for SparkR tests on Windows
    omit 8e82887ed25e [SPARK-46858][PYTHON][PS][BUILD] Upgrade Pandas to 2.2.0
    omit e6a3385e27fa [SPARK-47044][SQL] Add executed query for JDBC external datasources to explain output
    omit fb1e7872a3e6 [SPARK-47100][BUILD] Upgrade `netty` to 4.1.107.Final and `netty-tcnative` to 2.0.62.Final
    omit f0f35c8b1c8f [SPARK-45789][SQL] Support DESCRIBE TABLE for clustering columns
    omit c8e8e4090027 [SPARK-47080][CORE][TESTS] Fix `HistoryServerSuite` to use `constant value` and `getNumJobsRestful`
    omit bc066ac30284 [SPARK-47016][BUILD] Upgrade scalatest related dependencies to the 3.2.18 series
    omit b0aad59f1235 [SPARK-47097][CONNECT][TESTS] Deflake interrupt tag at SparkSessionE2ESuite
    omit 79d6182aa423 [SPARK-47093][TESTS] Upgrade `mockito` to 5.10.0
    omit 9389c6b00f5f [SPARK-46820][PYTHON] Fix error message regression by restoring `new_msg`
    omit aa53c8e18453 [SPARK-47062][CONNECT] Move Connect Plugins to Java for Compatibility
    omit 6dda3258533f [SPARK-47096][INFRA] Upgrade Python version in Maven builds
    omit 24d17e04d984 [SPARK-37434][TESTS] Disable unsupported `ExtendedLevelDBTest` on `MacOS/aarch64`
    omit 6e40d5d0035b [SPARK-47095][INFRA] Uses proper options for command script in macos-14 build
    omit ab117b5f292f [SPARK-47092][CORE][SQL][K8S] Add `getUriBuilder` to `o.a.s.u.Utils` and use it
    omit 581b6a747923 [SPARK-46934][SQL] Read/write roundtrip for struct type with special characters with HMS
    omit 9f582d616845 [SPARK-44826][TEST][CONNECT] Re-enable `test_series_iloc_setitem`
    omit 07e22811bfcb [SPARK-47089][BUILD][TESTS] Migrate `mockito 4` to `mockito 5`
    omit 5841260890a0 [SPARK-47067][INFRA] Add Daily Apple Silicon Github Action Job with Maven (Java/Scala)
    omit 9ec46a691a88 [SPARK-47087][SQL] Raise Spark's exception with an error class in config value check
    omit 63a5acd2431a [SPARK-46972][SQL][TESTS][FOLLOWUP] Remove the assertion for table existence
    omit c4e4497ff7e7 [SPARK-46812][SQL][PYTHON] Make mapInPandas / mapInArrow support ResourceProfile
    omit 081809667611 [MINOR][SQL] Remove `unsupportedOperationMsg` from `CaseInsensitiveStringMap`
    omit 91dfc31ff89c [SPARK-47085][SQL] reduce the complexity of toTRowSet from n^2 to n
    omit 174a19c1c039 [MINOR][INFRA] Rename build_maven.yml and build_maven_java21.yml
    omit 8a9a1be9b02a [SPARK-47066][INFRA][FOLLOW-UP] Deduplicate Apple Silicon Maven build definition
    omit b61a00c26f0b [SPARK-47083][BUILD] Upgrade `commons-codec` to 1.16.1
    omit 85108b0cb605 [SPARK-47084][BUILD] Upgrade joda-time to 2.12.7
    omit 7c60fe21a29d [MINOR][INFRA][DOCS] Remove undated comment in build_and_test.yml
    omit 63e8a1422059 [SPARK-47082][SQL] Fix out-of-bounds error condition
    omit 8f5a647b0bbb [MINOR][DOCS] Show sort order of NaN relative to infinity
    omit e02ee42b1930 Revert "[SPARK-47067][INFRA] Add Daily Apple Silicon Github Action Job (Java/Scala)"
    omit d4235963309e [SPARK-47057][PYTHON] Reeanble MyPy data test
    omit dc2f2673a73c [SPARK-42285][DOC] Update Parquet data source doc on the timestamp_ntz inference option
    omit 2a7e8900ab5f [SPARK-47076][CORE][TESTS] Fix HistoryServerSuite.`incomplete apps get refreshed` test to start with empty storeDir
    omit 5faa512bf8c7 [SPARK-47075][BUILD] Add `derby-provided` profile
    omit 6d30983bcc6a [SPARK-47074][INFRA] Fix outdated comments in GitHub Action scripts
    omit 73da134849b7 [SPARK-47040][CONNECT][FOLLOWUP] Improve usability of the start-conect-server-script.sh
    omit 2abd3a2f445e [SPARK-47073][BUILD] Upgrade several Maven plugins to the latest versions
    omit 074fcf280700 [SPARK-47072][SQL] Fix supported interval formats in error messages
    omit 64fa13b5744f [SPARK-47060][SQL][TESTS] Check `SparkIllegalArgumentException` instead of `IllegalArgumentException` in `catalyst`
    omit 61e25e1cdcbb [SPARK-47071][SQL] Inline With expression if it contains special expression
    omit 86f15e4a779e [SPARK-47054][PYTHON][TESTS] Remove pinned version of torch for Python 3.12 support
    omit 3bb762dc0328 [SPARK-47068][PYTHON][TESTS] Recover -1 and 0 case for spark.sql.execution.arrow.maxRecordsPerBatch
    omit 3a146ce46c55 [SPARK-46078][PYTHON][TESTS] Upgrade `pytorch` for Python 3.12
    omit 7668eb5daf22 [SPARK-47055][PYTHON] Upgrade MyPy 1.8.0
    omit 964545aa88e9 [SPARK-47067][INFRA] Add Daily Apple Silicon Github Action Job (Java/Scala)
    omit 86a387737eb6 [SPARK-47066][INFRA] Add `Apple Silicon` Maven build test to GitHub Action CI
    omit 638a68f8e566 [SPARK-47058][TESTS] Add `scalastyle` and `checkstyle` rules to ban `AtomicDoubleArray|CompoundOrdering`
    omit 135bdeb2ba10 [SPARK-47064][SQL][TESTS] Use Scala 2.13 Spark distribution in `HiveExternalCatalogVersionsSuite`
    omit e2cd71a4cd54 [SPARK-47059][SQL] Attach error context for ALTER COLUMN v1 command
    omit edf4ac4b518d [SPARK-47056][TESTS] Add `scalastyle` and `checkstyle` rules to ban `FileBackedOutputStream`
    omit 9ef552691e1d [SPARK-47045][SQL] Replace `IllegalArgumentException` by `SparkIllegalArgumentException` in `sql/api`
    omit d72efc038124 [SPARK-46687][TESTS][PYTHON][FOLLOW-UP] Skip MemoryProfilerParityTests when codecov enabled
    omit 1b48de5606fb [SPARK-47051][INFRA] Create a new test pipeline for `yarn` and `connect`
    omit 736d8ab3f00e [SPARK-47049][BUILD] Ban non-shaded Hadoop dependencies
    omit 4b9e9d7a9b7c [SPARK-47014][PYTHON][CONNECT] Implement methods dumpPerfProfiles and dumpMemoryProfiles of SparkSession
    omit c1321c01eeea [SPARK-47038][BUILD] Remove shaded `protobuf-java` 2.6.1 dependency from `kinesis-asl-assembly`
    omit 7e911cdd0344 [SPARK-47039][TESTS] Add a checkstyle rule to ban `commons-lang` in Java code
    omit 861cca3da4c4 [SPARK-46832][SQL] Introducing Collate and Collation expressions
    omit a6bed5e9bcc5 [SPARK-47040][CONNECT] Allow Spark Connect Server Script to wait
    omit 63b97c6ad82a [SPARK-46979][SS] Add support for specifying key and value encoder separately and also for each col family in RocksDB state store provider
    omit 49bcde612c59 [SPARK-47028][SQL][TESTS] Check `SparkUnsupportedOperationException` instead of `UnsupportedOperationException`
    omit 33a153a6bbcb [SPARK-47030][TESTS] Add `WebBrowserTest`
    omit d260f5753e9d [SPARK-47025][BUILD][TESTS] Upgrade `Guava` dependency in `docker-integration-tests` test module
    omit 7acc1563696a [SPARK-47004][CONNECT] Added more tests to ClientStreamingQuerySuite to increase Scala client test coverage
    omit 5da73d32e36a [SPARK-47026][SQL][TESTS] Enable JSON sources in default value nested type tests
    omit 615c423d363e [MINOR][SQL] Show only number of test blocks when there is a mismatch
    omit fb7760621c33 [SPARK-46944] Follow up to SPARK-46792 (ChannelBuilder refactoring): Fix minor typing oversight
    omit 24a9d25358f7 [SPARK-47027][PYTHON][TESTS] Use temporary directories for profiler test outputs
    omit b425cd866334 [SPARK-47023][BUILD] Upgrade `aircompressor` to 1.26
    omit f5b1e37c9e6a [SPARK-44445][BUILD][TESTS] Use `org.seleniumhq.selenium.htmlunit3-driver` instead of `net.sourceforge.htmlunit`
    omit f5b0de07eff4 [SPARK-46991][SQL] Replace `IllegalArgumentException` by `SparkIllegalArgumentException` in `catalyst`
    omit a926c7912a78 [SPARK-47021][BUILD][TESTS] Fix `kvstore` module to have explicit `commons-lang3` test dependency
    omit fa23d276e7e4 [SPARK-47020][CORE][TESTS] Fix `RealBrowserUISeleniumSuite`
    omit f80a83e98668 [MINOR][DOCS] Remove outdated `antlr4` version comment in `pom.xml`
    omit d179f7564541 [SPARK-46355][SQL][TESTS][FOLLOWUP] Test to check number of open files
    omit 3f5faaa24e3a [SPARK-46641][SS] Add maxBytesPerTrigger threshold
    omit 8603ed58a34d [SPARK-46831][INFRA][FOLLOWUP] Fix a wrong JIRA ID in MimaExcludes
    omit 6569f15490ec [SPARK-47002][PYTHON] Return better error message if UDTF 'analyze' method 'orderBy' field accidentally returns a list of strings
    omit a5e741e60ac9 [SPARK-47011][MLLIB] Remove deprecated `BinaryClassificationMetrics.scoreLabelsWeight`
    omit 1a66c8c78a46 [SPARK-46689][SPARK-46690][PYTHON][CONNECT] Support v2 profiling in group/cogroup applyInPandas/applyInArrow
    omit b7edc5fac0f4 [SPARK-39910][SQL] Delegate path qualification to filesystem during DataSource file path globbing
    omit b345b23b76c4 [SPARK-46999][SQL] ExpressionWithUnresolvedIdentifier should include other expressions in the expression tree
    omit 3d25525d876b [SPARK-46993][SQL] Fix constant folding for session variables
    omit 71b76dc3e66a [SPARK-43117][CONNECT] Make `ProtoUtils.abbreviate` support repeated fields
    omit 729fc8ec95e0 [SPARK-46615][CONNECT] Support s.c.immutable.ArraySeq in ArrowDeserializers
    omit 338bb31c2fac [SPARK-46997][CORE] Enable `spark.worker.cleanup.enabled` by default
    omit ab7fcacca41d [MINOR][PYTHON][SQL][TESTS] Don't load Python Data Source when Python executable is not available even for testing
    omit 45956f72d864 [SPARK-47005][PYTHON][DOCS] Refine docstring of `asc_nulls_first/asc_nulls_last/desc_nulls_first/desc_nulls_last`
    omit abf8770ffac7 [SPARK-46994][PYTHON] Refactor PythonWrite to prepare for supporting python data source streaming write
    omit becfb94e1c71 [SPARK-46865][SS] Add Batch Support for TransformWithState Operator
    omit a8ad71dbe417 [SPARK-46998][SQL] Deprecate the SQL config `spark.sql.legacy.allowZeroIndexInFormatString`
    omit cdd41a9f2c4f [SPARK-47003][K8S] Detect and fail on invalid volume sizes (< 1KiB) in K8s
    omit d8e402db2aa7 [SPARK-47000][CORE] Use `getTotalMemorySize` in `WorkerArguments`
    omit 9bd0d7c3ee13 [SPARK-46688][SPARK-46691][PYTHON][CONNECT] Support v2 profiling in aggregate Pandas UDFs
    omit 31f85e5ff77c [SPARK-46966][PYTHON] Add UDTF API for 'analyze' method to indicate subset of input table columns to select
    omit 578931678f5a [SPARK-46922][CORE][SQL] Do not wrap runtime user-facing errors
    omit becc04a9b6e2 [SPARK-46996][SQL] Allow AQE coalesce final stage in SQL cached plan
    omit dc73a8d7e96e [SPARK-46526][SQL] Support LIMIT over correlated subqueries where predicates only reference outer table
    omit a95aa7a7dda1 [SPARK-46987][CONNECT] `ProtoUtils.abbreviate` avoid unnecessary `setField` operation
    omit 0bed57325de5 [SPARK-46984][PYTHON] Remove pyspark.copy_func
    omit 5d5b3a54b7b5 [SPARK-46895][CORE] Replace Timer with single thread scheduled executor
    omit cfbf3c704f0f [SPARK-46976][PS] Implement `DataFrameGroupBy.corr`
    omit 10439902b6dd [SPARK-46954][SQL] XML: Wrap InputStreamReader with BufferedReader
    omit 1b7afa7afdb8 [SPARK-46958][SQL] Add missing timezone to coerce default values
    omit 614a0c7d0ce3 [MINOR][DOCS] Clarify docs on default fractional numeric literals in SQL
    omit 5100b2e5d1aa [SPARK-46960][SS] Testing Multiple Input Streams with TransformWithState operator
    omit 1a577b8f8881 [SPARK-46980][SQL][MINOR] Avoid using internal APIs in dataframe end-to-end tests
    omit 1e13243ca394 [SPARK-46849][SQL][FOLLOWUP] Column default value cannot reference session variables
    omit 006c2dca6d87 [SPARK-46977][CORE] A failed request to obtain a token from one NameNode should not skip subsequent token requests
    omit f8ff18223b36 [SPARK-46972][SQL] Fix asymmetrical replacement for char/varchar in V2SessionCatalog.createTable
    omit 2e917fb91924 [SPARK-46978][PYTHON][DOCS] Refine docstring of `sum_distinct/array_agg/count_if`
    omit e9697130a1ac [MINOR][TEST] Add output/exception to error message when schema not matched in `TPCDSQueryTestSuite`
    omit 6ff5f396632d [SPARK-46833][SQL] Collations - Introducing CollationFactory which provides comparison and hashing rules for supported collations
    omit 4797e52f6766 Revert "[MINOR][DOCS] The default fractional numeric literal in SQL is a decimal"
    omit b95f6d691cb5 [MINOR][DOCS] The default fractional numeric literal in SQL is a decimal
    omit 1e13d8d8accb [SPARK-46974][SQL][TEST] Recover a test case for day-of-year 2-letter 'DD' pattern parsing 3-digit values
    omit 69239729f35e [SPARK-46954][SQL] XML: Optimize schema index lookup
    omit 07f1341fac6f [MINOR][DOCS] Fix various broken links and link anchors
    omit d31c6a8eadf0 [MINOR][DOCS] Clean markup in NULL semantics documentation
    omit da73c123e648 [MINOR][DOCS] Add Missing space in `docs/configuration.md`
    omit 30190edb2df7 [SPARK-46955][PS] Implement `Frame.to_stata`
    omit ef3bc41336d7 [SPARK-46512][CORE] Optimize shuffle reading when both sort and combine are used
    omit ed6fe4fccabe [SPARK-46899][CORE][FOLLOWUP] Enable `/workers/kill` if `spark.decommission.enabled=true`
    omit 7ca355cbc225 [SPARK-46970][CORE] Rewrite `OpenHashSet#hasher` with `pattern matching`
    omit 062522e96a50 [SPARK-46967][CORE][UI] Hide `Thread Dump` and `Heap Histogram` of `Dead` executors in `Executors` UI
    omit fd476c1c855a [SPARK-46969][SQL][TESTS] Recover `to_timestamp('366', 'DD')` test case of `datetime-parsing-invalid.sql`
    omit 0154c059cddb [MINOR][DOCS] Remove Java 8/11 at `IgnoreUnrecognizedVMOptions` description
    omit f4525ff978a7 [SPARK-45276][INFRA][FOLLOWUP] Fix Java version comment from 11 to 17
    omit 9d4d41c43f1c [SPARK-46760][SQL][DOCS] Make the document of spark.sql.adaptive.coalescePartitions.parallelismFirst clearer
    omit 6e60b232c769 [SPARK-46968][SQL] Replace `UnsupportedOperationException` by `SparkUnsupportedOperationException` in `sql`
    omit 88d4681dd6ab [SPARK-46950][CORE][SQL] Align `not available codec` error-class
    omit 16ac82092bb7 [SPARK-46890][SQL] Fix CSV parsing bug with existence default values and column pruning
    omit 84387394c387 [SPARK-46965][CORE] Check `logType` in `Utils.getLog`
    omit 0965412d5174 [SPARK-46964][SQL] Change the signature of the hllInvalidLgK query execution error to take an integer as 4th argument
    omit e35e29a0517d [SPARK-46915][SQL] Simplify `UnaryMinus` `Abs` and align error class
    omit 25d96f7bacb4 [SPARK-46911][SS] Adding deleteIfExists operator to StatefulProcessorHandleImpl
    omit 6f87fe2f513d [SPARK-44815][CONNECT] Cache df.schema to avoid extra RPC
    omit 22e5938aefc7 [SPARK-46946][SQL] Supporting broadcast of multiple filtering keys in DynamicPruning
    omit 362a4d43159d [SPARK-46949][SQL] Support CHAR/VARCHAR through ResolveDefaultColumns
    omit a3432428e760 [SPARK-46953][TEST] Wrap withTable for a test in ResolveDefaultColumnsSuite
    omit 2e07662e24e2 [SPARK-46952][SQL] XML: Limit size of corrupt record
    omit 58c9b5ac8ff4 [SPARK-46683][SQL][TESTS][FOLLOW-UP] Fix typo, use queries in partition set
    omit c426b7285b58 [SPARK-46941][SQL] Can't insert window group limit node for top-k computation if contains SizeBasedWindowFunction
    omit 47fb5041b014 [MINOR][DOCS] Fix outgoing links from SELECT SQL reference page
    omit 457f6f59b1b7 [SPARK-46945][K8S] Add `spark.kubernetes.legacy.useReadWriteOnceAccessMode` for old K8s clusters
    omit cedef63faf14 [SPARK-45807][SQL] Return View after calling replaceView(..)
    omit 49e3f5e3bad4 [SPARK-46908][SQL] Support star clause in WHERE clause
    omit 53cbaeb20293 [SPARK-46936][PS] Implement `Frame.to_feather`
    omit 00e63d63f9af [SPARK-46864][SS] Onboard Arbitrary StateV2 onto New Error Class Framework
    omit 704e9a0785f4 [MINOR][SQL] Clean up outdated comments from `hash` function in `Metadata`
    omit d5ca61692c34 [SPARK-46940][CORE] Remove unused `updateSparkConfigFromProperties` and `isAbsoluteURI` in `o.a.s.u.Utils`
    omit 22586efb3cd4 [SPARK-46933][SQL] Add query execution time metric to connectors which use JDBCRDD
    omit e610d1d8f79b [SPARK-46852][SS] Remove use of explicit key encoder and pass it implicitly to the operator for transformWithState operator
    omit 1870de0b329a [SPARK-45110][BUILD] Upgrade rocksdbjni to 8.8.1
    omit 223afea9960c [SPARK-46473][SQL] Reuse `getPartitionedFile` method
    omit c63dea8f4235 [SPARK-46882][SS][TEST] Replace unnecessary AtomicInteger with int
    omit 88f121c47778 [SPARK-46931][PS] Implement `{Frame, Series}.to_hdf`
    omit d49265a170fb [SPARK-46930][SQL] Add support for a custom prefix for Union type fields in Avro
    omit 262ed5bcab0b [SPARK-46929][CORE][CONNECT][SS] Use ThreadUtils.shutdown to close thread pools
    omit f2a471e9cc75 [SPARK-46400][CORE][SQL] When there are corrupted files in the local maven repo, skip this cache and try again
    omit 6c19bf6b48e7 [SPARK-45522][BUILD][CORE][SQL][UI] Migrate from Jetty 9 to Jetty 10
    omit 0c7770f4de56 [MINOR][SQL] Use `DecimalType.MINIMUM_ADJUSTED_SCALE` instead of magic number `6` in `Divide` class
    omit e95e820ac137 [SPARK-46932] Clean up the imports in `pyspark.pandas.test_*`
    omit 0871a6f15623 [SPARK-46753][PYTHON][TESTS] Fix pypy3 python test
    omit 5d87ac6214bb [SPARK-46926][PS] Add `convert_dtypes`, `infer_objects` and `set_axis` in fallback list
    omit 8f406212aa09 [SPARK-46924][CORE] Fix `Load New` button in `Master/HistoryServer` Log UI
    omit 2f917c32feeb [SPARK-46927][PYTHON] Make `assertDataFrameEqual` work properly without PyArrow
    omit 4f0a9df534a4 [SPARK-46925][PYTHON][CONNECT] Add a warning that instructs to install memory_profiler for memory profiling
    omit ecdb38e2d059 [SPARK-46923][DOCS] Limit width of configuration tables
    omit 8e29c0d8c5cd [SPARK-46921][BUILD] Move `ProblemFilters` that do not belong to `defaultExcludes` to `v40excludes`
    omit 031df8fa6266 [SPARK-46747][SQL] Avoid scan in getTableExistsQuery for JDBC Dialects
    omit e32a8249c6dd [SPARK-46918][YARN] Replace self-defined variables with Hadoop ContainerExitStatus
    omit 4da7c3d316e3 [SPARK-46905][SQL] Add dedicated class to keep column definition instead of StructField in Create/ReplaceTable command
    omit abd9d27e87b9 [SPARK-46893][UI] Remove inline scripts from UI descriptions
    omit 41a1426e9ee3 [SPARK-46914][UI] Shorten app name in the summary table on the History Page
    omit e3143c4c2806 [SPARK-46916][PS][TESTS] Clean up `pyspark.pandas.tests.indexes.*`
    omit 3f7994217d5a [SPARK-46736][PROTOBUF] retain empty message field in protobuf connector
    omit 83fad32f1c68 [SPARK-46910][PYTHON] Eliminate JDK Requirement in PySpark Installation
    omit 29355c07580e [SPARK-46907][CORE] Show driver log location in Spark History Server
    omit 528ac8b3e854 [SPARK-46687][PYTHON][CONNECT] Basic support of SparkSession-based memory profiler
    omit c468c3d5c685 [SPARK-46904][UI] Fix display issue of History UI summary
    omit e211dbdee42c [SPARK-46831][SQL] Collations - Extending StringType and PhysicalStringType with collationId field
    omit 8dd395b2eabd [SPARK-46903][CORE] Support Spark History Server Log UI
    omit 1386b52f3eb6 [SPARK-46902][UI] Fix Spark History Server UI for using un-exported setAppLimit
    omit a368280708dd [SPARK-46900][BUILD] Upgrade slf4j to 2.0.11
    omit 487cbc086a30 [SPARK-46901][PYTHON] Upgrade `pyarrow` to 15.0.0
    omit 90e6c0cf2ca1 [SPARK-46721][CORE][TESTS] Make gpu fraction tests more robust
    omit 112f4acb6283 [MINOR][DOCS] Remove Canonicalize in docs
    omit 95a4abd5b5bc [SPARK-46899][CORE] Remove `POST` APIs from `MasterWebUI` when `spark.ui.killEnabled` is `false`
    omit 56633e697571 [SPARK-46898][CONNECT] Simplify the protobuf function transformation in Planner
    omit 5056a17919ac [SPARK-46897][PYTHON][DOCS] Refine docstring of `bit_and/bit_or/bit_xor`
    omit 8cd0d1854da0 [SPARK-46896][PS][TESTS] Clean up the imports in `pyspark.pandas.tests.{frame, series, groupby}.*`
    omit bb2195554e6d [SPARK-46874][PYTHON] Remove `pyspark.pandas` dependency from `assertDataFrameEqual`
    omit f078998df2f3 [MINOR][DOCS] Miscellaneous documentation improvements
    omit 901850cab748 [MINOR][DOCS] Remove unneeded comments from global.html
    omit 02c945d6ab61 [SPARK-46889][CORE] Validate `spark.master.ui.decommission.allow.mode` setting
    omit 89d86e617da2 [SPARK-46873][SS] Do not recreate new StreamingQueryManager for the same Spark Session
    omit d74aecd11dcd [SPARK-46892][BUILD] Upgrade dropwizard metrics 4.2.25
    omit 20b593811dc0 [SPARK-46888][CORE] Fix `Master` to reject `/workers/kill/` requests if decommission is disabled
    omit a96f399d5094 [SPARK-46883][CORE][FOLLOWUP] Fix `clusterutilization` API to handle 0 worker case
    omit 1787a5261e87 [SPARK-46887][DOCS] Document a few missed `spark.ui.*` configs to `Configuration` page
    omit c8d116bcfde9 [SPARK-46886][CORE] Enable `spark.ui.prometheus.enabled` by default
    omit f75c7a7b5240 [SPARK-46883][CORE] Support `/json/clusterutilization` API
    omit bc51c9fea364 [SPARK-46862][SQL][FOLLOWUP] Fix column pruning without schema enforcing in V1 CSV datasource
    omit 6d29c72f6c2b [SPARK-46875][SQL] When the `mode` is null, a `NullPointException` should `not` be thrown
    omit ecdacf8e14c8 [SPARK-46881][CORE] Support `spark.deploy.workerSelectionPolicy`
    omit e014248434ac [SPARK-46880][PYTHON][CONNECT][TESTS] Improve and test warning for Arrow-optimized Python UDF
    omit 7ab7509fa124 [SPARK-46849][SQL] Run optimizer on CREATE TABLE column defaults
    omit 40574bb36647 [SPARK-46819][CORE] Move error categories and states into JSON
    omit 829e742df825 [SPARK-46862][SQL] Disable CSV column pruning in the multi-line mode
    omit f9f413e5ff6a [SPARK-46871][PS][TESTS] Clean up the imports in `pyspark.pandas.tests.computation.*`
    omit 1eedb7507ae2 [SPARK-46872][CORE] Recover `log-view.js` to be non-module
    omit d6fc06bd4515 [SPARK-46870][CORE] Support Spark Master Log UI
    omit de71884273a3 [MINOR][INFRA] Update the location for error class README.md
    omit 48cd8604f953 [SPARK-46868][CORE] Support Spark Worker Log UI
    omit 545489a60048 [SPARK-46863][DOCS] Cleanup custom CSS
    omit 44b163d281b9 [SPARK-46432][BUILD] Upgrade Netty to 4.1.106.Final
    omit 25df5dad6761 [SPARK-46869][K8S] Add `logrotate` to Spark docker files
    omit 79918028b142 [SPARK-46867][PYTHON][CONNECT][TESTS] Remove unnecessary dependency from test_mixed_udf_and_sql.py
    omit 617014cc92d9 [SPARK-46861][CORE] Avoid Deadlock in DAGScheduler
    omit 1bee07e39f1b [SPARK-46855][INFRA] Add `sketch` to the dependencies of the `catalyst` in `module.py`
    omit 43adfa070a40 [SPARK-46850][SQL] Convert `_LEGACY_ERROR_TEMP_2102 ` to `UNSUPPORTED_DATATYPE`
    omit 8e1fa5616068 [SPARK-46620][PS][CONNECT] Introduce a basic fallback mechanism for frame methods
    omit 95ea2a6dd403 [SPARK-46851][DOCS] Remove `buf` version information from the doc `contributing.rst`
    omit 38a517de8e22 [SPARK-46856][PS][TESTS] Apply approximate equality in ewm tests
    omit c3872cd4f1b7 [SPARK-46818][SQL] Improve error messages for Range with non-foldable inputs
    omit 985417c9f206 [SPARK-46829][CORE] Remove createExecutorEnv from SparkEnv
    omit 289543eba5bd [SPARK-46816][SS] Add base support for new arbitrary state management operator, single valueState type, multiple state variables and underlying support for column families for RocksDBStateStoreProvider with/without changelog checkpointing
    omit 51cdf34226ed [SPARK-46828][SQL] Remove the invalid assertion of remote mode for spark sql shell
    omit df4370c0759b [SPARK-46808][PYTHON][FOLLOW-UP] Add a guide for automatic sorting in `test_error_classes_sorted`
    omit d3a8b303c5c0 [SPARK-46787][CONNECT] `bloomFilter` function should throw `AnalysisException` for invalid input
    omit b86053c430e5 [SPARK-46846][CORE] Make `WorkerResourceInfo` extend `Serializable` explicitly
    omit 3928b549dbe6 [SPARK-46792][PYTHON][CONNECT] Refactor ChannelBuilder into DefaultChannelBuilder and ChannelBuilder
    omit 1f23edfa84aa [SPARK-46827][CORE] Make `RocksDBPersistenceEngine` to support a symbolic link
    omit f25ebe52b9b8 [SPARK-46796][SS] Ensure the correct remote files (mentioned in metadata.zip) are used on RocksDB version load
    omit 74b6301f152d [SPARK-46826][INFRA] Reset `grpcio` installation version of `Python linter dependencies for branch-3.4/branch-3.5`
    omit ea0436752fe5 [SPARK-45593][BUILD][FOLLOWUP] Correct relocation connect guava dependency
    omit 1642e928478c [SPARK-46823][CONNECT][PYTHON] `LocalDataToArrowConversion` should check the nullability
    omit 7004dd9edcad [SPARK-46825][DOCS] Build Spark only once when building docs
    omit ee6ed43aa046 [SPARK-46733][CORE] Simplify the BlockManager by the exit operation only depend on interrupt thread
    omit fe4f8eac3efe [SPARK-46590][SQL][FOLLOWUP] Update CoalesceShufflePartitions comments
    omit 0a470430c81c [SPARK-46822][SQL] Respect spark.sql.legacy.charVarcharAsString when casting jdbc type to catalyst type in jdbc
    omit 6ea2094da252 [SPARK-46811][FOLLOWUP] Increase check_run response result from 30 to 100
    omit 2ae7575b9ed1 [SPARK-46530][PYTHON][SQL][FOLLOW-UP] Avoid checking Py4J and PySpark library when initial lookup of Python Data Sources
    omit bfcc5e0135a0 [SPARK-46811][INFRA] Fix Github Action: TypeError: Cannot read properties of undefined (reading 'head_sha')
    omit 20bb7c0b8584 Revert "[SPARK-46530][PYTHON][SQL][FOLLOW-UP] Uses path separator instead of file separator to correctly check PySpark library existence"
    omit 9caa378fe3e1 [SPARK-43403][UI] Ensure old SparkUI in HistoryServer has been detached before loading new one
    omit a43f93d76c9d [SPARK-46728][PYTHON] Check Pandas installation properly
    omit 00a92d328576 [SPARK-46817][CORE] Fix `spark-daemon.sh` usage by adding `decommission` command
    omit d1fbc4c7191a [SPARK-46764][DOCS] Reorganize script to build API docs
    omit 0356ac009472 [SPARK-40876][SQL] Widening type promotion from integers to decimal in Parquet vectorized reader
    omit fa60a7e216e6 [SPARK-46795][SQL] Replace `UnsupportedOperationException` by `SparkUnsupportedOperationException` in `sql/core`
    omit d26e871136e0 [SPARK-46794][SQL] Remove subqueries from LogicalRDD constraints
    omit 8ab69992584a [SPARK-46772][SQL][TESTS] Benchmarking Avro with Compression Codecs
    omit bc889c8c2adb [SPARK-46683] Write a subquery generator that generates subqueries permutations to increase testing coverage
    omit 2aed25b15dba Revert "[SPARK-46417][SQL] Do not fail when calling hive.getTable and throwException is false"
    omit 715b43428913 [SPARK-46763] Fix assertion failure in ReplaceDeduplicateWithAggregate for duplicate attributes
    omit de0c4ad3947f [SPARK-46590][SQL] Fix coalesce failed with unexpected partition indeces
    omit eeed19158ea5 [SPARK-46808][PYTHON] Refine error classes in Python with automatic sorting function
    omit e7fb0ad68f73 [SPARK-46807][DOCS] Add automation notice to generated SQL error class docs
    omit 8a7609f1cb2d [SPARK-46718][BUILD] Upgrade Arrow to 15.0.0
    omit 34beb02827ff [SPARK-46805][BUILD] Upgrade `scalafmt` to 3.7.17
    omit ae2d43f279d5 [SPARK-46806][PYTHON] Improve error message for spark.table when argument type is wrong
    omit 31ffb2d99900 [SPARK-46800][CORE] Support `spark.deploy.spreadOutDrivers`
    omit 00a9b94d1827 [SPARK-46804][DOCS][TESTS] Recover the generated documents
    omit 227611d70d52 [SPARK-46802][PYTHON][TESTS] Clean up obsolete code in PySpark coverage script
    omit 52b62921cadb [SPARK-46801][PYTHON][TESTS] Do not treat exit code 5 as a test failure in Python testing script
    omit f9feddfbc9de [SPARK-46799][CORE][TESTS] Improve `MasterSuite` to use nanoTime-based appIDs and workerIDs
    omit 8d1212837538 [SPARK-46797][CORE] Rename `spark.deploy.spreadOut` to `spark.deploy.spreadOutApps`
    omit b1b29d9eeb76 [SPARK-46781][PYTHON][TESTS] Test custom data source and input partition (pyspark.sql.datasource)
    omit 0a68e6ef1c54 [MINOR][DOCS] Miscellaneous link and anchor fixes
    omit 3f3cbae8680e [MINOR][DOCS] Fix SQL Error links and link anchors
    omit b80e8cb45522 [SPARK-46779][SQL] `InMemoryRelation` instances of the same cached plan should be semantically equivalent
    omit 77062021c5d0 [SPARK-46758][INFRA] Upgrade github cache action to v4
    omit 667c0a9dbbe0 [SPARK-46791][SQL] Support Java Set in JavaTypeInference
    omit 02533d71806e [SPARK-46777][SS] Refactor `StreamingDataSourceV2Relation` catalyst structure to be more on-par with the batch version
    omit b07bdea3616f [SPARK-46789][K8S][TESTS] Add `VolumeSuite` to K8s IT
    omit 9556da8834b0 [SPARK-46673][PYTHON][DOCS] Refine docstring `aes_encrypt/aes_decrypt/try_aes_decrypt`
    omit 498519ee6bb4 Revert "[SPARK-46219][SQL] Unwrap cast in join predicates"
    omit aa4944f99566 [SPARK-46788][DOCS] Add instructions for running Docker integration tests
    omit 1f68dc73f69f [SPARK-46775][DOCS] Fix formatting of Kinesis docs
    omit 0721c108588e [MINOR][DOCS] Fix broken ML paper links
    omit 7eec8be37a87 [MINOR][DOCS] Adjust formatting of "Building Spark" docs
    omit 26d3eca0a8d3 [SPARK-46009][SQL][DOCS][FOLLOWUP] Change the error code for invalid inverse distribution function
    omit 15c9ec7cbbbb [SPARK-42332][SQL] Changing the require to a SparkException in ComplexTypeMergingExpression
    omit f7f424d19c59 [SPARK-46725][SQL] Add DAYNAME function
    omit 45ec74415a4a [SPARK-46786][K8S] Fix `MountVolumesFeatureStep` to use `ReadWriteOncePod` instead of `ReadWriteOnce`
    omit d114e262ba29 [SPARK-44495][INFRA][K8S] Use the latest minikube in K8s IT
    omit e4e40762ca41 [SPARK-46769][SQL] Refine timestamp related schema inference
    omit b7356b6682b0 [SPARK-46783][K8S][TESTS] Use `built-in` storage classes in PVTestsSuite
    omit 7b4fd90aa5fd [SPARK-46784][K8S][TESTS] Create and use a K8s test tag for `PersistentVolume`
    omit 0a0a131e987c [SPARK-46755][K8S][INFRA][DOCS] Upgrade `Volcano` to 1.8.2
    omit dceb8bdc72ef [SPARK-46767][PYTHON][DOCS] Refine docstring of `abs/acos/acosh`
    omit 689ab0ee8ca8 [SPARK-45827] Disallow partitioning on Variant column
    omit 56730f6390a1 [SPARK-46731][SS] Manage state store provider instance by state data source - reader
    omit 3d400d333050 [SPARK-46780][K8S][TESTS] Improve SBT K8s IT to skip R image build step if not needed
    omit 39f8e1a5953b [SPARK-46773][BUILD][CONNECT] Change to use include-list to `generate assemblyExcludedJars` for the connect server module
    omit 3d395a6b874b [SPARK-46774][SQL][AVRO] Use mapreduce.output.fileoutputformat.compress instead of deprecated mapred.output.compress in Avro write jobs
    omit c4af64ec0dbb [SPARK-46766][SQL][AVRO] ZSTD Buffer Pool Support For AVRO datasource
    omit a09b726f880c [SPARK-46768][BUILD] Upgrade Guava used by the connect module to 33.0-jre
    omit a569fca2e9b4 [SPARK-46770][K8S][TESTS] Remove legacy `docker-for-desktop` logic
    omit 36da27fae56a [SPARK-46765][PYTHON][CONNECT] Make `shuffle` specify the datatype of `seed`
    omit c1ed3e60e67f [SPARK-46676][SS] dropDuplicatesWithinWatermark should not fail on canonicalization of the plan
    omit b5f592bdb1a1 [MINOR][DOCS] Remove unneeded production tag block method
    omit c0939fa1f5c6 [SPARK-46576][SQL][FOLLOWUP] Improve error messages for unsupported data source save mode
    omit 01bb1b1a3dbf [SPARK-46759][SQL][AVRO] Codec xz and zstandard support compression level for avro files
    omit c040824fd75c [MINOR][DOCS] Add zstandard as a candidate to fix the desc of spark.sql.avro.compression.codec
    omit b01b88ad68a2 [SPARK-46629] Fix for STRUCT type DDL not picking up nullability and comment
    omit e2aca4f792e2 [SPARK-40546][DOCS] Remove unused TODO
    omit 62956c92cfc7 [SPARK-46707][SQL] Added throwable field to expressions to improve predicate pushdown
    omit 38fc127a232e [SPARK-46696][CORE] In ResourceProfileManager, function calls should occur after variable declarations
    omit bede6141a639 [SPARK-46623][CORE][MLLIB][SQL] Replace SimpleDateFormat with DateTimeFormatter
    omit 461026c62ba1 [SPARK-46754][SQL][AVRO] Fix compression code resolution in avro table definition and write options
    omit 72e3f06c7d94 [SPARK-46618][SQL] Improve error messages for DATA_SOURCE_NOT_FOUND error
    omit 92035e3a983b [SPARK-46757][INFRA] Reduce the number of layers of testing dockerfile
    omit da0c31c629ca [SPARK-46745][INFRA] Purge pip cache in dockerfile
    omit 44d2c86e71fc [SPARK-45593][BUILD] Building a runnable distribution from master code running spark-sql raise error
    omit 89727bfa7529 [SPARK-46644] Change add and merge in SQLMetric to use isZero
    omit 55bc4cbf4a73 [SPARK-46752][SQL][TESTS] Use default ORC compression in data source benchmarks
    omit 751a81b23bd3 [SPARK-46751][PYTHON][TESTS] Skip test_datasource if PyArrow is not installed
    omit 4d7b1ba986bc [SPARK-46748][CORE] Remove `*slave*.sh` scripts
    omit be1e8d8792ed [SPARK-46749][DOCS] Document `SPARK_LOG_*` and `SPARK_PID_DIR`
    omit 764cb3e07364 [SPARK-46750][CONNECT][PYTHON] DataFrame APIs code clean up
    omit d8703dd1c7bb [SPARK-46686][PYTHON][CONNECT] Basic support of SparkSession based Python UDF profiler
    omit 9a2f39318e3a [SPARK-46746][SQL][AVRO] Attach codec extension to avro datasource files
    omit e3fd64b96765 [SPARK-46525][DOCKER][TESTS][FOLLOWUP] Fix docker-integration-tests on Apple Silicon for db2 and oracle with third-party docker environments
    omit bccbdb72b004 [SPARK-46739][SQL] Add the error class `UNSUPPORTED_CALL`
    omit 4a69ce68172d [SPARK-46734][INFRA] Combine pip installations for lint and doc
    omit f36c3f1d7892 [SPARK-46742][SQL][TESTS] Add ORC compression tests for `hive` module `OrcFileFormat`
    omit 962d54fb8fab [SPARK-46737][SQL][TESTS] Use the default ORC compression in OrcReadBenchmark
    omit 931eb938c70d [SPARK-46612][SQL] Do not convert array type string retrieved from jdbc driver
    omit 4ea8ae0105e5 [SPARK-46474][INFRA] Upgrade upload-artifact action to v4
    omit d968986704ad [SPARK-46732][CONNECT] Make Subquery/Broadcast thread work with Connect's artifact management
    omit 3960c57b4c1b [SPARK-46730][PYTHON][DOCS] Refine docstring of `str_to_map/map_filter/map_zip_with`
    omit 6c4747cdfb02 [SPARK-46735][PYTHON][TESTS] `pyspark.sql.tests.test_group` should skip Pandas/PyArrow tests if not available
    omit 48152b1779a5 [SPARK-46663][PYTHON] Disable memory profiler for pandas UDFs with iterators
    omit ebd2b78f87fa [SPARK-46727][SQL] Port `classifyException()` in JDBC dialects on error classes
    omit 872fbc33dc27 [SPARK-46677][CONNECT][FOLLOWUP] Fix `dataset.col("*")` in Scala Client
    omit bfb0f016817d [SPARK-46677][CONNECT][FOLLOWUP] Convert `count(df["*"])` to `count(1)` on client side
    omit 48f6c435b228 [SPARK-46729][DOCS] Withdraw the recommendation of using Concurrent Mark Sweep (CMS) Garbage Collector
    omit a58362ecbbf7 [SPARK-46722][CONNECT] Add a test regarding to backward compatibility check for StreamingQueryListener in Spark Connect (Scala/PySpark)
    omit 293b2a3d51da [SPARK-46670][PYTHON][SQL][FOLLOW-UP] Do not pass static Python Data Sources around when cloning DataSourceManager
    omit cd3fa2f6b937 [SPARK-46677][SQL][CONNECT] Fix `dataframe["*"]` resolution
    omit f5d7aef1b184 [SPARK-46723][CONNECT][SCALA] Make addArtifact retryable
    omit de79388923ee [SPARK-46717][CORE] Simplify `ReloadingX509TrustManager` by the exit operation only depend on interrupt thread
    omit 4207cbe67ead [SPARK-46724][DOCS] Update tuning.md to use java 17 doc links
    omit fb9c89a45db5 [SPARK-46094] Support Executor JVM Profiling
    omit b095960c7290 [SPARK-46726][PS][TESTS] Rebalance `pyspark_pandas_connect_part?`
    omit f3ffd3ca68d2 [SPARK-46720][SQL][PYTHON] Refactor Python Data Source to align with other DSv2 built-in Data Sources
    omit 1bdb81b2b547 [SPARK-45435][DOC][FOLLOWUPS] Fix two typos
    omit 2c8ea75d92ee [SPARK-46660][CONNECT] ReattachExecute requests updates aliveness of SessionHolder
    omit 02573741605d [SPARK-46713][PYTHON][DOCS] Refine docstring of `map_keys/map_values/map_entries`
    omit ee0d243f0141 [SPARK-46719][PS][TESTS] Rebalance `pyspark_pandas` and `pyspark_pandas_slow`
    omit 65d822c44d93 [SPARK-45435][DOC] Document that lazy checkpoint may not be a consistent snapshot
    omit b18b6cca28ef [SPARK-46716][SS][TESTS] Add a test regarding to backward compatibility check for Scala StreamingQueryListener
    omit 416b7b1cd5a6 [SPARK-46715][INFRA] Pin `sphinxcontrib-*`
    omit 4f72ef843c4e [MINOR][DOCS] Add missing part of RDD mapPartitions* docstrings
    omit 86bac8911a31 [SPARK-46711][SS] Fix RocksDB state provider race condition during rollback
    omit 173ba2c6327f [MINOR][SS] Fix indent for streaming aggregation operator
    omit 1c3b94150b44 [SPARK-46709][SS] Expose partition_id column for state data source
    omit 11890cc79b32 [SPARK-46693][SQL] Inject LocalLimitExec when matching OffsetAndLimit or LimitAndOffset
    omit 4be05f55fdb9 [SPARK-46697][SQL][HIVE] Eliminate Hive configuration time variables conversion
    omit 57121c7d815a [SPARK-42960][PYTHON][SS][TESTS] Factor Connect/non-Connect specific logics out
    omit 1c9b022f5264 [SPARK-46699][PS][TESTS] Split `ArithmeticTests`
    omit 4ea374257c1f [SPARK-46700][CORE] Count the last spilling for the shuffle disk spilling bytes metric
    omit 25c680cfd4dc [SPARK-46704][CORE][UI] Fix `MasterPage` to sort `Running Drivers` table by `Duration` column correctly
    omit 6e5431501ec0 [SPARK-46703][INFRA] Disable `fail-fast` in Python CI
    omit a3266b411723 [SPARK-46695][SQL][HIVE] Always setting hive.execution.engine to mr
    omit fc6657624be9 [SPARK-46698][CORE] Replace Timer with single thread scheduled executor for ConsoleProgressBar
    omit 755c892a7b40 [SPARK-46681][CORE] Refactor `ExecutorFailureTracker#maxNumExecutorFailures` to avoid calculating `defaultMaxNumExecutorFailures` when `MAX_EXECUTOR_FAILURES` is configured
    omit 9f061b0d03a3 [SPARK-46692][BUILD] Load `inputs.envs` always in `Python` and `Upload`-related steps
    omit d0a89f8fecc7 [SPARK-46665][PYTHON] Remove `assertPandasOnSparkEqual`
    omit 96f34bbeb40c [SPARK-46052][CORE] Remove function TaskScheduler.killAllTaskAttempts
    omit 9e68a4ca5a2b [SPARK-46586][SQL][TESTS][FOLLOWUP] Replace `appended` with `:+`
    omit 3a8c1bca4fff [MINOR][K8S] Remove unused local val `outstanding` from `ExecutorPodsAllocator#onNewSnapshots`
    omit 145e169a2c8b [SPARK-46694][SQL][TESTS] Drop the assumptions of 'hive version < 2.0' in Hive version related tests
    omit b7e976696f51 [SPARK-46684][PYTHON][CONNECT] Fix CoGroup.applyInPandas/Arrow to pass arguments properly
    omit 28da1d853477 [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of `TaskInfo.accumulables()`
    omit 4828f49ce15a [SPARK-46670][PYTHON][SQL] Make DataSourceManager self clone-able by separating static and runtime Python Data Sources
    omit bbeb8d7417ba [SPARK-46640][SQL] Fix RemoveRedundantAlias by excluding subquery attributes
    omit 462b3dd2bf39 [SPARK-46667][SQL] XML: Throw error on multiple XML data source
    omit 2a27d697d291 [SPARK-46682][BUILD] Upgrade `curator` to 5.6.0
    omit c18c87dc70d4 Revert "[SPARK-46474][INFRA] Upgrade upload-artifact action to v4"
    omit 89c71f7c195d [SPARK-46368][CORE] Support `readyz` in REST Submission API
    omit 4b0ceaddde17 [SPARK-46664][CORE][TESTS][FOLLOWUP] Add a persistent `DriverInfo`
    omit d30796021174 [SPARK-46655][SQL] Skip query context catching in `DataFrame` methods
    omit c84ac6f7cf4e [SPARK-46680][BUILD] Upgrade Apache commons-pool2 to 2.12.0
    omit bfaf9e47b35b [SPARK-46678][TEST] Set `datanucleus.autoStartMechanismMode=ignored` to clean the wall of noisy logs
    omit 2a65001ab4fb [SPARK-46672][BUILD] Upgrade log4j2 to 2.22.1
    omit 46d52fc45473 [SPARK-46675][SQL] Remove unused inferTimestampNTZ in ParquetReadSupport
    omit fa929d4ea4ad [SPARK-37039][PS][FOLLOWUP] Add migration guide for behavior change
    omit 5eec99df01f8 [SPARK-46668][DOCS] Parallelize Sphinx build of Python API docs
    omit 48d429dd8014 [SPARK-46614][PYTHON][DOCS] Refine docstring `make_timestamp/make_timestamp_ltz/make_timestamp_ntz/make_ym_interval`
    omit 442bf77426b0 [SPARK-46666][PYTHON][TESTS] Make lxml as an optional testing dependency in test_session
    omit 5c3b36aafb50 [SPARK-46575][SQL][HIVE] Make HiveThriftServer2.startWithContext DevelopApi retriable and fix flakiness of ThriftServerWithSparkContextInHttpSuite
    omit 514ecc6fc183 [SPARK-46656][PS][TESTS] Split `GroupbyParitySplitApplyTests`
    omit ef2e730662f2 [SPARK-46664][CORE] Improve `Master` to recover quickly in case of zero workers and apps
    omit dcfd37c8fee0 [SPARK-46662][K8S][BUILD] Upgrade `kubernetes-client` to 6.10.0
    omit 4653d501c156 [SPARK-46382][SQL] XML: Update doc for `ignoreSurroundingSpaces`
    omit 1e3892f939f8 [SPARK-46658][DOCS] Loosen Ruby dependency specification
    omit 186e1b3f4a43 [SPARK-46474][INFRA] Upgrade upload-artifact action to v4
    omit 5e9e9350de8a [SPARK-46657][INFRA] Install `lxml` in Python 3.12
    omit f7b0b4537917 [SPARK-46547][SS] Swallow non-fatal exception in maintenance task to avoid deadlock between maintenance thread and streaming aggregation operator
    omit a73ff66d9463 [SPARK-46652][SQL][TESTS] Remove `Snappy` from `TPCDSQueryBenchmark` benchmark case name
    omit bda9957bc1d8 [SPARK-46635][PYTHON][DOCS] Refine docstring of `from_csv/schema_of_csv/to_csv`
    omit fcdfc8cbb24e [SPARK-46437][FOLLOWUP] Update configuration.md to use include_api_gen
    omit 63758177e9c6 [SPARK-46651][PS][TESTS] Split `FrameTakeTests`
    omit a3991b17e379 [SPARK-46648][SQL] Use `zstd` as the default ORC compression
    omit 85b504d64701 [SPARK-46442][SQL] DS V2 supports push down PERCENTILE_CONT and PERCENTILE_DISC
    omit d2f572428be5 [MINOR][DOCS] Correct the usage example of Dataset in Java
    omit f526bea1dc0c [SPARK-46649][PYTHON][INFRA] Run PyPy 3 and Python 3.10 tests independently
    omit 4957c1a5fd42 [SPARK-46645][INFRA] Exclude unittest-xml-reporting in Python 3.12 image
    omit 0791e9f302fb [SPARK-46536][SQL] Support GROUP BY calendar_interval_type
    omit 24cb6117ba4a [SPARK-46646][SQL][TESTS] Improve `TPCDSQueryBenchmark` to support other file formats
    omit 686f428dc104 [SPARK-46541][SQL][CONNECT] Fix the ambiguous column reference in self join
    omit 48d22e9f876f [SPARK-46643][SQL][TESTS] Fix ORC tests to be independent from default compression
    omit d613772765fc [MINOR][DOCS] Add license header at docs/_plugins
    omit 0a66be8ded37 [SPARK-37039][PS] Fix `Series.astype` to work properly with missing value
    omit eb1119016088 [MINOR][INFRA] Ensure that docs build successfully with SKIP_API=1
    omit 3a6b9adc21c2 [SPARK-46633][SQL] Fix Avro reader to handle zero-length blocks
    omit 66794197e71f [SPARK-46437][DOCS] Add custom tags for conditional Jekyll includes
    omit d2033282ecb2 [MINOR][PYTHON][TESTS] Retry `test_map_in_pandas_with_column_vector`
    omit 2cf07f9b4595 [SPARK-46593][PS][TESTS] Refactor `data_type_ops` tests again
    omit 71468ebcc85e [SPARK-46637][DOCS] Enhancing the Visual Appeal of Spark doc website
    omit f76502202aca [SPARK-46630][SQL] XML: Validate XML element name on write
    omit 56dd1f7c101e [SPARK-46634][SQL] literal validation should not drill down to null fields
    omit ef7846aa236e [SPARK-46622][CORE] Override `toString` method for `o.a.s.network.shuffledb.StoreVersion`
    omit ee1fd926802b [SPARK-46331][SQL] Removing CodegenFallback from subset of DateTime expressions and version() expression
    omit 8fa794b13195 [SPARK-46627][SS][UI] Fix timeline tooltip content on streaming ui
    omit ee2a87b4642c [SPARK-40876][SQL][TESTS][FOLLOW-UP] Remove invalid decimal test case when ANSI mode is on
    omit f4c8dd68373e Revert "[SPARK-46474][INFRA] Upgrade upload-artifact action to v4"
    omit 03fc5e26b866 [SPARK-46600][SQL] Move shared code between SqlConf and SqlApiConf to SqlApiConfHelper
    omit 7f056d89164c [SPARK-46382][SQL] XML: Default ignoreSurroundingSpaces to true
    omit e7536f2484af [SPARK-46610][SQL] Create table should throw exception when no value for a key in options
    omit d008f81a9d8d [SPARK-46628][INFRA] Use SPDX short identifier in `license` name
    omit a4c950742869 [MINOR][CONNECT] Fix typo in error handling opType name
    omit 79f77f7dd259 [SPARK-46626][DOCS] Bump jekyll to 4.3.3 to enable support for Ruby 3.3.0
    omit bb0f77d076ee [SPARK-46624][BUILD] Upgrade joda-time to 2.12.6
    omit 2957bc9fa1f0 [SPARK-46608][SQL] Restore backward compatibility of `JdbcDialect.classifyException`
    omit 5a74e5a049aa [SPARK-46621][PYTHON] Address null from Exception.getMessage in Py4J captured exception
    omit db4ad6b9ac96 [SPARK-46611][CORE][FOLLOWUP] Correct new use of DateTimeFormat.withZone in UIUtils
    omit 40078a4c3726 [SPARK-46581][CORE] Update comment on isZero in AccumulatorV2
    omit 8facc992fe46 [SPARK-45352][SQL] Eliminate foldable window partitions
    omit d439e34d6bda [SPARK-40876][SQL] Widening type promotion for decimals with larger scale in Parquet readers
    omit f54ecd68e776 [SPARK-46601][CORE] Fix log error in handleStatusMessage
    omit b177df04024a [SPARK-46522][PYTHON] Block Python data source registration with name conflicts
    omit 443a95792073 [SPARK-46611][CORE] Remove ThreadLocal by replace SimpleDateFormat with DateTimeFormatter
    omit a88c64e7dbdd Revert "[SPARK-46437][DOCS] Remove cruft from the built-in SQL functions documentation"
    omit 3beec9aa6d20 [SPARK-46613][SQL][PYTHON] Log full exception when failed to lookup Python Data Sources
    omit 534ce11e1c6f [SPARK-46603][PYTHON][DOCS] Refine docstring of `parse_url/url_encode/url_decode`
    omit 1f2ee8d427bf [SPARK-46606][PYTHON][DOCS] Refine docstring of `convert_timezone/make_dt_interval/make_interval`
    omit 0e232e3aaf21 [SPARK-46607][PYTHON][TESTS] Check the testing mode
    omit 82c02a913012 [SPARK-46248][SQL] XML: Support for ignoreCorruptFiles and ignoreMissingFiles options
    omit 71e0fdbcbeb2 [SPARK-46382][SQL] XML: Refactor inference and parsing
    omit 018808236708 [SPARK-46599][SQL] XML: Use TypeCoercion.findTightestCommonType for compatibility check
    omit e4335206b59a [SPARK-46605][CONNECT] Make `lit/typedLit` function in connect module support `s.c.immutable.ArraySeq`
    omit b35e61f83ff9 [SPARK-46604][SQL] Make `Literal.apply` support `s.c.immuable.ArraySeq`
    omit 0c1c5e93e376 [SPARK-46598][SQL] OrcColumnarBatchReader should respect the memory mode when creating column vectors for the missing column
    omit e8dfcd3081ab [SPARK-46515] Add MONTHNAME function
    omit f8115da1a2bb [SPARK-46609][SQL] Avoid exponential explosion in PartitioningPreservingUnaryExecNode
    omit 9b3c70f6094c [SPARK-46602][SQL] Propagate `allowExisting` in view creation when the view/table does not exists
    omit 70b90c8b5778 [SPARK-46592][DOCKER][TESTS][FOLLOWUP] Add newline to ORACLE Docker preparation sql
    omit a98c885da09d [SPARK-46568][PYTHON] Make Python data source options a case-insensitive dictionary
    omit 20b6a32f4980 Revert "[SPARK-46593][PS][TESTS] Refactor `data_type_ops` tests"
    omit b96e13d09361 [SPARK-46595][PYTHON][DOCS] Refine docstring of `map_from_arrays/map_from_entries/map_concat`
    omit f03f3d01176d [SPARK-46560][PYTHON][DOCS] Refine docstring `reverse/map_contains_key`
    omit b71192cb08cb [SPARK-46586][SQL] Support `s.c.immutable.ArraySeq` as `customCollectionCls` in `MapObjects`
    omit f9ca519897d5 [SPARK-46179][SQL] Add CrossDbmsQueryTestSuites, which runs other DBMS against golden files with other DBMS, starting with Postgres
    omit 3793c2f44099 [SPARK-46593][PS][TESTS] Refactor `data_type_ops` tests
    omit ddcfe6b10980 [SPARK-46596][CORE][TESTS] Correct package name of `SslTestUtils`
    omit 760de230eb6e [SPARK-46597][SQL][TESTS] Optimize the run tests command in the doc of XXIntegrationSuite in `docker-integration-tests` module
    omit 9ec426dd81c5 [SPARK-46592][DOCKER][TEST] OracleIntegrationSuite is flaky because of ORA-04021
    omit ae2e00e8f8e9 [SPARK-45527][CORE] Use fraction to do the resource calculation
    omit 6a9a06d3d3a9 [SPARK-46569][SQL] Remove ThreadLocal due to SecureRandom is thread safe since JDK9
    omit c8137960a0ba [SPARK-46585][CORE] Directly constructed `metricPeaks` as an `immutable.ArraySeq` instead of use `mutable.ArraySeq.toSeq` in `Executor`
    omit c63e0641f2f3 [SPARK-46587][SQL] XML: Fix XSD big integer conversion
    omit b303eced7f86 [SPARK-46530][PYTHON][SQL][FOLLOW-UP] Uses path separator instead of file separator to correctly check PySpark library existence
    omit 69c46876b5a7 [SPARK-46576][SQL] Improve error messages for unsupported data source save mode
    omit 59d147a4f48f [SPARK-46504][PS][TESTS][FOLLOWUP] Break the remaining part of `IndexesTests` into small test files
    omit 1cd3a1b0e1c0 Revert "[SPARK-46582][R][INFRA] Upgrade R Tools version from 4.0.2 to 4.3.2 in AppVeyor"
    omit 56023635ab89 [SPARK-46412][K8S][DOCS] Update Java and JDK info in K8S testing
    omit f3e454a8323a [SPARK-45292][SQL][HIVE] Remove Guava from shared classes from IsolatedClientLoader
    omit 733be49a8078 [SPARK-46539][SQL][FOLLOWUP] fix golden files
    omit 7b6077a02fc3 [SPARK-46584][SQL][TESTS] Remove invalid attachCleanupResourceChecker in JoinSuite
    omit dbdeecbc3ffc [SPARK-46582][R][INFRA] Upgrade R Tools version from 4.0.2 to 4.3.2 in AppVeyor
    omit 893e69172560 [SPARK-46580][TESTS] Regenerate benchmark results
    omit 5c10fb3e509a [SPARK-44556][SQL] Reuse `OrcTail` when enable vectorizedReader
    omit 85b44ccef4c4 [SPARK-46546][DOCS] Fix the formatting of tables in `running-on-yarn` pages
    omit 49f94eb6e88a [SPARK-46579][SQL] Redact JDBC url in errors and logs
    omit 9c46d9dcd195 [SPARK-46539][SQL] SELECT * EXCEPT(all fields from a struct) results in an assertion failure
    omit 06f9e7419966 [SPARK-46550][BUILD][SQL] Upgrade `datasketches-java` to 5.0.1
    omit 605fecd22cc1 [SPARK-46577][SQL] HiveMetastoreLazyInitializationSuite leaks hive's SessionState
    omit 3b1d843da2de [SPARK-46567][CORE] Remove ThreadLocal for ReadAheadInputStream
    omit a3d999292f8e [SPARK-46524][SQL] Improve error messages for invalid save mode
    omit f844e78b563d [SPARK-46563][SQL] View simpleString not follow conf spark.sql.debug.maxToStringFields
    omit 347c955fe723 [SPARK-46573][K8S] Use `appId` instead of `conf.appId` in `LoggingPodStatusWatcherImpl`
    omit 9208c42b3a11 [SPARK-46525][DOCKER][TESTS] Fix docker-integration-tests on Apple Sillicon
    omit b74b1592c9ec [SPARK-46571][PS][TEST] Re-enable TODOs that are resolved from recent Pandas
    omit bdb617278c0b [SPARK-46565][PYTHON] Refine error classes and error messages for Python data sources
    omit 31b3f8187b84 [SPARK-44001][PROTOBUF] Add option to allow unwrapping protobuf well known wrapper types
    omit f8c10fcb5730 [SPARK-43299][FOLLOWUP][CONNECT][SS] Followup on StreamingQueryExceptions
    omit 679926270c75 [SPARK-46504][PS][TESTS][FOLLOWUPS] Moving more slow tests out of `IndexesTests`
    omit 50af8d792511 [SPARK-46570][INFRA] Run Python 3.11 and 3.12 tests independently
    omit 65e464769b5a [SPARK-45338][TEST][SQL][FOLLOWUP] Use scala.jdk.javaapi.CollectionConverters in JavaHigherOrderFunctionsSuite
    omit bc7e949cf993 [SPARK-46562][SQL] Remove retrieval of `keytabFile` from `UserGroupInformation` in `HiveAuthFactory`
    omit 4ab5197529ae [SPARK-46564][PYTHON][INFRA] Exclude unrelated files via using omit options properly in PySpark coverage report
    omit 381f3691bd48 [SPARK-46552][SQL] Replace `UnsupportedOperationException` by `SparkUnsupportedOperationException` in `catalyst`
    omit e690e96860a6 [SPARK-46559][MLLIB] Wrap the `export` in the package name with backticks
    omit 8cec63383085 [SPARK-46558][CONNECT] Extract a helper function to eliminate the duplicate code that retrieves `MessageParameters` from `ErrorParams` in `GrpcExceptionConverter`
    omit 7d09991610f9 [SPARK-46544][SQL] Support v2 DESCRIBE TABLE EXTENDED with table stats
    omit c5dd72cff124 [SPARK-46553][PS] `FutureWarning` for `interpolate` with object dtype
    omit 760cd665f6cd [SPARK-46557][PYTHON][DOCS] Refine docstring for DataFrame.schema/explain/printSchema
    omit 48a09c457cf5 [SPARK-46540][PYTHON] Respect column names when Python data source read function outputs named Row objects
    omit 3fd987606be4 [SPARK-46556][PYTHON][DOCS] Refine docstring for DataFrame.createGlobalTempView/createOrReplaceGlobalTempView
    omit fb594cbcfabe [SPARK-46555][PYTHON][DOCS] Refine docstring for DataFrame.createTempView/createOrReplaceTempView
    omit f3e162336f65 [SPARK-46543][PYTHON][CONNECT] Make `json_tuple` throw PySparkValueError for empty fields
    omit e6082119fedc [SPARK-46554][BUILD] Upgrade slf4j to 2.0.10
    omit 9bf5e82a5577 [SPARK-46551][PYTHON][DOCS] Refine docstring of `flatten/sequence/shuffle`
    omit ae9106370a9f [SPARK-46504][PS][TESTS][FOLLOWUPS] Moving move slow tests out of `IndexesTests`
    omit 9a6b27ecb15e [SPARK-46548][PYTHON][DOCS] Refine docstring of `get/array_zip/sort_array`
    omit b05c61266d83 [SPARK-46490][SQL] Require error classes in `SparkThrowable` sub-classes
    omit e7145ae51bfe [SPARK-46504][PS][TESTS][FOLLOWUPS] Moving slow tests out of `IndexesTests`: conversion and drop methods
    omit af3a22533ca0 [SPARK-46531][BUILD] Move the dependency management of `datasketches-java` from `catalyst` to `parent`
    omit 04b5128745e0 [SPARK-46545][INFRA] Pin `lxml==4.9.4`
    omit 0eeb60bd24cf [SPARK-46542][SQL] Remove the check for `c>=0` from `ExternalCatalogUtils#needsEscaping` as it is always true
    omit 9671abbc1dc9 [SPARK-46509][CORE][SS] Replace `.reverse.find` with `.findLast`
    omit d6334a3ba87c [SPARK-46530][PYTHON][SQL] Check Python executable when looking up available Data Sources
    omit 2e918e8b5c11 [SPARK-46532][CONNECT][PYTHON][FOLLOW-UP] Pass message parameters in Python Spark Connect client
    omit f99b86a1afa0 [SPARK-45914][PYTHON] Support commit and abort API for Python data source write
    omit e9c4e7ebd595 [SPARK-46397][PYTHON][CONNECT] Function `sha2` should raise `PySparkValueError` for invalid `numBits`
    omit c7c43cf5b1e1 [SPARK-46533][PYTHON][DOCS] Refine docstring of `array_min/array_max/array_size/array_repeat`
    omit b249cb8af355 [SPARK-46538][ML] Fix the ambiguous column reference issue in `ALSModel.transform`
    omit 826f8d9eb040 [SPARK-46484][SQL][CONNECT] Make `resolveOperators*` helper functions keep the plan id
    omit 4ec63be2202f [SPARK-46382][SQL] XML: Capture values interspersed between elements
    omit af8228ce9aee [SPARK-46535][SQL] Fix NPE when describe extended a column without col stats
    omit 5db682470441 [SPARK-46537][SQL] Convert NPE and asserts from commands to internal errors
    omit 0de70c4cb0b3 [SPARK-46517][PS][TESTS][FOLLOWUPS] Reorganize `IndexingTest`: Move to `pyspark.pandas.tests.indexes.*` and add the parity test
    omit 6fcc2685cd3f [SPARK-46532][CONNECT] Pass message parameters in metadata of `ErrorInfo`
    omit bb497eb7e8de [SPARK-46519][SQL] Clear unused error classes from `error-classes.json` file
    omit 93a2526528c7 [SPARK-46366][SQL] Use WITH expression in BETWEEN to avoid duplicate expressions
    omit a3e7bad7c16d [SPARK-46529][TESTS] Upgrade guava from 18.0 to 19.0 for `docker-integration-tests`
    omit ff881dad4c50 [SPARK-46517][PS][TESTS][FOLLOWUPS] Reorganize `IndexingTest`: factor out `test_loc2d*`
    omit 229a4eaf547e [SPARK-45917][PYTHON][SQL] Automatic registration of Python Data Source on startup
    omit d0245d34c004 [SPARK-46514][TESTS] Fix HiveMetastoreLazyInitializationSuite
    omit d2e87c84490b [SPARK-46521][PYTHON][DOCS] Refine docstring of `array_remove/array_distinct/array_compact`
    omit bd9a4b9b3863 [SPARK-46528][BUILD] Upgrade `zstd-jni` to 1.5.5-11
    omit 5430c700ba64 [SPARK-46502][SQL] Support timestamp types in UnwrapCastInBinaryComparison
    omit 7cad075652ee [SPARK-46517][PS][TESTS][FOLLOWUPS] Reorganize `IndexingTest`: factor out `test_iloc*`
    omit 3f08c9ff0d64 [MINOR][K8S] Remove an unused expression `None` from `KubernetesClientUtils#loadSparkConfDirFiles`
    omit 638078f4ea44 [SPARK-46520][PYTHON] Support overwrite mode for Python data source write
    omit b106f80521d6 [SPARK-46517][PS][TESTS] Reorganize `IndexingTest`: factor out `test_loc*` tests
    omit 5a5d3c70401f [SPARK-46508][BUILD] Upgrade Jackson to 2.16.1
    omit fb09e31fffc5 [SPARK-46513][PS][TESTS] Move `BasicIndexingTests` to `pyspark.pandas.tests.indexes.*`
    omit fcf5d571e9ac [SPARK-46506][PYTHON][DOCS] Refine docstring of `array_intersect/array_union/array_except`
    omit 8ea09de807f9 [SPARK-46498][CORE] Remove `shuffleServiceEnabled` from `o.a.spark.util.Utils#getConfiguredLocalDirs`
    omit c0ff0f579daa [SPARK-46371][BUILD] Clean up outdated items in `.rat-excludes`
    omit 8d1ca4b58370 [SPARK-46504][PS][TESTS][FOLLOWUPS] Make `test_insert` more stable by sorting before comparison
    omit 5ba6e106b858 [SPARK-46507][PS][TESTS] Split `IndexesSlowTests` into multiple tests
    omit d8fb91e61352 [SPARK-46444][SQL] V2SessionCatalog#createTable should not load the table
    omit 05c48e7450a9 [MINOR][DOCS] Fix rst link in Python API docs for .sql()
    omit ca147d2f30ee [SPARK-45597][PYTHON][SQL][FOLLOW-UP] Minor deduplicate datasource checking logic
    omit 41383e974d86 [SPARK-46504][PS][TESTS] Factor slow tests out of `IndexesTests`
    omit 4911a5bad4ac [SPARK-46505][CONNECT] Make bytes threshold configurable in `ProtoUtils.abbreviate`
    omit 439ec6b954b4 [SPARK-45600][SQL][PYTHON][FOLLOW-UP] Make Python data source registration session level
    omit 3ee2db8f0bee [MINOR][DOCS] Python docs also require Pandoc
    omit 6e3b9f74229e [SPARK-46503][PS][TESTS] Move `test_default_index` to `pyspark.pandas.tests.indexes.*`
    omit c1888cdf5361 [SPARK-40876][SQL][TESTS][FOLLOWUP] Fix failed test in `ParquetTypeWideningSuite` when `SPARK_ANSI_SQL_MODE` is set to true
    omit bb85bbcf16f7 [SPARK-46501][INFRA] List the python packages with the correct versions
    omit e314d3cd82d6 [SPARK-46497][SQL][TESTS] Re-enable the test cases that were ignored in SPARK-45309
    omit 768aba1d0582 [SPARK-46475][BUILD] Upgrade RoaringBitmap to 1.0.1
    omit a546ff4bbe5e [SPARK-46499][BUILD] Bump sbt-eclipse 6.2.0
    omit 1a7a2f7a889f [SPARK-46488][SQL] Skipping trimAll call during timestamp parsing
    omit c1d3fc1de30e [SPARK-46481] Execute immediate VariableReference foldable
    omit d1a333d554ad [MINOR][INFRA] Comments in GitHub scripts should start with #
    omit d85ad1c14182 [SPARK-46500][PS][TESTS] Reorganize `FrameParityPivotTests`
    omit eae09299c6c9 [SPARK-46496][BUILD] Upgrade Arrow to 14.0.2
    omit 5db7beb59a67 [SPARK-46455][CORE][SQL][SS][CONNECT][PYTHON] Remove redundant type conversion
    omit 53c7d16080cf [SPARK-46478][SQL] Revert SPARK-43049 to use oracle varchar(255) for string
    omit ef7e1e685f61 [SPARK-46471][PS][TESTS][FOLLOWUPS] Move `OpsOnDiffFramesEnabledTests` to `pyspark.pandas.tests.diff_frames_ops.*``
    omit bd29569c5ddd [SPARK-46495][INFRA] Merge pyspark-error to pyspark-core
    omit 4f56958c1231 [SPARK-46471][PS][TESTS][FOLLOWUPS] Reorganize `OpsOnDiffFramesEnabledTests`: Factor out more tests
    omit eacf82ad227e [SPARK-45231][INFRA][FOLLOWUP] Remove unrecognized and meaningless command about Ammonite from the GA testing workflow
    omit 98fef6ea5855 [SPARK-46471][PS][TESTS][FOLLOWUPS] Reorganize `OpsOnDiffFramesEnabledTests`: Factor out `test_assignment_*`
    omit 64d1cc96fa9c [SPARK-46492][BUILD] Simplify the Java version check in `SparkBuild.scala`
    omit 5c2896abce4a [SPARK-46493][INFRA] Upgrade `apache-rat` in the `dev/check-license` to 0.15
    omit 82d0fb494f3f [SPARK-46474][INFRA] Upgrade upload-artifact action to v4
    omit 0c29593436d5 [SPARK-46479][CORE] Change `Utils.isJavaVersionAtLeast21` to use the utility method from `commons-lang3`
    omit 0ad2bf6b8ec2 [SPARK-46483][INFRA] Exclude `apache-rat.jar` from `ScalaUnidoc/unidoc/unidocAllClasspaths`
    omit cb2f47bf7ec1 [SPARK-46485][SQL] V1Write should not add Sort when not needed
    omit 3361f25dc0ff [SPARK-40876][SQL] Widening type promotions in Parquet readers
    omit 43f79326106a [SPARK-46480][CORE][SQL] Fix NPE when table cache task attempt
    omit fc7d7bce7732 [SPARK-46464][DOC] Fix the scroll issue of tables when overflow
    omit 6b931530d75c [SPARK-46472][PYTHON][DOCS] Refine docstring of `array_prepend/array_append/array_insert`
    omit a921da8509a1 [SPARK-46443][SQL] Decimal precision and scale should decided by H2 dialect
    omit 8c63485189c8 [SPARK-46380][SQL][FOLLOWUP] Simplify the code for ResolveInlineTables and ResolveInlineTablesSuite
    omit 3432fd8dba5b [SPARK-46468][SQL] Handle COUNT bug for EXISTS subqueries with Aggregate without grouping keys
    omit 4fcd5bfe003b [SPARK-45525][SQL][PYTHON] Support for Python data source write using DSv2
    omit e4b5977c9b7b [SPARK-46437][DOCS] Remove cruft from the built-in SQL functions documentation
    omit db162e5a139d [SPARK-46471][PS][TESTS][FOLLOWUPS] Reorganize `OpsOnDiffFramesEnabledTests`: Factor out `test_arithmetic_chain_*`
    omit 2723bb11e3c7 [SPARK-46476][BUILD][CORE][CONNECT] Move `IvyTestUtils` back to `src/test` directory
    omit 87d22fd346ce [SPARK-46470][PS][TESTS] Move `test_series_datetime` to `pyspark.pandas.tests.series.*`
    omit 4d21e5547580 [SPARK-46466][SQL] Vectorized parquet reader should never do rebase for timestamp ntz
    omit c10b2c0fbaeb [SPARK-46471][PS][TESTS] Reorganize `OpsOnDiffFramesEnabledTests`: Factor out `test_arithmetic_*`
    omit 0e94f340a63a [SPARK-46378][SQL][FOLLOWUP] Do not rely on TreeNodeTag in Project
    omit 5fe963f8560e [SPARK-46380][SQL] Replace current time/date prior to evaluating inline table expressions
    omit 558e7a047bfd [SPARK-46465][PYTHON][DOCS][FOLLOWUPS] Add `Column.isNaN` to API references
    omit fd8d17a136eb [SPARK-46469][SQL] Clean up useless local variables in `InsertIntoHiveTable`
    omit e804f5360144 [SPARK-46179][SQL] Pull out code into reusable functions in SQLQueryTestSuite
    omit 56dc7f8c2199 [SPARK-46207][SQL] Support MergeInto in DataFrameWriterV2
    omit 57c01df22fd0 [MINOR][PYTHON][DOCS] Fix typo in pyspark.sql.functions.greatest docstring
    omit d56fe6c7b202 [SPARK-46465][PYTHON][CONNECT] Add `Column.isNaN` in PySpark
    omit 9b583809dd94 [SPARK-46462][PS][TESTS] Reorganize `OpsOnDiffFramesGroupByRollingTests`
    omit 091d8da54930 [SPARK-46463][PS][TESTS] Reorganize `OpsOnDiffFramesGroupByExpandingTests`
    omit 4faeb803670d [SPARK-46413][PYTHON] Validate returnType of Arrow Python UDF
    omit 28b0c2c11d68 [SPARK-46398][PYTHON][TESTS] Test rangeBetween window function (pyspark.sql.window)
    omit 847b65eac370 [SPARK-46456][CORE] Add `spark.ui.jettyStopTimeout` to set Jetty server stop timeout  to unblock SparkContext shutdown
    omit 1005cd5576ef [SPARK-46447][SQL] Remove the legacy datetime rebasing SQL configs
    omit c9cfaac90fd4 [SPARK-46452][SQL] Add a new API in DataWriter to write an iterator of records
    omit cf54e8f9a51b [SPARK-46330] Loading of Spark UI blocks for a long time when HybridStore enabled
    omit 9fdc3754d676 [SPARK-28386][SQL] Cannot resolve ORDER BY columns with GROUP BY and HAVING
    omit f9e468eafdf0 [SPARK-46399][CORE] Add exit status to the Application End event for the use of Spark Listener
    omit cc9d9911d7ed [SPARK-46272][SQL] Support CTAS using DSv2 sources
    omit f555e9983f6d [SPARK-46069][SQL][FOLLOWUP] Simplify the algorithm and add comments
    omit 771eb89ab2a4 [SPARK-46433][PS][TESTS][FOLLOWUPS] Reorganize `OpsOnDiffFramesGroupByTests`: Factor out remaining slow tests
    omit 6fe6828b59ac [SPARK-46302][TESTS] Skip maven daily testing as ivy uses some corrupted cache jar files
    omit 1591f7461e68 [SPARK-46446][SQL][FOLLOW-UP] Fix sql tests for subqueries with correlated OFFSET
    omit aa1ff3789e49 [SPARK-46246][SQL] EXECUTE IMMEDIATE SQL support
    omit ea7d00a8424b [SPARK-46454][SQL][DSTREAM] Remove redundant `.headOption`
    omit 5cca6ba82950 [SPARK-46451][PS][TESTS] Reorganize `GroupbyStatTests`
    omit dc0bfc4c700c [SPARK-46453][CONNECT] Throw exception from `internalError()` in `SessionHolder`
    omit 58b22fd17345 [SPARK-46433][PS][TESTS][FOLLOWUPS] Reorganize `OpsOnDiffFramesGroupByTests`: Factor out 4 more slow tests
    omit 0b8e0998c673 [SPARK-46446][SQL] Disable subqueries with correlated OFFSET to fix correctness bug
    omit 5cbad6aef2fd [SPARK-46424][PYTHON][SQL][FOLLOW-UP] Refactor and clean up Python metric implementation
    omit 8cd466136528 [SPARK-46308] Forbid recursive error handling by adding recursion guards
    omit eb17ddff1d77 [SPARK-46377][INFRA] Upgrade labeler action to v5
    omit 9a75a1d69aae [SPARK-46411][BUILD] Change to use `bcprov/bcpkix-jdk18on` for UT
    omit c41a5ff60a71 [SPARK-46439][PS][TESTS] Move IO-related tests to `pyspark.pandas.tests.io.*`
    omit 61f51113d884 [SPARK-46420][SQL] Remove unused transport in SparkSQLCLIDriver
    omit 9756bbe8026c [SPARK-46289][SQL] Support ordering UDTs in interpreted mode
    omit feeb9a0b6208 [SPARK-46438][UI] Fix the broken scope of toggleFramegraph when registering onclick event for the Frame Graph header
    omit 036af4bf2336 [SPARK-45725][SQL][FOLLOWUP] Fix arguments of a removed SQL config
    omit 5c80e6aec29e [SPARK-46069][SQL][FOLLOWUP] Simplify the code and add more UT
    omit bfafad4d47b4 [SPARK-46440][SQL] Set the rebase configs to the `CORRECTED` mode by default
    omit 78c2c2674b46 [SPARK-46436][INFRA] Clean up compatibility configurations related to branch-3.3 daily testing in `build_and_test.yml`
    omit 51dd7268f895 [SPARK-46435][INFRA] Exclude `branch-3.3` from `publish_snapshot.yml`
    omit 1b413a4bd61e [SPARK-46433][PS][TESTS] Reorganize `OpsOnDiffFramesGroupByTests`: Factor out `test_aggregate/apply/cum*`
    omit 5edd0ec696d6 [SPARK-46317][INFRA][FOLLOWUP] Install `lxml` in PyPy3.8
    omit dff0d10d7acd [SPARK-46434][INFRA] Remove `build_branch33.yml`
    omit d163b319e21a [SPARK-46431][SS] Convert `IllegalStateException` to `internalError` in session iterators
    omit 96bf373e9002 [SPARK-46430][CONNECT][TESTS] Add test for `ProtoUtils.abbreviate`
    omit 29db77653fe0 [SPARK-46419][PS][TESTS][FOLLOWUP] Reorganize DatetimeIndexTests: Factor out remaining slow tests
    omit 5e1b904ca54f [SPARK-46389][CORE] Manually close the `RocksDB/LevelDB` instance when `checkVersion` throw Exception
    omit 0745bb507f36 [SPARK-46376][SQL][TESTS] Simplify the code to generate the Spark tarball `filename` in the `HiveExternalCatalogVersionsSuite`
    omit 14a933bbe2eb [SPARK-46410][SQL] Assign error classes/subclasses to JdbcUtils.classifyException
    omit e955a5979cd3 [SPARK-46406][SQL] Assign a name to the error class _LEGACY_ERROR_TEMP_1023
    omit b702f63bed27 [SPARK-45502][BUILD] Upgrade Kafka to 3.6.1
    omit 20be0b0bffcb [MINOR][DOCS] Use user-facing feature names in Python metrics documentation
    omit 62d4bab3f2b3 [SPARK-46386][PYTHON] Improve assertions of observation (pyspark.sql.observation)
    omit 7004f9ed94be [SPARK-46424][PYTHON][SQL] Support Python metrics in Python Data Source
    omit f69f791ef878 [SPARK-45795][SQL] DS V2 supports push down Mode
    omit ac935f5074da [SPARK-46427][PYTHON][SQL] Change Python Data Source's description to be pretty in explain
    omit ed9a3a8b9593 [MINOR][SQL][PYTHON] Rename Python plan node names
    omit f350ddba6826 [SPARK-46403][SQL] Decode parquet binary with getBytesUnsafe method
    omit d0da1172b7d8 [SPARK-46425][INFRA] Pin the bundler version in CI
    omit f9a8e135761e [SPARK-46414][UI] Use prependBaseUri to render javascript imports
    omit d34d9b39aec0 [SPARK-46422][PS][TESTS] Move `test_window` to `pyspark.pandas.tests.window.*`
    omit 62fc27d79d5c [SPARK-46423][PYTHON][SQL] Make the Python Data Source instance at DataSource.lookupDataSourceV2
    omit 69de04f6aaac [SPARK-45807][SQL] Improve ViewCatalog API
    omit 3d447c21b426 [SPARK-46419][PS][TESTS] Reorganize `DatetimeIndexTests`: Factor out 3 slow tests
    omit a1b0da200b27 [SPARK-45597][PYTHON][SQL] Support creating table using a Python data source in SQL (DSv2 exec)
    omit 69237c9b09f3 [SPARK-46409][CONNECT] Fix spark-connect-scala-client launch script
    omit 15e99cf676d9 [MINOR][CONNECT][PYTHON] Fix minor test issue and properly yield extension from result
    omit 59488039f58b [SPARK-46417][SQL] Do not fail when calling hive.getTable and throwException is false
    omit d5d2c580df4d [SPARK-46418][PS][TESTS] Reorganize `ReshapeTests`
    omit c0caee754116 [SPARK-46402][PYTHON] Add getMessageParameters and getQueryContext support
    omit 081bc36356cd [SPARK-46416][CORE] Add `@tailrec` to  `HadoopFSUtils#shouldFilterOutPath`
    omit 7d30bd7e25ed [SPARK-46384][SPARK-46404][SS][UI] Fix Operation Duration Stack Chart on Structured Streaming Page
    omit b1e8b50e018c [SPARK-46407][PS][TESTS] Reorganize `OpsOnDiffFramesDisabledTests`
    omit 09365a5e3c2f [SPARK-45005][PS][TESTS][FOLLOWUPS] Deduplicate `test_properties`
    omit 50e668c780da [SPARK-46294][SQL] Clean up semantics of init vs zero value
    omit c045a425bf0c [SPARK-46043][SQL][FOLLOWUP] Remove the catalog and identifier check in DataSourceV2Relation
    omit e3f46ed57dc0 [SPARK-45796][SQL] Support MODE() WITHIN GROUP (ORDER BY col)
    omit 2893cd304a9b [SPARK-46391][PS][TESTS] Reorganize `ExpandingParityTests`
    omit ab9ca96ca0ae [SPARK-46401][CORE] Use `!isEmpty()` on `RoaringBitmap` instead of `getCardinality() > 0` in `RemoteBlockPushResolver`
    omit 4a79ae9d821e [SPARK-46396][SQL] Timestamp inference should not throw exception
    omit f1e5a136fa79 [SPARK-46393][SQL] Classify exceptions in the JDBC table catalog
    omit b8c49a168998 [SPARK-46379][PS][TESTS][FOLLOWUPS] Deduplicate `test_interpolate_error`
    omit deb1d1b86162 [SPARK-46394][SQL] Fix spark.catalog.listDatabases() issues on schemas with special characters when `spark.sql.legacy.keepCommandOutputSchema` set to true
    omit 7b24c7c8d23f Revert "[SPARK-46377][INFRA] Upgrade labeler action to v5"
    omit 270da6f9b7b3 [SPARK-46377][INFRA] Upgrade labeler action to v5
    omit 977f64f0904e [SPARK-46357] Replace incorrect documentation use of setConf with conf.set
    omit 6afc0f856298 [SPARK-46360][PYTHON][FOLLOWUP][DOCS] Add `getMessage` to API reference
    omit 79f61985c8de [SPARK-46385][PYTHON][TESTS] Test aggregate functions for groups (pyspark.sql.group)
    omit 96107ca42ddb [SPARK-46387][PS][DOCS] Add an information of `isocalendar` into migration guide
    omit ff26b602c92b [SPARK-45798][CONNECT] Followup: add serverSessionId to SessionHolderInfo
    omit 4c8d01aa3a51 [SPARK-46153][SQL] XML: Add TimestampNTZType support
    omit ccc436d829cd [SPARK-46388][SQL] HiveAnalysis misses the pattern guard of `query.resolved`
    omit b3398e695d92 [SPARK-46381][SQL] Migrate sub-classes of `AnalysisException` to error classes
    omit 4f65413031d8 [SPARK-46379][PS][TESTS] Reorganize `FrameInterpolateTests`
    omit 83434af22284 [SPARK-46132][CORE] Support key password for JKS keys for RPC SSL
    omit b035bb177c08 [SPARK-46370][SQL] Fix bug when querying from table after changing column defaults
    omit dcbebce9eacb [SPARK-46202][CONNECT] Expose new ArtifactManager APIs to support custom target directories
    omit 01d61a5fc963 [SPARK-46253][PYTHON] Plan Python data source read using MapInArrow
    omit c1ba963e64a2 [SPARK-46378][SQL] Still remove Sort after converting Aggregate to Project
    omit 2215cef40043 [SPARK-46353][CORE] Refactor to improve `RegisterWorker` unit test coverage
    omit a122b8acc2f4 [SPARK-46075][CONNECT] Improvements to SparkConnectSessionManager
    omit 071c684dee44 [MINOR][SQL] Convert `UnresolvedException` to an internal error
    omit 8d64cb4144d1 [SPARK-46360][PYTHON] Enhance error message debugging with new `getMessage` API
    omit 2a49feeb5d72 [SPARK-43427][PROTOBUF] spark protobuf: allow upcasting unsigned integer types
    omit e434c9f0d579 [SPARK-46369][CORE] Remove `kill` link from `RELAUNCHING` drivers in `MasterPage`
    omit 3e0808c33f18 [SPARK-46351][SQL] Require an error class in `AnalysisException`
    omit 7db85642600b [SPARK-46273][SQL] Support INSERT INTO/OVERWRITE using DSv2 sources
    omit 35bc0b4ecd26 [SPARK-46341][PS][TESTS][FOLLOWUP] Sort indices before dataframe comparison
    omit d5241ff26892 [SPARK-46347][PS][TESTS] Reorganize `RollingTests `
    omit fc5868791a5d [SPARK-46355][SQL] XML: Close InputStreamReader on read completion
    omit 051b1781827d [SPARK-46358][CONNECT] Simplify the condition check in the `ResponseValidator#verifyResponse`
    omit bb886abcc4a1 [SPARK-46356][BUILD] Upgrade `sbt-assembly` to 2.1.5, `sbt-checkstyle-plugin` to 4.0.1
    omit bab884082c0f [MINOR][DOCS] Fix documentation for `spark.sql.legacy.doLooseUpcast` in SQL migration guide
    omit 6a197efeb3c1 [SPARK-45649][SQL] Unify the prepare framework for OffsetWindowFunctionFrame
    omit bacdb3b5fec9 [SPARK-46215][CORE][FOLLOWUP] Handle symbolic links
    omit fd009d652f79 [SPARK-45642][CORE][SQL] Fix `FileSystem.isFile & FileSystem.isDirectory is deprecated`
    omit 7a43de193aa5 [SPARK-45969][DOCS] Document configuration change of executor failure tracker
    omit a77713081a80 [SPARK-46285][SQL] Add `foreachWithSubqueries`
    omit d02fbba6491f [SPARK-46342][SQL] Replace `IllegalStateException` by `SparkException.internalError` in sql
    omit 109b1e4a79d9 [SPARK-46346][CORE] Fix Master to update a worker from `UNKNOWN` to `ALIVE` on `RegisterWorker` msg
    omit 82a02321b873 [SPARK-46348][CORE] Support `spark.deploy.recoveryTimeout`
    omit e4570f871ad4 [SPARK-46345][PS][TESTS] Reorganize `DiffFramesParityCovCorrWithTests`
    omit bac3492980a3 [SPARK-46341][PS][TESTS] Reorganize `SeriesInterpolateTests`
    omit 55d6b51af7cd [SPARK-46340][PS][CONNECT][TESTS] Reorganize `EWMTests`
    omit 166df8b1df59 [SPARK-46344][CORE] Warn properly when a driver exists successfully but master is disconnected
    omit 75805f07f5ca [SPARK-46339][SS] Directory with batch number name should not be treated as metadata log
    omit cae4bdc88161 [SPARK-46322][PYTHON][DOCS] Replace external link with internal link for error documentation
    omit 1af4b658064d [SPARK-46338][PS][TESTS] Re-enable the `get_item` test for `BasicIndexingTests`
    omit 8e95929ac423 [SPARK-46333][SQL] Replace `IllegalStateException` by `SparkException.internalError` in catalyst
    omit 79d1cded8555 [SPARK-46337][SQL][CONNECT][PYTHON] Make `CTESubstitution` retain the `PLAN_ID_TAG`
    omit 48e773451eeb [SPARK-46334][INFRA][PS] Upgrade `Pandas` to 2.1.4
    omit b28988dbc6ff [SPARK-46335][BUILD] Upgrade Maven to 3.9.6 for MNG-7913
    omit 75cb37011fa2 [SPARK-46325][CONNECT] Remove unnecessary override functions when constructing `WrappedCloseableIterator` in `ResponseValidator#wrapIterator`
    omit d9f0fccd967b [SPARK-46332][SQL] Migrate `CatalogNotFoundException` to the error class `CATALOG_NOT_FOUND`
    omit 3224cddcf9da [SPARK-46324][SQL][PYTHON] Fix the output name of pyspark.sql.functions.user and session_user
    omit 20c9b3dc4fac [SPARK-46328][SQL] Allocate capacity of array list of TColumns by columns size in TRowSet generation
    omit 20df062d85e8 [SPARK-46327][PS][CONNECT][TESTS] Reorganize `SeriesStringTests`
    omit f41bacb07d04 [SPARK-46326][PYTHON][TESTS] Test missing cases for functions (pyspark.sql.functions)
    omit e76eb9fa4c42 [SPARK-46323][PYTHON] Fix the output name of pyspark.sql.functions.now
    omit 309c796876f3 [SPARK-46275] Protobuf: Return null in permissive mode when deserialization fails
    omit 9ffdcc398ed5 [SPARK-46321][PS][TESTS] Re-ennable `IndexesTests.test_asof` that was skipped due to Pandas bug
    omit c06d41859f08 [SPARK-46320][CORE] Support `spark.master.rest.host`
    omit b6b450927ec8 [SPARK-46317][PYTHON][CONNECT] Match minor behaviour matching in SparkSession with full test coverage
    omit 19193854c759 [SPARK-46315][PYTHON][TESTS] Test invalid key for spark.conf.get (pyspark.sql.conf)
    omit cc254e6e9c2e [SPARK-46318][PYTHON][INFRA] Exclude ported pyspark.loose_version from the code coverage report
    omit 7e4a63a0588f [SPARK-46316][CORE] Enable `buf-lint-action` on `core` module
    omit 80dc64a573e1 Revert "[SPARK-46316][CORE] Enable `buf-lint-action` on `core` module"
    omit dcbae0643ce1 [SPARK-46316][CORE] Enable `buf-lint-action` on `core` module
    omit 105eee73cfa0 [SPARK-46313][CORE] Log `Spark HA` recovery duration
    omit 61a3e0587df6 [SPARK-46312][CORE] Use `lower_camel_case` in `store_types.proto`
    omit 9585cf6d56e3 [SPARK-46309][PS][TESTS] Remove unused code in `pyspark.pandas.tests.indexes.* `
    omit 82e67461511e [SPARK-46311][CORE] Log the final state of drivers during `Master.removeDriver`
    omit 027aeb1764a8 [SPARK-46277][PYTHON] Validate startup urls with the config being set
    omit e359318c4493 [SPARK-46293][CONNECT][PYTHON] Use `protobuf` transitive dependency
    omit 8132e1700c81 [SPARK-46261][CONNECT] `DataFrame.withColumnsRenamed` should keep the dict/map ordering
    omit 0692856bb124 [SPARK-46307][PS][TESTS] Enable `fill_value` tests for `GroupByTests.test_shift`
    omit 06c1317bcf37 [SPARK-46112][BUILD][PYTHON] Implement lint check for PySpark custom errors
    omit 7ba5d0c5ce04 [SPARK-46305][BUILD][SS] Remove Zookeeper `3.5.7` test dependency from the `streaming-kafka-0-10` and `sql-kafka-0-10` modules
    omit b4d90dd2a622 [SPARK-46304][INFRA] Upgrade setup-java action to v4 and setup-python to v5
    omit b92d64d6ef0c [SPARK-46301][CORE] Support `spark.worker.(initial|max)RegistrationRetries`
    omit 12c075c9d6cb [SPARK-46303][PS][TESTS] Remove unused code in `pyspark.pandas.tests.series.* `
    omit 19fa431ef611 [SPARK-46300][PYTHON][CONNECT] Match minor behaviour matching in Column with full test coverage
    omit b3c56cffdb6f [SPARK-45515][CORE][SQL][FOLLOWUP] Use enhanced switch expressions to replace the regular switch statement
    omit 2ddb6be47243 [SPARK-46298][PYTHON][CONNECT] Match deprecation warning, test case, and error of Catalog.createExternalTable
    omit 6337412973af [SPARK-46296][PYTHON][TESTS] Test missing test coverage for captured errors (pyspark.errors.exceptions)
    omit 43ca0b929ab3 [SPARK-46299][DOCS] Make `spark.deploy.recovery*` docs up-to-date
    omit d02674713ac2 [SPARK-46297][PYTHON][INFRA] Exclude generated files from the code coverage report
    omit 9c9020ebafd8 [SPARK-46058][CORE] Add separate flag for privateKeyPassword
    omit e2441c41de47 [SPARK-46292][CORE][UI] Show a summary of workers in MasterPage
    omit 9da1e4ca7bb8 [SPARK-46290][PYTHON] Change saveMode to a boolean flag for DataSourceWriter
    omit e013c4e859e5 [SPARK-46231][PYTHON][FOLLOWUP] Cleanup duplicated test
    omit 9815bb19ef10 [MINOR][PYTHON][DOCS] Rename "Koalas" > "Pandas-on-Spark" for testing util doc
    omit 9fd575ae46f8 [SPARK-46274][SQL] Fix Range operator computeStats() to check long validity before converting
    omit 31a48381f513 [SPARK-46230][PYTHON] Migrate `RetriesExceeded` into PySpark error
    omit 231d89f89ede [SPARK-46270][SQL][CORE][SS] Use java16 instanceof expressions to replace the java8 instanceof statement
    omit 982c7268f4c7 [SPARK-46186][CONNECT][TESTS][FOLLOWUP] Remove flakiness of `ReattachableExecuteSuite`
    omit c96fef2ea55e [SPARK-45580][SQL] Handle case where a nested subquery becomes an existence join
    omit 467df65ceba5 [SPARK-46232][PYTHON][FOLLOWUP] Migrate `ValueError` into `PySparkValueError`
    omit 809cec012055 [SPARK-46283][INFRA] Remove `streaming-kinesis-asl` module from `MODULES_TO_TEST` for branch-3.x daily tests
    omit 6b6980de451e [SPARK-46286][DOCS] Document `spark.io.compression.zstd.bufferPool.enabled`
    omit f4e41e0e318e [SPARK-46287][PYTHON][CONNECT] `DataFrame.isEmpty` should work with all datatypes
    omit f6861c3918bd [SPARK-46288][PS][TESTS] Remove unused code in `pyspark.pandas.tests.frame.*`
    omit 348d88161a9b [SPARK-46173][SQL] Skipping trimAll call during date parsing
    omit 59777222e726 [SPARK-45888][SS] Apply error class framework to State (Metadata) Data Source
    omit ea8b1392d847 [SPARK-46284][PYTHON][CONNECT] Add `session_user` function to Python
    omit b9cc813adf0b [SPARK-46281][PS][TESTS] Remove unused code in `pyspark.pandas.tests.computation.*`
    omit d6f61d4e9d58 [SPARK-46280][PS][TESTS] Move test_parity_frame_resample and test_parity_series_resample to `pyspark.pandas.tests.connect.resample`
    omit e7616dc95021 [SPARK-46276][PYTHON][TESTS] Improve test coverage of pyspark utils
    omit 032e78297b02 [SPARK-46260][PYTHON][SQL] DataFrame.withColumnsRenamed` should respect the dict ordering
    omit 35a99a87c51c [SPARK-46213][PYTHON] Introduce `PySparkImportError` for error framework
    omit 44036f2be3d5 [SPARK-46252][PYTHON][TESTS] Improve test coverage of memory_profiler.py
    omit 891a24e6a5ee [SPARK-46278][CONNECT][PS][TESTS] Re-organize `GroupByTests`
    omit 6cdca10f1484 [SPARK-46263][SQL][SS][ML][MLLIB][UI] Clean up `SeqOps.view` and `ArrayOps.view` conversions
    omit 6f79faadc016 [SPARK-44977][SPARK-46257][BUILD][CORE][SQL] Upgrade Derby to 10.16.1.1
    omit fbb4bcad088a [SPARK-46226][PYTHON] Migrate all remaining `RuntimeError` into PySpark error framework
    omit 5d47aae836f4 [SPARK-46044][PYTHON][TESTS] Improve test coverage of udf.py
    omit c9df53f5d3ce [SPARK-46241][PYTHON][CONNECT] Fix error handling routine so it wouldn't fall into infinite recursion
    omit 89673da0beb2 [SPARK-45807][SQL] Add createOrReplaceView(..) / replaceView(..) to ViewCatalog
    omit ef05fb632424 [SPARK-46268][PS][CONNECT][TESTS] Re-organize `StatsTests`
    omit a56ca8a673d7 [SPARK-46271][PS][TESTS] Remove unused properties in `pyspark.pandas.tests.groupby.*`
    omit e9ba896f8446 [SPARK-46266][PS][CONNECT][TESTS] Re-organize `NumOpsTests`
    omit 9808ad3ac322 [SPARK-46259][PYTHON][DOCS] Add an appropriate link for error class usage documentation
    omit 7c741c8c25fc [SPARK-46265][CONNECT] Assertions in AddArtifact RPC make the connect client incompatible with older clusters
    omit 8235f1d56bf2 [SPARK-46219][SQL] Unwrap cast in join predicates
    omit c8451b58a3c4 [SPARK-46069][SQL][FOLLOWUP] Make sure the cast expression is date type when unwrap timestamp type to date type
    omit b4bf9ec065bf [SPARK-46269][PS] Enable more NumPy compatibility function tests
    omit a85070baae74 [SPARK-46256][CORE] Parallel Compression Support for ZSTD
    omit 60bf6817d862 [SPARK-46262][PS] Enable test for `np.left_shift` for Pandas-on-Spark object
    omit 12817035c575 [SPARK-46264][PS][CONNECT][TESTS] Re-organize the resampling tests
    omit dd11075db618 [SPARK-46258][CORE] Add `RocksDBPersistenceEngine`
    omit 1d80d80a56c4 [SPARK-46249][SS] Require instance lock for acquiring RocksDB metrics to prevent race with background operations
    omit 3c8c7ad92cdc [SPARK-46255][PYTHON][CONNECT] Support complex type -> string conversion
    omit f3e113b687be [SPARK-46254][PYTHON] Remove stale Python 3.8/3.7 version checking
    omit 5fec76dc8db2 [SPARK-46043][SQL] Support create table using DSv2 sources
    omit 7f59565b9fc1 Revert "[SPARK-46213][PYTHON] Introduce `PySparkImportError` for error framework"
    omit f1283c126878 [SPARK-46009][SQL][CONNECT] Merge the parse rule of PercentileCont and PercentileDisc into functionCall
    omit 7af6a666859d [SPARK-46229][PYTHON][CONNECT][FOLLOW-UP] Remove unnecessary Python version check lower than 3.8
    omit 91a02e5d9701 [SPARK-46040][SQL][PYTHON] Update UDTF API for 'analyze' partitioning/ordering columns to support general expressions
    omit be49ca6dd71b [SPARK-46233][PYTHON] Migrate all remaining `AttributeError` into PySpark error framework
    omit 19e68ba34098 [SPARK-45684][SQL][SS][TESTS][FOLLOWUP] Use `++` instead of `s.c.SeqOps#concat`
    omit d5f2539f83f6 [SPARK-45940][FOLLOWUP][TESTS] Only test Python data source when Python and PySpark environments are available
    omit d314e4bba54a [SPARK-46250][CONNECT][SS] Deflake test_parity_listener
    omit 47e8e02141d4 [SPARK-46243][SQL][DOCS] Describe arguments of `decode()`
    omit ff4f59341215 [SPARK-46239][CORE] Hide `Jetty` info
    omit ac0bd2eb7b40 [SPARK-46245][CORE][SQL][SS][YARN][K8S][UI] Replcace `s.c.MapOps.view.filterKeys` with `s.c.MapOps.filter`
    omit 64ec5f1017e1 [SPARK-32246][BUILD][INFRA] Enable `streaming-kinesis-asl` tests in Github Action CI
    omit 9666bf37958e [SPARK-46231][PYTHON] Migrate all remaining `NotImplementedError` & `TypeError` into PySpark error framework
    omit 37d19b9ee0e4 [SPARK-46237][SQL][TESTS] Make `HiveDDLSuite` independently testable
    omit 2423a4517ace [SPARK-46234][PYTHON] Introduce `PySparkKeyError` for PySpark error framework
    omit be9a710efb7b [SPARK-46186][CONNECT] Fix illegal state transition when ExecuteThreadRunner interrupted before started
    omit 4398bb5d3732 [SPARK-46236][PYTHON][DOCS] Using brighter color for docs h3 title for better visibility
    omit fc56bb53910e [SPARK-46229][PYTHON][CONNECT] Add applyInArrow to groupBy and cogroup in Spark Connect
    omit b23ae15da019 [SPARK-46232][PYTHON] Migrate all remaining ValueError into PySpark error framework
    omit 6f112f7b1a50 [SPARK-46182][CORE] Track `lastTaskFinishTime` using the exact task finished event
    omit 712352e37ec5 [SPARK-40559][PYTHON][DOCS][FOLLOW-UP] Fix the docstring and document both applyInArrows
    omit 0c029e70706c [SPARK-46218][BUILD] Upgrade commons-cli to 1.6.0
    omit aee6b1582775 [SPARK-46227][SQL] Move `withSQLConf` from `SQLHelper` to `SQLConfHelper`
    omit 108e25086e07 [SPARK-46224][PYTHON][MLLIB][TESTS] Test string representation of TestResult (pyspark.mllib.stat.test)
    omit c6ad0b0825d7 [SPARK-40559][PYTHON][FOLLOW-UP] Fix linter for `getfullargspec`
    omit 91a43c6db185 [SPARK-46223][PS] Test SparkPandasNotImplementedError with cleaning up unused code
    omit 0f3c58aa9e7d [SPARK-46048][PYTHON][CONNECT][FOLLOWUP] Correct the string representation
    omit 52b94a6f6a93 [SPARK-46220][SQL] Restrict charsets in `decode()`
    omit 6ddd0081989e [SPARK-46222][PYTHON][TESTS] Test invalid error class (pyspark.errors.utils)
    omit 393c69a515b4 [SPARK-40559][PYTHON] Add applyInArrow to groupBy and cogroup
    omit 5031e52f9e03 [SPARK-46217][CORE][TESTS] Include `Driver/App` data in `PersistenceEngineBenchmark`
    omit 349037567a67 [SPARK-46221][PS][DOCS] Change `to_spark_io` to `spark.to_spark_io` in `quickstart_ps.ipynb`
    omit 400573d33af3 [SPARK-45093][CONNECT][PYTHON] Properly support error handling and conversion for AddArtifactHandler
    omit 3da2e5c63246 [SPARK-46216][CORE] Improve `FileSystemPersistenceEngine` to support compressions
    omit a3d70195dd52 [SPARK-46208][PS][DOCS] Adding a link for latest Pandas API specifications
    omit 73a09ed7bd73 [SPARK-46206][PS] Use a narrower scope exception for SQL processor
    omit 938b7f580519 [SPARK-46212][CORE][SQL][SS][CONNECT][MLLIB][GRAPHX][DSTREAM][PROTOBUF][EXAMPLES] Use other functions to simplify the code pattern of `s.c.MapOps#view.mapValues`
    omit 0e8e303fb4b2 [SPARK-46069][SQL] Support unwrap timestamp type to date type
    omit d1311f75dea2 [SPARK-46215][CORE] Improve `FileSystemPersistenceEngine` to allow nonexistent parents
    omit 0ddf4fdb99cb [SPARK-43228][SQL] Join keys also match PartitioningCollection in CoalesceBucketsInJoin
    omit 75b0eb2d6017 [SPARK-46213][PYTHON] Introduce `PySparkImportError` for error framework
    omit aa1eca8d26cc [MINOR][DOCS] Add `since` tag for `Scan.reportDriverMetrics`
    omit ccedc1988fa5 [SPARK-41532][CONNECT][PYTHON][FOLLOWUP] Expose `SessionNotSameException` as PySpark exceptions
    omit 37debbbbb1c1 [SPARK-46210][K8S][DOCS] Update `YuniKorn` docs with v1.4
    omit a6afd09f7f2f [SPARK-46205][CORE][TESTS][FOLLOWUP] Simplify PersistenceEngineBenchmark
    omit 8f8e41ea5203 [SPARK-46092][SQL] Don't push down Parquet row group filters that overflow
    omit 341b7d7a6e3a [SPARK-46214][BUILD] Upgrade commons-io to 2.15.1
    omit 0d40b1aea758 [SPARK-46205][CORE] Improve `PersistenceEngine` performance with `KryoSerializer`
    omit 8060e7e73170 [SPARK-46199][PYTHON][DOCS] Add PyPi link icon to PySpark doc header
    omit a4678143f6bb [SPARK-46201][PYTHON][DOCS] Correct the typing of `schema_of_{csv, json, xml}`
    omit c85d41c558b6 [SPARK-46200][INFRA] re-org the testing dockerfile
    omit d78126978799 [SPARK-46204][K8S] Fix `dev-run-integration-tests.sh` to use 17 for `JAVA_VERSION`
    omit 0e689611f099 [SPARK-46193][CORE][TESTS] Add `PersistenceEngineBenchmark`
    omit e93bff6fc0bc [SPARK-46187][SQL] Align codegen and non-codegen implementation of `StringDecode`
    omit 2ff9b7241e46 [SPARK-46197][INFRA] Upgrade `memory-profiler>=0.61.0` for Python 3.12
    omit f0ec157fe833 [SPARK-46196][PYTHON][DOCS] Add missing `toDegrees/toRadians/atan2/approxCountDistinct` function descriptions
    omit a0677302c62d [SPARK-46194][BUILD] Remove TODO comments left in SPARK-33775
    omit 9c6782fff9ec [SPARK-46055][SQL][FOLLOWUP] Respect code style for scala.
    omit a186324b9ad1 [SPARK-45629][CORE][SQL][CONNECT][ML][STREAMING][BUILD][EXAMPLES] Fix `Implicit definition should have explicit type`
    omit 042d8546be5d [SPARK-46189][PS][SQL] Perform comparisons and arithmetic between same types in various Pandas aggregate functions to avoid interpreted mode errors
    omit bfc5ad79bc81 [SPARK-46191][CORE] Improve `FileSystemPersistenceEngine.persist` error msg in case of the existing file
    omit 783c7bc0092a [SPARK-46184][CORE][SQL][CONNECT][MLLIB] Reduce stack depth by replace Option.isDefined with Option.isEmpty
    omit 74b55ed4f87f [SPARK-46181][INFRA][FOLLOW-UP] Use a proper JSON syntax to pass PYTHON_TO_TEST
    omit d14578812d2a [SPARK-46135][PYTHON][DOCS] Fix table format error in ipynb docs
    omit 29923d5d7e7b [MINOR][DOCS] Fix the result file description of CoalescedRDDBenchmark
    omit 2fad7b7ba28e [SPARK-45940][PYTHON] Add InputPartition to DataSourceReader interface
    omit 99b80a7f17e2 [SPARK-46188][DOC] Fix the CSS of Spark doc's generated tables
    omit 9bb358b51e30 [SPARK-46170][SQL] Support inject adaptive query post planner strategy rules in SparkSessionExtensions
    omit 67f628e3bf87 [SPARK-46145][SQL] spark.catalog.listTables does not throw exception when the table or view is not found
    omit c162f6df1b3d [SPARK-46144][SQL] Fail INSERT INTO ... REPLACE statement if the condition contains subquery
    omit f666f1aa1c1b [SPARK-46181][PYTHON][INFRA] Split scheduled Python build
    omit 02e175b91d6d [SPARK-46178][CONNECT][TESTS] Move MockDF under PlanOnlyTestFixture
    omit e86934b0787f [SPARK-46177][PYTHON][CONNECT][ML][TESTS] Skip 'CrossValidatorTests.test_crossvalidator_with_fold_col' with Python 3.12
    omit b4570794bb28 [MINOR][PYTHON][DOCS] Fix some outdated info for development/testing.rst
    omit 4303aa61c3a3 [SPARK-46151][PYTHON][DOCS] Hide the `More drop-down button` in the PySpark docs navigation bar
    omit cfcc25099834 [SPARK-46152][SQL] XML: Add DecimalType support in XML schema inference
    omit 350af641f981 [SPARK-46056][SQL] Fix Parquet vectorized read NPE with byteArrayDecimalType default value
    omit d1aea92daf25 [SPARK-45943][SQL] Move DetermineTableStats to resolution rules
    omit 6eaf0bd3bab2 [SPARK-45962][SQL][FOLLOW-UP] Handles whitespaces when `nullValue` option is enabled
    omit 9987cab84bb0 [SPARK-46060][BUILD][TESTS] Upgrade `MySQL/MariaDB/PostgreSQL/DB2` test dependencies
    omit c6793fd48475 [SPARK-46133][CORE][DOCS] Make `ShuffleWriteProcessor.write` method description up-to-date
    omit 06c8a5fe32e4 [SPARK-45267][PS][FOLLOWUP] Remove duplicated test
    omit 96507921b3b0 [SPARK-46169][PS] Assign appropriate JIRA numbers for missing parameters from `DataFrame` API
    omit 1bd2a975ba86 [SPARK-46172][SQL][DOCS] Describe arguments of `encode()`
    omit 7bdabaae32de [SPARK-46174][BUILD] Upgrade `gcs-connector` to 2.2.18
    omit 8a4890ddb554 [SPARK-46124][CORE][SQL][SS][CONNECT][DSTREAM][MLLIB][ML][PYTHON][R][AVRO][K8S][YARN][UI] Replace explicit `ArrayOps#toSeq` with `s.c.immutable.ArraySeq.unsafeWrapArray`
    omit f5e4e84ce3a7 [SPARK-45746][PYTHON] Return specific error messages if UDTF 'analyze' or 'eval' method accepts or returns wrong values
    omit 8bd9c8c00dba [SPARK-46155][INFRA] Upgrade github-script action to v7
    omit 3509174f7e5f [SPARK-46141][SQL] Change default for spark.sql.legacy.ctePrecedencePolicy to CORRECTED
    omit fa4096eb6aba [SPARK-46036][SQL] Removing error-class from raise_error function
    omit d5e54bbf91b3 [SPARK-46154][INFRA] Add a new Daily testing GitHub Action job for Maven with Java 21
    omit ad7e1bff534a [SPARK-46150][INFRA] Combine python codegen check and protobuf braking change
    omit 14d854beeabc [SPARK-46117][DOCS][PYTHON] Enhancing readability of PySpark API reference by hiding verbose typehints
    omit bbc33dfb4ca1 [SPARK-46149][ML][PYTHON][TESTS] Skip 'TorchDistributorLocalUnitTests.test_end_to_end_run_locally' with Python 3.12
    omit 131cc30479ee [SPARK-46148][PS] Fix pyspark.pandas.mlflow.load_model test (Python 3.12)
    omit 72534072b88b [SPARK-46146][INFRA] Unpin `markupsafe`
    omit f9e69642d380 [SPARK-46003][UI][TESTS] Create an ui-test module with Jest to test ui javascript code
    omit 587c863f261f [SPARK-46147][PS][TESTS] Fix the doctest in pyspark.pandas.series.Seires.to_dict (Python 3.12)
    omit 6a300ce9ed3c [SPARK-46140][SQL][TESTS] Remove comments about JVM module options from `HiveExternalCatalogVersionsSuite`
    omit d2cd98bdd324 [SPARK-46029][SQL] Escape the single quote, `_` and `%` for DS V2 pushdown
    omit 126b74a792c3 [SPARK-46055][SQL] Rewrite Catalog Database APIs implementation
    omit 1a73d52e7be5 [SPARK-46142][INFRA] Remove `dev/ansible-for-test-node` directory
    omit 9183ddec108d [SPARK-46134][PROTOBUF][TESTS] Replace `df.take(1).toSeq(0)` with `df.first()` in `ProtobufFunctionsSuite`
    omit 7a0d0411aa02 [SPARK-46101][CORE][SQL][MLLIB][SS][R][CONNCT][GRAPHX] Reduce stack depth by replace (string|array).size with (string|array).length
    omit ec71e223fc4f [SPARK-46129][DOCS][PYTHON] Add GitHub link icon to PySpark doc header
    omit 7b2f6bfa202d [SPARK-46119][SQL] Override `toString` method for `UnresolvedAlias`
    omit 931e68b88d0b [SPARK-46139][SQL][TESTS] Fix `QueryExecutionErrorsSuite` with Java 21
    omit 1ee3d2ae3b63 [SPARK-45827][SQL] Variant fixes with codegen and vectorized reader disabled
    omit a6cda2302c29 [SPARK-45760][SQL][FOLLOWUP] Inline With inside conditional branches
    omit f2ea75f24690 [SPARK-32407][SPARK-35375][FOLLOWUP] Delete remaining comments on `Sphinx` and `Jinja2`
    omit b5c94f1c02c6 [SPARK-46103][INFRA][FOLLOWUP] Unpin jinja2 in Python linter
    omit 04c9583cecdf [SPARK-45833][SS][DOCS] Document the new introduction of state data source
    omit 486439334702 [SPARK-46127][TESTS][PYTHON][FOLLOW-UP] Fix skip condition from " < (3, 12)" to " > (3, 11)"
    omit bfb08823b490 [SPARK-32407][SPARK-35375][INFRA][DOCS] Delete comments on `Sphinx` and `Jinja2`
    omit 984e797e02ca [SPARK-46131][PYTHON][INFRA] Install torchvision for Python 3.12 build
    omit 158f87621570 [SPARK-46127][PYTHON][TESTS] Disable pyspark.tests.test_worker.WorkerSegfaultNonDaemonTest.test_python_segfault with Python 3.12
    omit 2800b5849309 [SPARK-46116][DOCS][PYTHON] Adding "Stack Overflow" and "Mailing Lists" link into PySpark doc homepage
    omit dbc8756bdac8 [SPARK-46006][YARN][FOLLOWUP] YarnAllocator set target executor number to 0 to cancel pending allocate request when driver stop
    omit 4f59e1b66381 [SPARK-46126][PYTHON][TESTS] Fix the doctest in pyspark.pandas.frame.DataFrame.to_dict (Python 3.12)
    omit ec7d07c635c4 [SPARK-46111][DOCS][PYTHON] Add copyright to the PySpark official documentation
    omit 753b2f232064 [SPARK-46121][PYTHON][DOCS] Refine docstring of `concat/array_position/element_at/try_element_at`
    omit 2430e87ac939 [SPARK-46115][SQL] Restrict charsets in `encode()`
    omit 11ac85691981 [SPARK-46103][FOLLOWUP] Keep Sphinx version consistency in spark-rm
    omit 86971665ed47 [SPARK-45957][CONNECT] Avoid generating execution plan for non-executable commands
    omit e6edb1013b74 [SPARK-46089][BUILD] Upgrade commons-lang3 to 3.14.0
    omit 92e832a20bd0 [SPARK-46118][SQL][SS][CONNECT] Use `SparkSession.sessionState.conf` instead of `SQLContext.conf` and mark `SQLContext.conf` as deprecated
    omit 7d1db89358e0 [SPARK-46110][PYTHON] Use error classes in catalog, conf, connect, observation, pandas modules
    omit 8ad02304f873 [SPARK-46120][CONNECT][PYTHON] Remove helper function `DataFrame.withPlan`
    omit cd47a4c5e01e [SPARK-46123][DOCS][PYTHON] Using brighter color for document title for better visibility
    omit f9ea66819624 [SPARK-46113][BUILD][DOCS] Update `pydata_sphinx_theme` version requirement to `>=0.13`
    omit 8d646fe50965 [MINOR][PYTHON] Remove _inferSchema in SQLContext
    omit 685f536a5947 [MINOR][PYTHON][DOCS] Remove since decorator usages in pyspark.sql
    omit 08f23f7b6bee [MINOR][PYTHON] Remove obsolete TODOs for ignores at SQLContext and HiveContext
    omit 0193d0f88a95 [SPARK-46114][PYTHON] Add PySparkIndexError for error framework
    omit e1a2255f99be [SPARK-45699][BUILD][CORE][SQL][SS][CONNECT][MLLIB][ML][DSTREAM][GRAPHX][K8S][UI] Fixing all compilation warnings related to widening conversions
    omit ef27b9b15687 [SPARK-46099][PS][DOCS] Refactor "Supported pandas API" generation script
    omit d971dc461f5c [SPARK-46084][PS][FOLLOWUP] More refactoring by using `create_map`
    omit 400db88d00e5 [SPARK-46103][PYTHON][INFRA][BUILD][DOCS] Enhancing PySpark documentation
    omit d5fad6381014 [SPARK-46074][CONNECT][SCALA] Insufficient details in error message on UDF failure
    omit ade861d19910 [SPARK-45974][SQL] Add scan.filterAttributes non-empty judgment for RowLevelOperationRuntimeGroupFiltering
    omit d30c9a90c6cf [SPARK-45826][SQL] Add a SQL config for stack traces in DataFrame query context
    omit 7b58fffdeeb7 [SPARK-46100][CORE][PYTHON] Reduce stack depth by replace (string|array).size with (string|array).length
    omit 5211f6b140a7 [SPARK-46085][CONNECT] Dataset.groupingSets in Scala Spark Connect client
    omit a694a8a0be54 [SPARK-46095][DOCS] Document `REST API` for Spark Standalone Cluster
    omit 50f189b3f48a [SPARK-46066][SQL] Use the Separators API instead of the String API to construct the `DefaultPrettyPrinter`
    omit 2f6a38cfcb38 [SPARK-45922][CONNECT][CLIENT] Minor retries refactoring (follow-up to multiple policies)
    omit 132bb63a897f [SPARK-46016][DOCS][PS] Fix pandas API support list properly
    omit 441e15ef5f11 [SPARK-46087][PYTHON] Sync PySpark dependencies in docs and dev requirements
    omit 4530b12090ed [SPARK-45356][INFRA] Adjust the test configuration of `maven_test.yml`
    omit 6ad80fd03683 [SPARK-46084][PS] Refactor data type casting operation for Categorical type
    omit 1131a56d1f3c [SPARK-46073][SQL] Remove the special resolution of UnresolvedNamespace for certain commands
    omit 10b9c61ee9a7 [SPARK-46083][PYTHON] Make SparkNoSuchElementException as a canonical error API
    omit cd1ec48b2127 [SPARK-46082][PYTHON][CONNECT] Fix protobuf string representation for Pandas Functions API with Spark Connect
    omit 6c7ebe85f334 [SPARK-46067][BUILD] Upgrade commons-compress to 1.25.0
    omit 6e925a45b667 [MINOR][DOCS] Update Description of `UnknownException` in PySpark Documentation
    omit d3d35ab3e647 [SPARK-46080][PYTHON] Upgrade Cloudpickle to 3.0.0
    omit 4be5e218dc39 [SPARK-46081][INFRA] Set `DEDICATED_JVM_SBT_TESTS` in `build_java21.yml`
    omit 37f8221926b8 [SPARK-46079][INFRA] Install `torch` nightly only at Python 3.12 in Infra docker image
    omit 2b07d7786208 [SPARK-46076][PYTHON][TESTS] Remove `unittest` deprecated alias usage for Python 3.12
    omit 4670410f59ab [MINOR][CORE][TESTS] Move `TransportConfSuite.java` to correct directory
    omit e470e7442f90 [SPARK-45887][SQL] Align codegen and non-codegen implementation of `Encode`
    omit 240706600dbf [SPARK-46070][SQL] Compile regex pattern in SparkDateTimeUtils.getZoneId outside the hot loop
    omit 6e4d75a349f4 [SPARK-43980][SQL] introducing `select * except` syntax
    omit 43046631a5d4 [SPARK-46062][SQL] Sync the isStreaming flag between CTE definition and reference
    omit 9c21238ce0e6 [SPARK-46021][CORE] Support cancel future jobs belonging to a job group
    omit 4c36ca3f6577 [SPARK-46065][PS] Refactor `(DataFrame|Series).factorize()` to use `create_map`
    omit a703dace0aa4 [SPARK-46064][SQL][SS] Move out EliminateEventTimeWatermark to the analyzer and change to only take effect on resolved child
    omit 4221c66919d9 [SPARK-46002][INFRA][FOLLOWUP] Upgrade to Python 3.9 in daily maven test
    omit d14410c6777e [SPARK-46048][PYTHON][CONNECT] Support DataFrame.groupingSets in Python Spark Connect
    omit f6e4a4667057 [SPARK-46063][PYTHON][CONNECT] Improve error messages related to argument types in cute, rollup, groupby, and pivot
    omit 7b5dd1571977 [MINOR][BUILD] Remove scala-2.13 profile for Scalafmt in lint-scala script
    omit d9157101e260 [SPARK-46061][PYTHON][TESTS] Add the test party for reattach test case
    omit 7220d134670e [SPARK-46059][INFRA][PYTHON] Install `six==1.16.0` explicitly for `pandas` in Python 3.12
    omit c8ad616b988e [SPARK-45600][PYTHON] Make Python data source registration session level
    omit 9c291e1165c1 [SPARK-46042][FOLLOWUP][CONNECT] Test and adapt to streaming RPC behavior change from grpc 1.56 to 1.59
    omit 2966c791cb90 [SPARK-46042][CONNECT] Reenable a `releaseSession` test case in SparkConnectServiceE2ESuite
    omit c4a7588cbd5f [SPARK-46053][TESTS] Support `DEDICATED_JVM_SBT_TESTS`
    omit b7e480a43d6b [SPARK-46047][BUILD][SQL] Fix the usage of Scala 2.13 deprecated APIs and make the usage of Scala 2.13 deprecated APIs as compilation error
    omit 06635e25f170 [SPARK-46006][YARN] YarnAllocator miss clean targetNumExecutorsPerResourceProfileId after YarnSchedulerBackend call stop
    omit 90470ff0bd0b [SPARK-46048][PYTHON][SQL] Support DataFrame.groupingSets in PySpark
    omit b6d6cf30efb2 [SPARK-46045][DOCS][PS] Add individual categories for `Options and settings` to API reference
    omit 6622d56c0620 [SPARK-46038][BUILD] Upgrade log4j2 to 2.22.0
    omit 354a53742086 [SPARK-46035][BUILD] Upgrade zstd-jni to 1.5.5-10
    omit 7d3f5bde323b [SPARK-45697][BUILD] Fix `Unicode escapes in triple quoted strings are deprecated`
    omit cc6c10110f73 [SPARK-45696][CORE] Fix method tryCompleteWith in trait Promise is deprecated
    omit f5d84ef229a6 [SPARK-46022][SPARK-46017][DOCS][PYTHON] Remove deprecated function APIs from documents
    omit f0cdb0946f24 [SPARK-46013][PYTHON][SQL][DOCS] Improve data source load and save functions docs page
    omit 6be4a0358265 [SPARK-46012][CORE][FOLLOWUP] Invoke `fs.listStatus` once and reuse the result
    omit 7f9cc867ed26 [SPARK-46039][BUILD][CONNECT] Upgrade `grpcio*` to 1.59.3 for Python 3.12
    omit d8314c8d73cc [SPARK-45511][SS][FOLLOWUP] fix comment in StateDataSourceReadSuite
    omit 247ce5c84164 [MINOR][TEST] Relax `excludedTpcdsQueries` scope in `TPCDSBase`
    omit 2ac8ff76a516 [SPARK-44973][SQL] Fix `ArrayIndexOutOfBoundsException` in `conv()`
    omit 536445cc9544 [SPARK-46028][CONNECT][PYTHON] Make `Column.__getitem__` accept input column
    omit fdcd20f4b51c [SPARK-46019][SQL][TESTS] Fix `HiveThriftServer2ListenerSuite` and `ThriftServerPageSuite` to create `java.io.tmpdir` if it doesn't exist
    omit c8c97b7b8ab9 [SPARK-46031][SQL] Replace `!Optional.isPresent()` with `Optional.isEmpty()`
    omit b7930e718f45 [SPARK-46033][SQL][TESTS] Fix flaky ArithmeticExpressionSuite
    omit 7fc4aa3f5ff9 [SPARK-45533][CORE] Use j.l.r.Cleaner instead of finalize for RocksDBIterator/LevelDBIterator
    omit b4f01818713f [SPARK-46027][INFRA][FOLLOWUP] Fix a typo
    omit 90125b6c9151 [SPARK-46026][PYTHON][DOCS] Refine docstring of UDTF
    omit 4507b48771a4 [SPARK-46023][PYTHON][DOCS] Annotate parameters at docstrings in pyspark.sql module
    omit 0b7a537be557 [SPARK-46027][INFRA] Add Python 3.12 to the Daily Python Github Action job
    omit 483b80f18862 [SPARK-46024][PYTHON][DOCS] Document parameters and examples for RuntimeConf get, set and unset
    omit c476641812cc Revert "[SPARK-45649][SQL] Unify the prepare framework for `OffsetWindowFunctionFrame`"
    omit d908a212ef61 [MINOR][BUILD] Rename gprcVersion to grpcVersion in SparkBuild
    omit e4e84134e098 [SPARK-46004][PYTHON][DOCS] Refine docstring of `DataFrame.dropna/fillna/replace`
    omit 22932385df3f [SPARK-46020][INFRA] Add `Python 3.12` to Infra docker image
    omit df1280cb10ee [MINOR][DOCS] Correct Python Spark Connect documentation about pip installation
    omit f1a3a977fd9d [SPARK-46015][DOCS][PS] Remove broken link for Koalas issues
    omit 9bb1fe2a8410 [SPARK-46014][SQL][TESTS] Run `RocksDBStateStoreStreamingAggregationSuite` on a dedicated JVM
    omit 6ca1c67de082 [SPARK-46012][CORE] EventLogFileReader should not read rolling logs if app status file is missing
    omit 847199fb6d95 [SPARK-45929][SQL] Support groupingSets operation in dataframe api
    omit 434aa30d2349 [SPARK-45856] Move ArtifactManager from Spark Connect into SparkSession (sql/core)
    omit 2235fa9590f2 [SPARK-45827][SQL] Use StructNullableTypeConverter for Variant
    omit 6a0cb556f6a8 [MINOR][CORE][TESTS] Fix EventLogFileWritersSuite typo
    omit 25ee62e19bb0 [SPARK-45927][PYTHON] Update path handling for Python data source
    omit 5ded56735517 [SPARK-45998][CORE][SQL][PYTHON][PROTOBUF] Cleanup redundant type cast
    omit 839f0c98bd85 [SPARK-46002][INFRA] Upgrade to Python 3.9 in SQL test
    omit 8dcdac234ba9 [SPARK-46005][INFRA] Upgrade `scikit-learn` to 1.3.2 and `mlflow` to 2.8.1
    omit 9c8d418fff93 [SPARK-46007][TEST] Add `AmmoniteTest` test tag and mark `ReplE2ESuite`
    omit 398bff77b1c8 [SPARK-45999][PS] Use dedicated `PandasProduct` in `cumprod`
    omit 12707981ad78 [SPARK-45973][SPARK-45982][INFRA][FOLLOWUPS] Restore apt-get/r installations for branch-3.3/3.4/3.5
    omit 938d925e222a [SPARK-45764][FOLLOWUP][BUILD] Use latest `sphinx-copybutton` from CI
    omit 55ae57f725c9 [SPARK-45997][PS][PYTHON][CONNECT][DOCS] Remove deprecated APIs from legacy Koalas
    omit 2504fe1e2df7 [SPARK-45982][INFRA] re-org R package installations
    omit baff56d9d5fb [SPARK-45682][CORE][FOLLOW-UP] Restore console progress bar in Spark shells
    omit 70bb9319e504 [SPARK-45996][PYTHON][CONNECT] Show proper dependency requirement messages for Spark Connect
    omit 6f7ce917f311 [SPARK-45942][CORE] Only do the thread interruption check for putIterator on executors
    omit ed3a2edcf1b7 [SPARK-45995][R][INFRA] Upgrade R version from 4.3.1 to 4.3.2 in AppVeyor
    omit 51b8014c3b8e [SPARK-45986][ML][PYTHON] Fix `pyspark.ml.torch.tests.test_distributor` in Python 3.11
    omit cf970b76f946 [SPARK-45994][PYTHON] Change `description-file` to `description_file`
    omit 08f1d8b6ffed [SPARK-45988][SPARK-45989][PYTHON] Fix typehints to handle `list` GenericAlias in Python 3.11+
    omit 5384c50dd079 [SPARK-45992][BUILD] Switch from `fgrep` to `grep -F`
    omit dc68e4f7dc8f [SPARK-45965][DOCS][FOLLOW-UP] Document DSv2 partitioning expressions moved to functions.partitioning
    omit de17724308bd [SPARK-45991][BUILD] Upgrade `protobuf-java` to 3.25.1 to match with `protobuf` 4.25.1
    omit caec93ae70aa [SPARK-45990][SPARK-45987][PYTHON][CONNECT] Upgrade `protobuf` to 4.25.1 to support `Python 3.11`
    omit d31c8596cd71 [SPARK-45965][SQL] Move DSv2 partitioning expressions into functions.partitioning
    omit b2aa17ca5afd [SPARK-45985][PYTHON][DOCS] Refine docstring of `DataFrame.intersect`
    omit e6bbd3172940 [SPARK-45984][PYTHON[DOCS] Refine docstring of `DataFrame.intersectAll`
    omit 361072dad35c [SPARK-45983][PYTHON][DOCS] Refine docstring of `DataFrame.substract`
    omit c17b0922788f [SPARK-45967][BUILD] Upgrade jackson to 2.16.0
    omit 2055a76ba656 [SPARK-45980][INFRA] Add `Python 3.11` to the Daily Python Github Action job
    omit 17ff9511674d [SPARK-45956][BUILD] Upgrade `Apache ZooKeeper` to 3.9.1
    omit 8263c7fe0812 [SPARK-45971][CORE][SQL] Correct the package name of `SparkCollectionUtils` to `org.apache.spark.util`
    omit a15d399ba77d [SPARK-45681][UI][FOLLOWUP] Check maybeErrorClass instead of maybeErrorClass.length
    omit f20731dd68df [SPARK-45979][INFRA] Add `Python 3.11` to Infra docker image
    omit 00c749768843 [SPARK-45978][INFRA] Skip unidoc step in `docker-integration-tests` GitHub Action job
    omit e22e1acabc9b [SPARK-45975][TESTS] Reset storeAssignmentPolicy to original
    omit b6199fcb0f15 [SPARK-45973][INFRA] re-org `apt-get` installations
    omit 4520f3b2da01 [SPARK-45764][FOLLOWUP][PYTHON][DOCS] Remove upper bound for `sphinx-copybutton`
    omit 949de3416a8e [SPARK-45786][SQL][FOLLOWUP][TEST] Fix Decimal random number tests with ANSI enabled
    omit 2814293b2896 [SPARK-45962][SQL] Remove `treatEmptyValuesAsNulls` and use `nullValue` option instead in XML
    omit a7147c8e0471 [SPARK-45963][SQL][DOCS] Restore documentation for DSv2 API
    omit a5fe85fc1165 [SPARK-45966][DOCS][PS] Add missing methods for API reference
    omit 0c1ba5c1e64a [SPARK-45968][INFRA] Upgrade github docker action to latest version
    omit 6d25af4a4c81 [MINOR][SQL] Remove unimplemented instances in XML Data Source
    omit 7c146c925b36 [SPARK-45762][CORE] Support shuffle managers defined in user jars by changing startup order
    omit 8147620ac49e [SPARK-45964][SQL] Remove private sql accessor in XML and JSON package under catalyst package
    omit db0da0c0b52b [SPARK-45952][PYTHON][DOCS] Use built-in math constants in math functions
    omit fcf340a1de37 [SPARK-45912][SQL] Enhancement of XSDToSchema API: Change to HDFS API for cloud storage accessibility
    omit f6b670a650d2 [SPARK-45950][INFRA][CORE] Fix `IvyTestUtils#createIvyDescriptor` function and make `common-utils` module can run tests on GitHub Action
    omit 49cc081936c1 [SPARK-45960][INFRA] Add Python 3.10 to the Daily Python Github Action job
    omit c1dad17c48e1 [SPARK-45961][DOCS] Document `spark.master.*` configurations
    omit f78596264723 [SPARK-45958][BUILD] Upgrade Arrow to 14.0.1
    omit aff9eab90392 [SPARK-45511][SS] Fix state reader suite flakiness by clean up resources after each test run
    omit bbe95cfcd057 [SPARK-45946][SS] Fix use of deprecated FileUtils write to pass default charset in RocksDBSuite
    omit 0b7736c1d121 [SPARK-45953][INFRA] Add `Python 3.10` to Infra docker image
    omit fab408f018e8 [SPARK-45955][UI] Collapse Support for Flamegraph and thread dump details
    omit 334d952f9555 [SPARK-45951][INFRA] Upgrade `buf` to v1.28.1
    omit 182e2d236c5c [SPARK-45851][CONNECT][SCALA] Support multiple policies in scala client
    omit 1a651753f4e7 [SPARK-45945][CONNECT] Add a helper function for `parser`
    omit 79ccdfa31e28 [SPARK-45935][PYTHON][DOCS] Fix RST files link substitutions error
    omit daebb5996e20 [SPARK-45920][SQL] group by ordinal should be idempotent
    omit 17f84358fbfb [SPARK-45948][K8S] Make single-pod spark jobs respect `spark.app.id`
    omit 30c9c8dd9fe0 [SPARK-45919][CORE][SQL] Use Java 16 `record` to simplify Java class definition
    omit 352c55178e51 [SPARK-45949][INFRA] Upgrade `pyarrow` to 14
    omit 22f3221f5db8 [SPARK-45938][INFRA] Add `utils` to the dependencies of the `core/unsafe/network_common` module in `module.py`
    omit c91dbde2f94c [SPARK-33393][SQL] Support SHOW TABLE EXTENDED in v2
    omit bd14136b3978 [SPARK-45747][SS] Use prefix key information in state metadata to handle reading state for session window aggregation
    omit f7d56e2827c4 [SPARK-45827][SQL] Fix variant parquet reader
    omit ef93dc247844 [MINOR] Fix some typo
    omit 1a1d3034b1d7 [SPARK-45930][SQL] Support non-deterministic UDFs in MapInPandas/MapInArrow
    omit e795dc12833a [SPARK-45931][PYTHON][DOCS] Refine docstring of mapInPandas
    omit 2bb36fb271b6 [SPARK-45936][PS] Optimize `Index.symmetric_difference`
    omit e8c2a590f99d [SPARK-45934][DOCS] Fix `Spark Standalone` documentation table layout
    omit f6a59edc79ab [SPARK-45719][K8S][TESTS] Upgrade AWS SDK to v2 for Kubernetes IT
    omit 369834431127 [SPARK-45810][PYTHON] Create Python UDTF API to stop consuming rows from the input table
    omit d378b35f54b8 [SPARK-45868][CONNECT] Make sure `spark.table` use the same parser with vanilla spark
    omit aa646d305002 [SPARK-45941][PS] Upgrade `pandas` to version 2.1.3
    omit 39fc6108bfaa [SPARK-45562][DOCS] Regenerate `docs/sql-error-conditions.md` and add `42KDF` to `SQLSTATE table` in `error/README.md`
    omit 7120e6b88f23 [SPARK-45905][SQL] Least common type between decimal types should retain integral digits first
    omit 6c8bd94c5ed5 [SPARK-45915][SQL] Treat decimal(x, 0) the same as IntegralType in `PromoteStrings`
    omit d10f0eecf780 [SPARK-45764][PYTHON][DOCS] Make code block copyable
    omit afc4c49927cb [SPARK-43393][SQL] Address sequence expression overflow bug
    omit 86da6b83976d [MINOR][DOCS] Correct additional Conda documentation URL to fix 404 errors
    omit b2de91cb6c11 [SPARK-45918][PS] Optimize `MultiIndex.symmetric_difference`
    omit 121b6b89c28f [SPARK-45844][SQL][FOLLOWUP] Improve the caseSensitivityOrdering for XmlInferSchema
    omit 39c012ec5942 [SPARK-45562][SQL][FOLLOW-UP] XML: Fix SQLSTATE for missing rowTag error
    omit 3e9033a63c5f [SPARK-45913][PYTHON] Make the internal attributes private from PySpark errors
    omit 07ec264da9ed [SPARK-45813][CONNECT][PYTHON] Return the observed metrics from commands
    omit 74a9c6cfc6ea [SPARK-45511][SS] State Data Source - Reader
    omit f8ccf20360df [SPARK-45911][CORE] Make TLS1.3 the default for RPC SSL
    omit 128f5523194d [SPARK-45592][SPARK-45282][SQL] Correctness issue in AQE with InMemoryTableScanExec
    omit 2e22575da64f [SPARK-45921][INFRA] Use Hadoop `3.3.5` winutils in AppVeyor build
    omit 4049ca0786d2 [SPARK-45898][SQL] Rewrite catalog table APIs to use unresolved logical plan
    omit 2cac768fabaa [SPARK-45844][SQL] Implement case-insensitivity for XML
    omit aa10ac79fecd [SPARK-45827][SQL] Add Variant data type in Spark
    omit cd19d6c299a7 [SPARK-44733][PYTHON][DOCS] Add Python to Spark type conversion page to PySpark docs.
    omit ac293423543f [SPARK-45845][SS][UI] Add number of evicted state rows to streaming UI
    omit 961dcdfb9845 [SPARK-45882][SQL] BroadcastHashJoinExec propagate partitioning should respect CoalescedHashPartitioning
    omit 7cea52c96f5b [SPARK-45892][SQL] Refactor optimizer plan validation to decouple `validateSchemaOutput` and `validateExprIdUniqueness`
    omit a6b089fa00c2 [SPARK-45909][SQL] Remove `NumericType` cast if it can safely up-cast in `IsNotNull`
    omit 2b046d2eb5aa [SPARK-45907][CORE] Use  Java9+  ProcessHandle APIs to computeProcessTree in ProcfsMetricsGetter
    omit 1ece299b15fd [SPARK-45902][SQL] Remove unused function `resolvePartitionColumns` from `DataSource`
    omit d2a1a31a3800 [SPARK-45783][PYTHON][CONNECT] Improve error messages when Spark Connect mode is enabled but remote URL is not set
    omit c29b127dcdd9 [SPARK-45906][YARN] Fix error message extraction from ResourceNotFoundException
    omit 1a276bdb3d36 [SPARK-45723][PYTHON][CONNECT][FOLLOWUP] Replace `toPandas` with `_to_table` in catalog methods
    omit adcea1c37b80 [SPARK-45859][ML] Make UDF objects in ml.functions lazy
    omit bfbd3df69956 [MINOR][PYTHON] Better error message when Python worker crushes
    omit 9a990b5a40de [SPARK-45901][DOCS] Collect and update `spark-standalone.md` with new confs
    omit ef240cdf6eaa [SPARK-45733][CONNECT][PYTHON] Support multiple retry policies
    omit ba0e09830757 [SPARK-45899][CONNECT] Set errorClass in errorInfoToThrowable
    omit 3013d8b4d310 [SPARK-45857][SQL] Enforce the error classes in sub-classes of `AnalysisException`
    omit 4397c09bc0d7 [SPARK-45848][BUILD] Make `spark-version-info.properties` generated by `spark-build-info.ps1` include `docroot`
    omit 6e6a3a1f631a [SPARK-45830][CORE] Refactor `StorageUtils#bufferCleaner`
    omit 36d57f8a2973 [SPARK-45874][SQL] Remove Java version check from `IsolatedClientLoader`
    omit 9ab75f65cee6 [MINOR][CONNECT][TESTS]  Use `Some(())` instead of `Some()` for `Option[Unit]` return type
    omit e440f3245243 [SPARK-45896][SQL] Construct `ValidateExternalType` with the correct expected type
    omit 202884f4c571 [SPARK-45770][SQL][PYTHON][CONNECT] Introduce plan `DataFrameDropColumns` for `Dataframe.drop`
    omit 375aacd2e730 [SPARK-45886][SQL] Output full stack trace in `callSite` of DataFrame context
    omit 89ff8298dcc7 [SPARK-45875][CORE] Remove `MissingStageTableRowData` from `core` module
    omit eccf07694e3e [SPARK-45870][INFRA] Upgrade action/checkout to v4
    omit 2605b87990c9 [SPARK-45655][SQL][SS] Allow non-deterministic expressions inside AggregateFunctions in CollectMetrics
    omit f9c8c7a3312e [SPARK-45871][CONNECT] Optimizations collection conversion related to `.toBuffer` in the `connect` modules
    omit b501a223bfcf [MINOR][DOCS] Fix the example value in the docs
    omit 0a791993be7b [SPARK-45686][INFRA][CORE][SQL][SS][CONNECT][MLLIB][DSTREAM][AVRO][ML][K8S][YARN][PYTHON][R][UI][GRAPHX][PROTOBUF][TESTS][EXAMPLES] Explicitly convert `Array` to `Seq` when function input is defined as `Seq` to avoid compilation warnings related to `class LowPriorityImplicits2 is deprecated`
    omit 917947e62e1e [SPARK-45883][BUILD] Upgrade ORC to 1.9.2
    omit 605aa0c299c1 [SPARK-45687][CORE][SQL][ML][MLLIB][KUBERNETES][EXAMPLES][CONNECT][STRUCTURED STREAMING] Fix `Passing an explicit array value to a Scala varargs method is deprecated`
    omit 6851cb96ec65 [SPARK-45752][SQL] Simplify the code for check unreferenced CTE relations
    omit b34723773509 [SPARK-45878][SQL][TESTS] Fix ConcurrentModificationException in CliSuite
    omit 49ca6aa6cb75 [MINOR][SQL] Pass `cause` in `CannotReplaceMissingTableException` costructor
    omit bd526986a5f0 [SPARK-45837][CONNECT] Improve logging information in handling retries
    omit f5b1b8306cf1 [SPARK-45562][SQL] XML: Add SQL error class for missing rowTag option
    omit 9bac48d4bd68 [SPARK-45852][CONNECT][PYTHON] Gracefully deal with recursion error during logging
    omit 2085ac521757 [SPARK-45850][BUILD] Upgrade oracle jdbc driver to 23.3.0.23.09
    omit a566099133ff [SPARK-45756][CORE] Support `spark.master.useAppNameAsAppId.enabled`
    omit d9c5f9d6d42a [SPARK-45798][CONNECT] Assert server-side session ID
    omit ce818ba96953 [SPARK-45731][SQL] Also update partition statistics with `ANALYZE TABLE` command
    omit 7bc96e8e3767 [SPARK-45867][CORE] Support `spark.worker.idPattern`
    omit 1e93c408e19f [SPARK-45815][SQL][STREAMING] Provide an interface for other Streaming sources to add `_metadata` columns
    omit 5ac88b12f86b [SPARK-44886][SQL] Introduce CLUSTER BY clause for CREATE/REPLACE TABLE
    omit 06d8cbe07349 [SPARK-45847][SQL][TESTS] CliSuite flakiness due to non-sequential guarantee for stdout&stderr
    omit c128f811820e [SPARK-45814][CONNECT][SQL] Make ArrowConverters.createEmptyArrowBatch call close() to avoid memory leak
    omit 24edc0ef5bee [SPARK-45752][SQL] Unreferenced CTE should all be checked by CheckAnalysis0
    omit 093fbf1aa852 [SPARK-45831][CORE][SQL][DSTREAM] Use collection factory instead to create immutable Java collections
    omit 974313994a05 [SPARK-45835][INFRA] Make gitHub labeler more accurate and remove outdated comments
    omit dfd7cde91c5d [SPARK-45842][SQL] Refactor Catalog Function APIs to use analyzer
    omit 4df4fec622f3 [SPARK-45843][CORE] Support `killall` in REST Submission API
    omit eabea643c742 [SPARK-42821][SQL] Remove unused parameters in splitFiles methods
    omit 9d93b7112a31 [SPARK-45639][SQL][PYTHON] Support loading Python data sources in DataFrameReader
    omit e331de06dd05 [MINOR][CORE][SQL] Clean up expired comments: `Note: this class supports Scala 2.13. A parallel source tree has a 2.12 implementation.`
    omit b5408e1ce61c [SPARK-45828][SQL] Remove deprecated method in dsl
    omit 6abc4a1a58ef [SPARK-45841][SQL] Expose stack trace by `DataFrameQueryContext`
    omit f866549a5aa8 [SPARK-45816][SQL] Return `NULL` when overflowing during casting from timestamp to integers
    omit 1d8df4f6b99b [SPARK-45606][SQL] Release restrictions on multi-layer runtime filter
    omit a9127068194a [SPARK-45829][DOCS] Update the default value for spark.executor.logs.rolling.maxSize
    omit d582b74d97a7 [SPARK-45824][SQL] Enforce the error class in `ParseException`
    omit 44522756382d [SPARK-45794][SS] Introduce state metadata source to query the streaming state metadata information
    omit bfcd2c47742d [SPARK-45431][DOCS] Document new SSL RPC feature
    omit 5a49af205411 [SPARK-45555][PYTHON] Includes a debuggable object for failed assertion
    omit b0791b513da3 [SPARK-45223][PYTHON][DOCS] Refine docstring of `Column.when`
    omit 57fc1abc1ac8 [SPARK-45811][PYTHON][DOCS][FOLLOW-UP] Remove an example Spark Connect does not support
    omit 1ca543b5595e [SPARK-45808][CONNECT][PYTHON] Better error handling for SQL Exceptions
    omit beb0238273b9 [SPARK-45819][CORE] Support `clear` in REST Submission API
    omit 1157ffddde97 [SPARK-45186][PYTHON][DOCS] Refine docstring of `schema_of_xml`
    omit 6a44b627f40f [SPARK-45811][PYTHON][DOCS] Refine docstring of `from_xml`
    omit 8a9b44ea211e [SPARK-45809][PYTHON][DOCS] Refine docstring of `lit`
    omit 90c6c2b36743 [SPARK-45222][PYTHON][DOCS] Refine docstring of `DataFrameReader.json`
    omit 5d851d998905 [SPARK-45260][PYTHON][DOCS] Refine docstring of `count_distinct`
    omit 563b3cab749f [SPARK-45259][PYTHON][DOCS] Refine docstring of `count`
    omit b701d6e8951d [SPARK-45258][PYTHON][DOCS] Refine docstring of `sum`
    omit a0dea7ca40bc [SPARK-45802][CORE] Remove no longer needed Java `majorVersion` checks in `Platform`
    omit 50d137d752e5 [SPARK-44751][SQL] XML: Refactor TypeCast and timestampFormatter
    omit 5ef3a846f52a [SPARK-45786][SQL] Fix inaccurate Decimal multiplication and division results
    omit 01c294b05f3a [SPARK-45760][SQL] Add With expression to avoid duplicating expressions
    omit 2aa172dba117 [SPARK-45822][CONNECT] SparkConnectSessionManager may look up a stopped sparkcontext
    omit 8c4162903350 [SPARK-45509][SQL] Fix df column reference behavior for Spark Connect
    omit 1370e9de0e49 [SPARK-45013][TEST] Flaky Test with NPE: track allocated resources by taskId
    omit ff4ca0b1bcab [SPARK-45812][BUILD][PYTHON][PS] Upgrade Pandas to 2.1.2
    omit 296d3c50d2b6 [SPARK-45804][UI] Add spark.ui.threadDump.flamegraphEnabled config to switch flame graph on/off
    omit 370870b7a030 [SPARK-45803][CORE] Remove the no longer used `RpcAbortException`
    omit 2764ee0e9329 [SPARK-45805][SQL] Make `withOrigin` more generic
    omit 58c30ad4f6c5 [SPARK-44751][SQL] Move `XSDToSchema` from `catalyst` to `core` package
    omit 74f9ebeac8a3 [MINOR][INFRA] Correct Java version in RM Dockerfile description
    omit 82ffd2535202 [SPARK-45773][PYTHON][DOCS] Refine docstring of SparkSession.builder.config
    omit f6038302dd61 [SPARK-45793][CORE] Improve the built-in compression codecs
    omit fc867266f089 [SPARK-45758][SQL] Introduce a mapper for hadoop compression codecs
    omit 7c513807352b [MINOR][DOCS] Fix some spelling typos
    omit 9cbc2d138a1d [SPARK-45710][SQL] Assign names to error _LEGACY_ERROR_TEMP_21[59,60,61,62]
    omit 8553de36d8d2 [SPARK-45799][SQL] Remove some unused method in QueryExecutionErrors & QueryCompilationErrors
    omit 8e1cf59ab689 [SPARK-45790][INFRA] Move `graphx` to `mllib*` test pipeline
    omit 89cbe85b0b26 [SPARK-45779][INFRA][FOLLOWUP] Add more spaces to the prompts
    omit 6d669fa95746 [SPARK-45791][CONNECT][TESTS] Rename `SparkConnectSessionHodlerSuite.scala` to `SparkConnectSessionHolderSuite.scala`
    omit 749c79e6db1c [SPARK-45781][BUILD] Upgrade Arrow to 14.0.0
    omit 3363c2af3f6a [MINOR][CORE] Validate spark.deploy.defaultCores value during config setting
    omit 4f56e3852b92 [SPARK-45785][CORE] Support `spark.deploy.appNumberModulo` to rotate app number
    omit 31a81983fb2d [SPARK-45780][CONNECT] Propagate all Spark Connect client threadlocals in InheritableThread
    omit afdce266f0ff [SPARK-44843][TESTS] Double streamingTimeout for StateStoreMetricsTest to make RocksDBStateStore related streaming tests reliable
    omit da5aa706147c [SPARK-45779][INFRA] Add a check step for jira issue ticket to be resolved
    omit ec875c521fee [SPARK-45688][SPARK-45693][CORE] Clean up the deprecated API usage related to `MapOps` & Fix `method += in trait Growable is deprecated`
    omit e9339e60dd41 [SPARK-45774][CORE][UI] Support `spark.master.ui.historyServerUrl` in `ApplicationPage`
    omit 1359c1327345 [SPARK-45776][CORE] Remove the defensive null check for `MapOutputTrackerMaster#unregisterShuffle` added in SPARK-39553
    omit 1649b256133f [SPARK-45742][CORE][FOLLOWUP] Remove unnecessary null check from `ArrayImplicits.SparkArrayOps#toImmutableArraySeq`
    omit b9d379a6b84b [SPARK-45777][CORE] Support `spark.test.appId` in `LocalSchedulerBackend`
    omit 0ebfa691fc8e [SPARK-45730][CORE] Make ReloadingX509TrustManagerSuite less flaky
    omit bc9c255e28c3 [SPARK-45694][SPARK-45695][SQL] Clean up deprecated API usage `View.force` and `ScalaNumberProxy.signum`
    omit eda9911057b8 [SPARK-45768][SQL][PYTHON] Make faulthandler a runtime configuration for Python execution in SQL
    omit cdc66a71ea3d [SPARK-45757][ML] Avoid re-computation of NNZ in Binarizer
    omit eba37d2d408b [SPARK-45718][PS] Remove remaining deprecated Pandas features from Spark 3.4.0
    omit 653b31e18b3f [SPARK-45771][CORE] Enable `spark.eventLog.rolling.enabled` by default
    omit 5970d353360d [SPARK-45767][CORE] Delete `TimeStampedHashMap` and its UT
    omit 30ec6e358536 [SPARK-45742][CORE][CONNECT][MLLIB][PYTHON] Introduce an implicit function for Scala Array to wrap into `immutable.ArraySeq`
    omit a04d4e2233c0 [SPARK-45761][K8S][INFRA][DOCS] Upgrade `Volcano` to 1.8.1
    omit 59e291d36c4a [SPARK-45680][CONNECT] Release session
    omit b14c1f036f8f [SPARK-45763][CORE][UI] Improve `MasterPage` to show `Resource` column only when it exists
    omit d6b53c34c5c5 [SPARK-45754][CORE] Support `spark.deploy.appIdPattern`
    omit 3cac0df1f5f8 [SPARK-45327][BUILD] Upgrade zstd-jni to 1.5.5-7
    omit 63cf0fec05e6 [SPARK-45753][CORE] Support `spark.deploy.driverIdPattern`
    omit 86510dbd8949 [SPARK-45743][BUILD] Upgrade dropwizard metrics 4.2.21
    omit c7bba9bfcc35 [SPARK-45755][SQL] Improve `Dataset.isEmpty()` by applying global limit `1`
    omit e6b4fa835de3 [SPARK-45751][DOCS] Update the default value for spark.executor.logs.rolling.maxRetainedFile
    omit feea99e9d8c1 [SPARK-45022][SQL] Provide context for dataset API errors
    omit e1bc48b729e4 [SPARK-45704][BUILD] Fix compile warning - using symbols inherited from a superclass shadow symbols defined in an outer scope
    omit 0a7023f55ddc [SPARK-45734][BUILD] Upgrade commons-io to 2.15.0
    omit f72510ca9e04 [SPARK-45749][CORE][WEBUI] Fix `Spark History Server` to sort `Duration` column properly
    omit 14d347c6c851 [SPARK-45654][PYTHON] Add Python data source write API
    omit 45f73bc69655 [SPARK-43380][SQL][FOLLOW-UP] Fix slowdown in Avro read
    omit 285e97465a53 [MINOR][DOCS] Fix the variable name in the docs - testing_pyspark.ipynb
    omit f7e70e047a0a [SPARK-45741][BUILD] Upgrade Netty to 4.1.100.Final
    omit 18e07956e947 [SPARK-45739][PYTHON] Catch IOException instead of EOFException alone for faulthandler
    omit af8907a0873f [SPARK-45242][SQL][FOLLOWUP] Canonicalize DataFrame ID in CollectMetrics
    omit 49f9e74973fa [SPARK-45481][SPARK-45664][SPARK-45711][SQL][FOLLOWUP] Avoid magic strings copy from parquet|orc|avro compression codes
    omit 936c1cd22b8c [SPARK-45737][SQL] Remove unnecessary `.toArray[InternalRow]` in `SparkPlan#executeTake`
    omit 9169d93ed5b8 [SPARK-45700][SPARK-45702][SPARK-45703] Fix the compile warning suppression rule added in SPARK-35496
    omit 0487507c8fe0 [SPARK-45683][CORE] Fix `method any2stringadd in object Predef is deprecated`
    omit 5ba708f8fffd [SPARK-45725][SQL] Remove the non-default IN subquery runtime filter
    omit 102daf9d1490 [SPARK-45368][SQL] Remove scala2.12 compatibility logic for DoubleType, FloatType, Decimal
    omit d38f07459494 [SPARK-45732][BUILD] Upgrade commons-text to 1.11.0
    omit 8dd3ec87e269 [SPARK-45713][PYTHON][FOLLOWUP] Fix SparkThrowableSuite for GA
    omit 76d9a70932df [SPARK-45735][PYTHON][CONNECT][TESTS] Reenable CatalogTests without Spark Connect
    omit 29eb577cefab [SPARK-45701][SPARK-45684][SPARK-45692][CORE][SQL][SS][ML][K8S] Clean up the deprecated API usage related to `mutable.SetOps/c.SeqOps/Iterator/Iterable/IterableOps`
    omit 97ba9d29cd1f [SPARK-38723][SS][TEST][FOLLOWUP] Deflake the newly added test in QueryExecutionErrorsSuite
    omit 6cb0cd5bcb50 [SPARK-45713][PYTHON] Support registering Python data sources
    omit 2fd1f10a1642 [SPARK-45729][PYTHON][DOCS] Fix PySpark testing guide links
    omit fc6a5cca06cf [SPARK-45727][SS] Remove unused map in watermark propagation simulation
    omit 2be03d81cea3 [SPARK-45592][SQL] Correctness issue in AQE with InMemoryTableScanExec
    omit 311602a849a3 [SPARK-45728][BUILD][K8S] Upgrade `kubernetes-client` to 6.9.1
    omit 320694c87314 [SPARK-45245][PYTHON][CONNECT] PythonWorkerFactory: Timeout if worker does not connect back
    omit 2ad79e196a60 [SPARK-45506][CONNECT] Add ivy URI support to SparkConnect addArtifact
    omit c381480b4ff5 [SPARK-45481][SQL][FOLLOWUP] Add `lowerCaseName` for `ParquetCompressionCodec`
    omit 7e689c7ed548 [SPARK-45689][SPARK-45690][SPARK-45691][CORE][SQL] Clean up the deprecated API usage related to `StringContext/StringOps/RightProjection/LeftProjection/Either` and type use of `BufferedIterator/CanBuildFrom/Traversable`
    omit dc02991fa662 [SPARK-43380][SQL][FOLLOWUP] Deprecate toSqlType(avroSchema: Schema, …useStableIdForUnionType: Boolean): SchemaType
    omit 99da659dbafa [SPARK-45681][UI] Clone a js version of UIUtils.errorMessageCell for consistent error rendering on UI
    omit e499ba80be68 [SPARK-45723][PYTHON][CONNECT] Catalog methods avoid pandas conversion
    omit b92265a98f24 [SPARK-45685][CORE][SQL] Use `LazyList` instead of `Stream`
    omit 4339e0c0e4d7 [SPARK-45707][SQL] Simplify `DataFrameStatFunctions.countMinSketch` with `CountMinSketchAgg`
    omit 59a9ee776570 [SPARK-45682][CORE][SQL][ML][MLLIB][GRAPHX][YARN][DSTREAM][UI][EXAMPLES] Fix "method + in class Byte/Short/Char/Long/Double/Int is deprecated"
    omit f12bc05e4409 [SPARK-45664][SQL] Introduce a mapper for orc compression codecs
    omit 00d2b4fa2def [SPARK-44790][SQL] XML: to_xml implementation and bindings for python, connect and SQL
    omit 0245b842ccc4 [SPARK-45554][PYTHON] Introduce flexible parameter to `assertSchemaEqual`
    omit 4af4ddea116d [SPARK-45552][PS] Introduce flexible parameters to `assertDataFrameEqual`
    omit 884f6f711721 [SPARK-45544][CORE] Integrate SSL support into TransportContext
    omit 89ca8b6065e9 [SPARK-45605][CORE][SQL][SS][CONNECT][MLLIB][GRAPHX][DSTREAM][PROTOBUF][EXAMPLES] Replace `s.c.MapOps.mapValues` with `s.c.MapOps.view.mapValues`
    omit 4ae99f9320ca [SPARK-45636][BUILD] Upgrade jersey to 2.41
    omit 1f8422b77ae9 [SPARK-45674][CONNECT][PYTHON] Improve error message for JVM-dependent attributes on Spark Connect
    omit e46c8ae84e89 [SPARK-45711][SQL] Introduce a mapper for avro compression codecs
    omit 57e73dafeb9f [SPARK-45678][CORE] Cover BufferReleasingInputStream.available/reset under tryOrFetchFailedException
    omit aa232f214878 [SPARK-45705][TESTS] Fix flaky test: Status of a failed DDL/DML with no jobs should be FAILED
    omit 3d6e31a13a4b [SPARK-45649][SQL] Unify the prepare framework for `OffsetWindowFunctionFrame`
    omit f1d1dc1f87a7 [SPARK-45706][PYTHON][DOCS] Makes entire Binder build fails fast during setting up
    omit efa891c4ea86 [SPARK-45328][DOCS][FOLLOWUP] Update docs for Hive metastore supported versions
    omit d072833d50ab [SPARK-45698][CORE][SQL][SS] Clean up the deprecated API usage related to `Buffer`
    omit 7e5ee50d3d52 [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow
    omit 23c4cce61b78 [SPARK-45614][SQL] Assign names to error _LEGACY_ERROR_TEMP_215[6,7,8]
    omit e1dc571051de [SPARK-40820][PYTHON][SQL] Creating StructType from Json
    omit 7d94c5769a8b [SPARK-43380][SQL] Fix slowdown in Avro read
    omit 0f145f514e35 [SPARK-45667][CORE][SQL][CONNECT] Clean up the deprecated API usage related to `IterableOnceExtensionMethods`
    omit 62a3868b93a5 [SPARK-45481][SQL] Introduce a mapper for parquet compression codecs
    omit 929405ab7532 [SPARK-45575][SQL] Support time travel options for df read API
    omit 519b582ca0e8 [SPARK-45679][CONNECT] Add proto for clusterBy
    omit df0262f29969 Revert "[SPARK-45651][BUILD] Log memory usage of publish snapshot workflow"
    omit 0d665fe8c87b Revert "[SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow"
    omit d948be6c963d [SPARK-45677] Better error logging for Observation API
    omit 00c638cd4d91 [SPARK-45651][BUILD][FOLLOWUP] Reduce mvn -Xmx option to 2g in publish_snapshot workflow
    omit 536439244593 [SPARK-45670][CORE] SparkSubmit does not support `--total-executor-cores` when deploying on K8s
    omit 8c11804c4406 [SPARK-45652][SQL] SPJ: Handle empty input partitions after dynamic filtering
    omit f2672fcf3cf3 [SPARK-43923][CONNECT][FOLLOWUP] Correct the message abbreviation
    omit d12646ee224f [SPARK-45596][CONNECT] Use java.lang.ref.Cleaner instead of org.apache.spark.sql.connect.client.util.Cleaner
    omit 261b281e6e57 [SPARK-45659][SQL][SS] Add `since` field to Java API marked as `@Deprecated`
    omit 7d7afb06f682 [SPARK-38723][SS][TESTS] Add test for streaming query resume race condition
    omit a3146c83d98f [SPARK-45663][CORE][MLLIB] Replace `IterableOnceOps#aggregate` with `IterableOnceOps#foldLeft`
    omit 35c628d9b27a [SPARK-44407][BUILD][TESTS] Clean up the compilation warnings related to `it will become a keyword in Scala 3` and prohibit use these keywords as variable name
    omit 8cdcfd262f9f [SPARK-45665][INFRA] Uses different `ORACLE_DOCKER_IMAGE_NAME` in the scheduled builds in other branches
    omit 25c74d0d4e28 [SPARK-45635][PYTHON][TESTS] Cleanup unused import for PySpark testing
    omit dcbe275543e0 [SPARK-45634][PS] Remove `DataFrame.get_dtype_counts` from Pandas API on Spark
    omit e57a16d7af14 [SPARK-45651][BUILD] Log memory usage of publish snapshot workflow
    omit 3e2b146eb81d [SPARK-45136][CONNECT] Enhance ClosureCleaner with Ammonite support
    omit 0219eb5984f0 [SPARK-45661][SQL][PYTHON] Add toNullable in StructType, MapType and ArrayType
    omit 344453761cbc [SPARK-45660] Re-use Literal objects in ComputeCurrentTime rule
    omit 94607dd001b1 [SPARK-40154][PYTHON][DOCS] Correct storage level in Dataframe.cache docstring
    omit a073bf38c7d8 [SPARK-45209][CORE][UI] Flame Graph Support For Executor Thread Dump Page
    omit 7db9b2293fa7 [SPARK-45656][SQL] Fix observation when named observations with the same name on different datasets
    omit 08c7bacb4f9d [SPARK-45545][CORE] Pass SSLOptions wherever we create a SparkTransportConf
    omit 093b7e1846b4 [SPARK-45648][INFRA] Add `sql/api` and `common/utils` to `modules.py`
    omit bb77f5d91738 [SPARK-45651][BUILD] Publish snapshot manually
    omit a6503b22169d [SPARK-45640][SQL][TESTS] Fix flaky ProtobufCatalystDataConversionSuite
    omit 4eac0244c765 [SPARK-45503][SS] Add Conf to Set RocksDB Compression
    omit 68c0f64dddc9 [SPARK-45523][PYTHON] Refactor the null-checking to have shortcuts
    omit 7ef96ee5b888 [SPARK-45524][PYTHON][SQL] Initial support for Python data source read API
    omit 840306d867e2 [SPARK-45643][CORE][SQL] Replace `s.c.mutable.MapOps#transform` with `s.c.mutable.MapOps#mapValuesInPlace`
    omit 5cbfb07f94c9 [SPARK-45452][SQL][FOLLOWUP] Do not fail too early if FileSystem does not implement getSchema
    omit 7bd149b60da1 [SPARK-45646][SQL] Remove hardcoding time variables prior to Hive 2.0
    omit 5092c8970246 [SPARK-45561][SQL] Add proper conversions for TINYINT in MySQLDialect
    omit a484826244af [SPARK-44752][SQL] XML: Update Spark Docs
    omit 00ee0e397836 [SPARK-45641][UI] Display the application start time on AllJobsPage
    omit 38222c4e8c58 [SPARK-45626][SQL] Convert _LEGACY_ERROR_TEMP_1055 to REQUIRES_SINGLE_PART_NAMESPACE
    omit 1f396f5aaf92 [SPARK-45630][CORE][SQL][SS][YARN] Replace `mutable.MapOps#retain` with `mutable.MapOps#filterInPlace`
    omit 32e1e5841191 [SPARK-45430] Fix for FramelessOffsetWindowFunction when IGNORE NULLS and offset > rowCount
    omit 28ac70241612 [SPARK-44032][BUILD] Remove `threeten-extra` exclusion in `enforceBytecodeVersion` rule
    omit 6e6aa2e0f527 [SPARK-45632][SQL] Table cache should avoid unnecessary ColumnarToRow when enable AQE
    omit 43852733307a [SPARK-45618][CORE] Remove BaseErrorHandler
    omit 2daa66ec7856 [SPARK-45638][SQL][AVRO] Improve test coverage of decimal conversion
    omit 0cbe863e77c0 [SPARK-45547][ML] Validate Vectors with built-in function
    omit f12d2441a1b9 [SPARK-45619][CONNECT][PYTHON] Apply the observed metrics to Observation object
    omit 9aabc527ec27 [SPARK-45620][PYTHON] Fix user-facing APIs related to Python UDTF to use camelCase
    omit ae100267c28b [SPARK-44735][SQL] Add warning msg when inserting columns with the same name by row that don't match up
    omit e40e74497eb1 [SPARK-45623][INFRA] Move `mllib` and `mllib-local` to separate test group
    omit e2af8aba8b44 [SPARK-45627][BUILD] Fix `symbol literal is deprecated`
    omit 75bc5ac9f2b0 [SPARK-45631][SS][PYSPARK] Remove @abstractmethod from onQueryIdle in PySpark StreamingQueryListener
    omit 57d7fc390a34 [SPARK-45633][PYTHON][SQL][TESTS] Make SQL test group depend on `analyze_udtf.py`
    omit 3de9ebccaa3b Revert "[SPARK-45620][PYTHON] Fix user-facing APIs related to Python UDTF to use camelCase"
    omit 959382f7fbc2 [SPARK-45628][ML] Improve `vector_to_array` for `.mllib.linalg.SparseVector`
    omit dbf2f4943c17 [SPARK-45625][BUILD] Upgrade log4j to 2.21.0
    omit 376de8a502fc [SPARK-45616][CORE] Avoid ParVector, which does not propagate ThreadLocals or SparkSession
    omit e3ba9cf0403a [SPARK-45620][PYTHON] Fix user-facing APIs related to Python UDTF to use camelCase
    omit 9f675c54a56e [SPARK-44753][PYTHON][CONNECT] XML: pyspark sql xml reader writer
    omit 7763c91950e0 [SPARK-45624][CORE][TESTS] Use `AccessibleObject#canAccess` instead of `AccessibleObject#isAccessible`
    omit 48e207f4a219 [SPARK-45610][BUILD][CORE][SQL][SS][CONNECT][GRAPHX][DSTREAM][ML][MLLIB][K8S][YARN][SHELL][PYTHON][R][AVRO][UI][EXAMPLES] Fix the compilation warning "Auto-application to `()` is deprecated" and turn it into a compilation error
    omit 2709426f0f62 [SPARK-45541][CORE] Add SSLFactory
    omit 936e98dbc073 [SPARK-45574][SQL] Add :: syntax as a shorthand for casting
    omit 13b67ee8cc37 [SPARK-45604][SQL] Add LogicalType checking on INT64 -> DateTime conversion on Parquet Vectorized Reader
    omit 4023ec9bb447 [SPARK-45484][SQL][FOLLOWUP][DOCS] Update the document of parquet compression codec
    omit 920fb673b264 [MINOR] Fix typos
    omit 227cd8bb8f5d [SPARK-45523][PYTHON] Return useful error message if UDTF returns None for any non-nullable column
    omit 0a45a8662dbd [SPARK-30848][SQL] Remove productHash from TreeNode
    omit 77c790cdfec8 [SPARK-45602][CORE][SQL][SS][YARN][K8S] Replace `s.c.MapOps.filterKeys` with `s.c.MapOps.view.filterKeys`
    omit 4abab8b83085 [SPARK-45595] Expose SQLSTATE in error message
    omit fa8f4b1edd67 [SPARK-45609][CONNECT] Include SqlState in SparkThrowable proto message
    omit b7121e1cfe75 [SPARK-45613][CORE] Expose DeterministicLevel as a DeveloperApi
    omit 3c847c6c11f6 [SPARK-44837][SQL] Change ALTER TABLE ALTER PARTITION column error message
    omit 28292d51e7db [SPARK-45591][BUILD] Upgrade ASM to 9.6
    omit b9ac7d38ea7f [SPARK-45484][SQL] Fix the bug that uses incorrect parquet compression codec lz4raw
    omit cd827d14bcb2 [SPARK-45562][SQL][FOLLOWUP] Restore test case not require `rowTag` option
    omit 820418aec1cd [SPARK-45603][INFRA] merge_spark_pr shall notify us about GITHUB_OAUTH_KEY expiry
    omit 8fd915ffaba1 [SPARK-45584][SQL] Fix subquery execution failure with TakeOrderedAndProjectExec
    omit 63bfc10ff97c [SPARK-45428][PYTHON][DOCS] Add Matomo analytics to pyspark docs pages
    omit 6953b0037975 [SPARK-45611][PYTHON][DOCS] Typo fixed yyy to yyyy at date_format function
    omit 69fa51cc21f6 [SPARK-45569][SQL] Assign name to the error _LEGACY_ERROR_TEMP_2152
    omit d6d4e52ecc30 [SPARK-45543][SQL] `InferWindowGroupLimit` causes bug if the other window functions haven't the same window frame as the rank-like functions
    omit eda850708e30 [SPARK-45392][SQL][FOLLOWUP] Restore the original exception from data source
    omit 7057952f6bc2 [SPARK-45569][SQL] Assign name to the error _LEGACY_ERROR_TEMP_2153
    omit db162369941d [SPARK-45220][PYTHON][DOCS] Refine docstring of DataFrame.join
    omit 47d3fa83e70e [SPARK-45589][PYTHON][DOCS] Supplementary exception class
    omit e359f210c45a [SPARK-45588][PROTOBUF][CONNECT][MINOR] Scaladoc improvement for StreamingForeachBatchHelper
    omit 3042ffd08fbd [MINOR][CONNECT] Fix a typo in org.apache.spark.sql.connect.client.ArtifactManager
    omit f3e280b952da [SPARK-45553][PS] Deprecate `assertPandasOnSparkEqual`
    omit e37dd3ab8e07 Revert "[SPARK-45546][BUILD][INFRA] Make `publish-snapshot` support `package` first then `deploy`"
    omit 706872d4de23 Revert "[SPARK-45546][BUILD][FOLLOW-UP] Remove snapshot repo mistakenly added"
    omit a3c17b2e2296 [SPARK-45562][SQL][FOLLOW-UP] XML: Make 'rowTag' option check case insensitive
    omit 8a972c2fe873 [SPARK-45507][SQL] Correctness fix for nested correlated scalar subqueries with COUNT aggregates
    omit 1e94415739cc [SPARK-45586][SQL] Reduce compiler latency for plans with large expression trees
    omit a14f90941caf [SPARK-45585][TEST] Fix time format and redirection issues in SparkSubmit tests
    omit b42ac52e787c [SPARK-45582][SS] Ensure that store instance is not used after calling commit within output mode streaming aggregation
    omit 7e82e1bc43e0 [SPARK-45581] Make SQLSTATE mandatory
    omit b41ea9162f4c [SPARK-45549][CORE] Remove unused `numExistingExecutors` in `CoarseGrainedSchedulerBackend`
    omit 4a35a31c038f [SPARK-45370][PYTHON][TESTS] Fix python test when ansi mode enabled
    omit 69bb68bc997c [SPARK-45410][INFRA][FOLLOW-UP] Rename "Python - PyPy3.8 (master)" to "Build / Python-only (master, PyPy 3.8)"
    omit 5d5a9dbfba18 [SPARK-45542][CORE] Replace `setSafeMode(HdfsConstants.SafeModeAction, boolean)` with `setSafeMode(SafeModeAction, boolean)`
    omit 103de914a5f9 [SPARK-44649][SQL] Runtime Filter supports passing equivalent creation side expressions
    omit 3005dc890848 [SPARK-45558][SS] Introduce a metadata file for streaming stateful operator
    omit d7d38fbc1843 [SPARK-45587][INFRA] Skip UNIDOC and MIMA in `build` GitHub Action job
    omit b1e57a2b359d [SPARK-45534][CORE] Use java.lang.ref.Cleaner instead of finalize for RemoteBlockPushResolver
    omit 11e7ea4f11df [SPARK-45035][SQL] Fix ignoreCorruptFiles/ignoreMissingFiles with multiline CSV/JSON will report error
    omit fbf8b7be090b [SPARK-45576][CORE][FOLLOWUP] Remove unused imports to fix Java linter errors
    omit 73f9f5296e36 [SPARK-45500][CORE][WEBUI][FOLLOWUP] Show `RELAUNCHING` drivers too
    omit f4bd99da12f7 [SPARK-45009][SQL][FOLLOW UP] Turn off decorrelation in join conditions for AQE InSubquery test
    omit 4901548d4c36 [SPARK-45576][CORE] Remove unnecessary debug logs in ReloadingX509TrustManagerSuite
    omit 74dc5a3d8c0f [SPARK-45546][BUILD][FOLLOW-UP] Remove snapshot repo mistakenly added
    omit 2189f9bf2894 [MINOR][INFRA] Skip if JIRA ID is an empty string
    omit 0cb4a84f6ab0 [MINOR][DOCS] Update the docs for spark.sql.optimizer.canChangeCachedPlanOutputPartitioning configuration
    omit 3ef18e2d00f3 [SPARK-45546][BUILD][INFRA] Make `publish-snapshot` support `package` first then `deploy`
    omit 3a3b8c14b8c0 [SPARK-45578][CORE] Remove `InaccessibleObjectException` usage by using `trySetAccessible`
    omit cf59b1f51c16 [MINOR][SQL] Remove signature from Hive thriftserver exception
    omit 4a0ed9cd7253 [SPARK-45577][PYTHON] Fix UserDefinedPythonTableFunctionAnalyzeRunner to pass folded values from named arguments
    omit b10fea96b5b0 [SPARK-45566][PS] Support Pandas-like testing utils for Pandas API on Spark
    omit f1ae56b152bd [MINOR][DOCS] Fix one typo
    omit 922844fff65a [SPARK-45564][SQL] Simplify 'DataFrameStatFunctions.bloomFilter' with 'BloomFilterAggregate' expression
    omit f00ec39542a5 [SPARK-45568][TESTS] Fix flaky WholeStageCodegenSparkSubmitSuite
    omit 3b46cc816143 [SPARK-45512][CORE][SQL][SS][DSTREAM] Fix compilation warnings related to `other-nullary-override`
    omit 7af95b49b89e [SPARK-45572][PS][DOCS] Enable doctest of Frame.transpose
    omit 5280d492ad63 [SPARK-45550][PS] Remove deprecated APIs from Pandas API on Spark
    omit 4d63ca6394fe [SPARK-45562][SQL] XML: Make 'rowTag' a required option
    omit bd627503f967 [SPARK-45485][CONNECT] User agent improvements: Use SPARK_CONNECT_USER_AGENT env variable and include environment specific attributes
    omit c7b20b5dbfbe [SPARK-45567][CONNECT] Remove redundant if in org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender#run
    omit ac70daf73373 [SPARK-45565][UI] Unnecessary JSON.stringify and JSON.parse loop for task list on stage detail
    omit 28961a6ce001 [SPARK-45517][CONNECT] Expand more exception constructors to support error framework parameters
    omit 721ea9bbb2ff [MINOR][CORE] Add `@Deprecated` for `SparkLauncher#DEPRECATED_CHILD_CONNECTION_TIMEOUT`
    omit 86001c13865e [SPARK-45540][BUILD] Upgrade jetty to 9.4.53.v20231009
    omit 62653b96c9c9 Revert "[SPARK-45502][BUILD] Upgrade Kafka to 3.6.0"
    omit 6994bad5e6ea [SPARK-44262][SQL] Add `dropTable` and `getInsertStatement` to JdbcDialect
    omit 9bdad3103913 [SPARK-45538][PYTHON][CONNECT] pyspark connect overwrite_partitions bug
    omit 7796d8a63318 [SPARK-45531][SQL][DOCS] Add more comments and rename some variable name for InjectRuntimeFilter
    omit c69117742b7c [SPARK-45491] Add missing SQLSTATES 2/2
    omit c35ca7cab626 [SPARK-45539][SS] Add assert and log to indicate watermark definition is required for streaming aggregation queries in append mode
    omit 4fd2d6807bff [SPARK-45009][SQL] Decorrelate predicate subqueries in join condition
    omit d8dbb6674d0f [SPARK-45514][SQL][MLLIB] Replace `scala.runtime.Tuple3Zipped` to `scala.collection.LazyZip3`
    omit db35193abc96 [SPARK-45476][SQL][FOLLOWUP] Raise exception directly instead of calling `resolveColumnsByPosition`
    omit 8c3923aaa59a [SPARK-45486][CONNECT] Make add_artifact request idempotent
    omit c93d40d2f0e9 [SPARK-44594][SS] Remove redundant method parameter in kafka connector
    omit c921ade2aeff Revert "[SPARK-45536][BUILD] Lower the default `-Xmx` of `build/mvn` to 3g"
    omit 41420ca10336 [SPARK-45429][CORE] Add helper classes for SSL RPC communication
    omit 30d95703ddd6 [SPARK-45530][CORE] Use `java.lang.ref.Cleaner` instead of `finalize` for `NioBufferedFileInputStream`
    omit 26aaf1ca04cd [SPARK-45427][CORE] Add RPC SSL settings to SSLOptions and SparkTransportConf
    omit 2f6cca55286a [SPARK-45439][SQL][UI] Reduce memory usage of LiveStageMetrics.accumIdsToMetricType
    omit 6f46ea2f9bba [SPARK-45513][CORE][SQL][MLLIB][CONNECT] Replace `scala.runtime.Tuple2Zipped` to `scala.collection.LazyZip2`
    omit d4ceedd8b2f9 [SPARK-45487] Fix SQLSTATEs and temp errors
    omit bc7094e168f4 [SPARK-45426][CORE] Add support for a ReloadingX509TrustManager
    omit 632eabdb6dfb [SPARK-45495][CORE] Support stage level task resource profile for k8s cluster when dynamic allocation disabled
    omit 3e2470de7ea8 [SPARK-45536][BUILD] Lower the default `-Xmx` of `build/mvn` to 3g
    omit 0257b77528a3 [SPARK-45532][DOCS] Restore codetabs for the Protobuf Data Source Guide
    omit 96bac6c033b5 [SPARK-45508][CORE] Add "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED" so Platform can access Cleaner on Java 9+
    omit f1f856d54636 [SPARK-45526][PYTHON][DOCS] Improve the example of DataFrameReader/Writer.options to take a dictionary
    omit fb3b707bc1c8 [SPARK-45498][CORE] Followup: Ignore task completion from old stage a…
    omit 12880c846b56 [SPARK-45266][PYTHON][FOLLOWUP] Fix to resolve UnresolvedPolymorphicPythonUDTF when the table argument is specified as a named argument
    omit f72b87b90bea [SPARK-45515][CORE][SQL] Use enhanced `switch` expressions to replace the regular `switch` statement
    omit 4df9fa24f561 [SPARK-45521][ML] Avoid re-computation of nnz in `VectorAssembler`
    omit 12638b851f37 [SPARK-45418][SQL][PYTHON][CONNECT] Change current_database() column alias to current_schema()
    omit 280f6b33110d [SPARK-45505][PYTHON] Refactor analyzeInPython to make it reusable
    omit e720cce1813e [SPARK-45516][CONNECT] Include QueryContext in SparkThrowable proto message
    omit 7663fdfa3e84 [SPARK-45499][CORE][TESTS][FOLLOWUP] Use `ReferenceQueue#remove` instead of `ReferenceQueue#poll`
    omit b0576fff9b72 [SPARK-45501][CORE][SQL] Use pattern matching for type checking and conversion
    omit d1bd21a2a219 [SPARK-45502][BUILD] Upgrade Kafka to 3.6.0
    omit f0b2e6da5211 [SPARK-45132][SQL] Fix IDENTIFIER for function invocation
    omit 470aaf32a43e [SPARK-43664][CONNECT][PS] Raise exception for `ps.sql` with Pandas-on-Spark object on Spark Connect
    omit 385a2f5f0475 [SPARK-45510][SQL] Replace `scala.collection.generic.Growable` to `scala.collection.mutable.Growable`
    omit e69752aa66c0 [SPARK-45488][SQL] XML: Add support for value in 'rowTag' element
    omit 5a00631e5805 [SPARK-45442][PYTHON][DOCS] Refine docstring of DataFrame.show
    omit 95653904a116 [SPARK-42881][SQL][FOLLOWUP] Update the results of JsonBenchmark-jdk21 after get_json_object supports codgen
    omit 69cf80d25f0e [SPARK-45402][SQL][PYTHON] Add UDTF API for 'eval' and 'terminate' methods to consume previous 'analyze' result
    omit 045eb2d6dade [SPARK-45113][PYTHON][DOCS][FOLLOWUP] Make doctests deterministic
    omit 4027474cc744 [SPARK-45221][PYTHON][DOCS] Refine docstring of DataFrameReader.parquet
    omit a5f019554991 [SPARK-45415] Allow selective disabling of "fallocate" in RocksDB statestore
    omit 292a1131b542 [SPARK-44855][CONNECT] Small tweaks to attaching ExecuteGrpcResponseSender to ExecuteResponseObserver
    omit 5ad57a70e51d [SPARK-45204][CONNECT] Add optional ExecuteHolder to SparkConnectPlanner
    omit eae5c0e1efce [SPARK-45433][SQL] Fix CSV/JSON schema inference when timestamps do not match specified timestampFormat
    omit 8e70c39650f9 [SPARK-45483][CONNECT] Correct the function groups in connect.functions
    omit d1aff011fe1e [SPARK-45499][CORE][TESTS] Replace `Reference#isEnqueued` with `Reference#refersTo(null)`
    omit c25253083309 [SPARK-42881][SQL] Codegen Support for get_json_object
    omit acd5dc499d13 [SPARK-45467][CORE] Replace `Proxy.getProxyClass()` with `Proxy.newProxyInstance().getClass`
    omit 972180513081 [SPARK-45496][CORE][DSTREAM] Fix the compilation warning related to `other-pure-statement`
    omit 11af786b35ca [SPARK-45451][SQL] Make the default storage level of dataset cache configurable
    omit ae112e4279fe [SPARK-45116][SQL] Add some comment for param of JdbcDialect `createTable`
    omit e1a7b84f47b2 [SPARK-45397][ML][CONNECT] Add array assembler feature transformer
    omit 8394ebb52b94 [SPARK-45469][CORE][SQL][CONNECT][PYTHON] Replace `toIterator` with `iterator` for `IterableOnce`
    omit 0c1e9a5b19c4 [SPARK-45500][CORE][WEBUI] Show the number of abnormally completed drivers in MasterPage
    omit 305db078ff25 [SPARK-45480][SQL][UI] Selectable Spark Plan Node on UI
    omit 64ac59d71296 [SPARK-45497][K8S] Add a symbolic link file `spark-examples.jar` in K8s Docker images
    omit 2784349eda65 [SPARK-45494][CORE][PYTHON] Introduce util functions to read/write a byte array for PythonWorkerUtils
    omit 3f2a728c2853 [SPARK-45310][CORE] Report shuffle block status should respect shuffle service during decommission migration
    omit 57e616d0e299 [SPARK-45416][CONNECT] Sanity check consistency of Arrow results
    omit 128d072b0e5e [SPARK-45464][CORE] Fix network-yarn distribution build
    omit c47a9205b2de [SPARK-45477][INFRA] Use `matrix.java/inputs.java` to replace the hardcoded Java version in `test results/unit tests log` naming
    omit 6373f19f537f [SPARK-45213][SQL] Assign name to the error _LEGACY_ERROR_TEMP_2151
    omit 397bbb68cc4a [SPARK-45479][INFRA] Uses the default Python versions in the scheduled builds in other branches
    omit 39cc4abaff73 [SPARK-45475][SQL] Uses DataFrame.foreachPartition instead of RDD.foreachPartition in JdbcUtils
    omit d33644a9f8cf [SPARK-45473][SQL] Fix incorrect error message for RoundBase
    omit 3a51a5892772 [SPARK-45474][CORE][WEBUI] Support top-level filtering in MasterPage JSON API
    omit d073f2d3e2f6 [SPARK-45449][SQL] Cache Invalidation Issue with JDBC Table
    omit c9c99222e828 [SPARK-45205][SQL] CommandResultExec to override iterator methods to avoid triggering multiple jobs
    omit 762307656748 [SPARK-45458][SQL] Convert IllegalArgumentException to SparkIllegalArgumentException in bitwiseExpressions
    omit f378b506bf1f [SPARK-45470][SQL] Avoid paste string value of hive orc compression kind
    omit 4dbe4ffebfc8 [SPARK-45450][PYTHON] Fix imports according to PEP8: pyspark.pandas and pyspark (core)
    omit 08640961e3ba [SPARK-45472][SS] RocksDB State Store Doesn't Need to Recheck checkpoint path existence
    omit ff87c0958f79 [SPARK-45419][SS] Avoid reusing rocksdb sst files in a dfferent rocksdb instance
    omit 26527c43e652 [SPARK-43299][SS][CONNECT] Convert StreamingQueryException in Scala Client
    omit 7e8021b6f52c [SPARK-45456][BUILD] Upgrade maven to 3.9.5
    omit 1e4797e88aa4 [SPARK-45452][SQL][FOLLOWUP] Simplify path check logic
    omit ced321c8b5a3 [SPARK-45383][SQL] Fix error message for time travel with non-existing table
    omit f5cf31019913 [SPARK-45412][PYTHON][CONNECT][FOLLOW-UP] Remove unnecessary check
    omit af800b505956 [SPARK-45459][SQL][TESTS][DOCS] Remove the last 2 extra spaces in the automatically generated `sql-error-conditions.md` file
    omit 4493b431192f [SPARK-45424][SQL] Fix TimestampFormatter return optional parse results when only prefix match
    omit ffc6994f4653 [SPARK-44913][SQL][FOLLOWUP] DS V2 supports push down V2 UDF that has magic method
    omit 89acebb43b9a [SPARK-45465][BUILD][K8S] Upgrade `kubernetes-client` to 6.9.0 for K8s 1.28
    omit 2e945d7a0692 [SPARK-44527][SQL] Replace ScalarSubquery with null if its maxRows is 0
    omit d6b1fd4a7cda [SPARK-45401][PYTHON] Add a new method `cleanup` in the UDTF interface
    omit 2f58ef2ba7e0 [SPARK-45457][DOCS] Surface sc.setLocalProperty() value=NULL param meaning
    omit f05403c27a8f [SPARK-45460][SQL] Replace `scala.collection.convert.ImplicitConversions` to `scala.jdk.CollectionConverters`
    omit a8507ceba967 [SPARK-45461][CORE][SQL][MLLIB] Introduce a mapper for StorageLevel
    omit 83d7c63173d2 [SPARK-45336][TEST] Use Oracle Database 23 Free for integration and testing
    omit 94f5f4ff6591 [SPARK-43704][CONNECT][PS] Support `MultiIndex` for `to_series()`
    omit 8d3199d54144 [SPARK-45462][CORE][WEBUI] Show `Duration` in `ApplicationPage`
    omit b7eb65625ddf [SPARK-45413][CORE] Add warning for prepare drop LevelDB support
    omit 4bad4b6c4bb7 [SPARK-45454][SQL] Set the table's default owner to current_user
    omit 2d6d09b71e77 [MINOR][DOCS] Update `CTAS` with `LOCATION` behavior with Spark 3.2+
    omit 24b82dfd6cfb [SPARK-45452][SQL] Improve `InMemoryFileIndex` to use `FileSystem.listFiles` API
    omit 2bca4426e6dd [SPARK-45447][MLLIB][TESTS] Reduce the memory required to start the `LocalClusterSparkContext` in the `mllib` module test cases
    omit 30d0fe072c8b [SPARK-45441][PYTHON] Introduce more util functions for PythonWorkerUtils
    omit 9ec85d3c6b30 [SPARK-45239][CONNECT] Reduce default spark.connect.jvmStacktrace.maxSize
    omit e43ef424feae [SPARK-45436][PYTHON][CONNECT] DataFrame methods check same session
    omit b342a8cce85d [SPARK-45331][DOCS][FOLLOWUP] Update `SCALA_VERSION` in `_config.yml` to 2.13.12
    omit c45be4b24d4c [SPARK-45448][PYTHON] Fix imports according to PEP8: pyspark.testing, pyspark.mllib, pyspark.resource and pyspark.streaming
    omit b724955bb7c2 [SPARK-45446][PYTHON] Fix imports according to PEP8: pyspark.errors and pyspark.ml
    omit b54088f63f11 [SPARK-45444][BUILD] Upgrade `commons-io` to 1.24.0
    omit 6323abc75d4d [SPARK-45445][BUILD] Upgrade snappy to 1.1.10.5
    omit 4c740613d2cd [SPARK-45385][SQL][TESTS][FOLLOWUP] Improve the test case for deprecate `spark.sql.parser.escapedStringLiterals`
    omit 3593f8a8919d [SPARK-45425][SQL] Mapped TINYINT to ShortType for MsSqlServerDialect
    omit e3b1bb117fe9 [SPARK-45262][SQL][TESTS][DOCS] Improve examples for regexp parameters
    omit 39d43e0ac3b5 [SPARK-45434][ML][CONNECT] LogisticRegression checks the training labels
    omit bb56226c10df [SPARK-45432][CORE] Remove deprecated Hadoop-2 `LocatedFileStatus` constructor
    omit 46c7ff8ddf49 [SPARK-45357][CONNECT][TESTS] Normalize `dataframeId` when comparing `CollectMetrics` in `SparkConnectProtoSuite`
    omit 78863ad5870a [MINOR][SQL] Remove Scalac 2.12-specific code in `InMemoryFileIndex`
    omit 2ce1a8769846 [SPARK-44219][SQL] Adds extra per-rule validations for optimization rewrites
    omit 94666e9cd63a [SPARK-45421][SQL] Catch AnalysisException over InlineCTE
    omit 58863dfa1b4b [SPARK-45394][SPARK-45093][PYTHON][CONNECT] Add retries for artifact API. Improve error handling (follow-up to [])
    omit c0d9ca3be14c [SPARK-45400][SQL][DOCS] Refer to the unescaping rules from expression descriptions
    omit 51973aa4c7b7 [SPARK-45423][SQL] Lower `ParquetWriteSupport` log level to debug
    omit 5446f548bbc8 [SPARK-45231][INFRA] Remove unrecognized and meaningless command about `Ammonite` from the GA testing workflow
    omit f1e9dc2a4b31 [SPARK-45408][CORE] Add RPC SSL settings to TransportConf
    omit e8eb2459e1c1 [SPARK-36112][SQL] Support correlated EXISTS and IN subqueries using DecorrelateInnerQuery framework
    omit 7984b1a4b2da [SPARK-45420][SQL][PYTHON][CONNECT] Add DataType.fromDDL into PySpark
    omit fa71fc3c25b9 [SPARK-43656][CONNECT][PS][TESTS] Enable numpy compat tests for Spark Connect
    omit 350b8d8388c9 [PYTHON][DOCS] Fix incorrect doc in pyspark.sql.DataFrame.repartition
    omit 1fce8291425d [MINOR][PYTHON] Fix python linter in master
    omit fe46698e740e [SPARK-45268][PYTHON][DOCS][FOLLOWUP] Correct the group of `equal_null`
    omit 05cdec8e0650 [SPARK-45412][PYTHON][CONNECT] Validate the plan and session in `DataFrame.__init__`
    omit 8792109a1f46 [SPARK-45345][BUILD] Clean Scala version change leftovers in  `release-build.sh`
    omit 35b627a934b1 [SPARK-45396][PYTHON] Add doc entry for `pyspark.ml.connect` module, and adds `Evaluator` to `__all__` at `ml.connect`
    omit 8d4dc9cf0022 [SPARK-45410][INFRA] Add Python GitHub Action Daily Job
    omit d5c8dfc46666 [SPARK-45404][CORE] Support `AWS_ENDPOINT_URL` env variable
    omit a99f3106c802 Revert "[SPARK-45088][PYTHON][CONNECT] Make `getitem` work with duplicated column"
    omit 17da43803fd4 [SPARK-45355][SQL][DOCS] Fix function groups in Scala Doc
    omit cc4ecb5104e3 [SPARK-45398][SQL] Append `ESCAPE` in `sql()` of the `Like` expression
    omit 5e6986b81982 [SPARK-45409][INFRA] Pin `torch<=2.0.1`
    omit 215fa85c2719 [SPARK-45407][INFRA] Skip Unidoc in SparkR GitHub Action Job
    omit 97597ba52842 [SPARK-45406][PYTHON][CONNECT] Delete `schema` from DataFrame constructor
    omit d86208d043a2 [SPARK-43620][CONNECT][PS] Fix Pandas APIs depends on unsupported features
    omit 9830901beb33 [SPARK-45351][CORE] Change `spark.shuffle.service.db.backend` default value to ROCKSDB
    omit ee2eeb70a589 [SPARK-45283][CORE][TESTS] Make StatusTrackerSuite less fragile
    omit 4f09b6cc101b [SPARK-45376][CORE] Add netty-tcnative-boringssl-static dependency
    omit d5e13234dfb2 [SPARK-45399][SQL] XML: Add XML Options using newOption
    omit 2e46a8616fbf [SPARK-45347][SQL][CONNECT] Include SparkThrowable in FetchErrorDetailsResponse
    omit 39228d4ed60d [SPARK-45361][SQL][DOCS] Describe characters unescaping in string literals
    omit 5b80639e643b [SPARK-45250][CORE] Support stage level task resource profile for yarn cluster when dynamic allocation disabled
    omit b01dce2b2a57 [SPARK-45378][CORE] Add convertToNettyForSsl to ManagedBuffer
    omit e53abbbceaa2 [SPARK-45371][CONNECT] Fix shading issues in the Spark Connect Scala Client
    omit cdbb301143de [SPARK-45377][CORE] Handle InputStream in NettyLogger
    omit 8b3ad2fc329e [SPARK-45389][SQL][HIVE] Correct MetaException matching rule on getting partition metadata
    omit d708fd7b68bf [MINOR][DOCS] Fix Python code sample for StreamingQueryListener: Reporting Metrics programmatically using Asynchronous APIs
    omit a0c9ab63f3bc [SPARK-45386][SQL]: Fix correctness issue with persist using StorageLevel.NONE on Dataset (#43188)
    omit 8f1b02880cb9 [SPARK-45392][CORE][SQL][SS] Replace `Class.newInstance()` with `Class.getDeclaredConstructor().newInstance()`
    omit 643553a2f911 [SPARK-45338][MLLIB][PYTHON][FOLLOWUP] Use `immutable.ArraySeq.unsafeWrapArray` instead of `array.toSeq`
    omit a73430492c4c [SPARK-45388][SQL] Eliminate unnecessary reflection invocation in Hive shim classes
    omit cdebe7885f78 [SPARK-45385][SQL] Deprecate `spark.sql.parser.escapedStringLiterals`
    omit 3d231710268d [SPARK-44120][PYTHON] Support Python 3.12
    omit dc6b43d84781 [SPARK-45390][PYTHON] Remove `distutils` usage
    omit b70c88bbef0c [SPARK-45384][CORE][SQL][SS][DSTREAM][CONNECT] Replace `TraversableOnce` with `IterableOnce`
    omit 58c24a5719b8 [SPARK-44762][CONNECT][CORE] Doc for SparkConnect.addJobTag and Connect SparkSession.addTag
    omit 43210fe71fa8 [SPARK-45331][BUILD] Upgrade Scala to 2.13.12
    omit 0ff6065e205e [SPARK-45359][PYTHON][CONNECT] DataFrame.{columns, colRegex, explain}` should raise exceptions when plan is invalid
    omit 122e9f8a7465 [SPARK-45358][SQL] Remove shim classes for Hive prior 2.0.0
    omit eac5a8c7e6da [SPARK-45330][BUILD ] Upgrade ammonite to 2.5.11
    omit bef11d89cab7 [SPARK-44913][SQL] DS V2 supports push down V2 UDF that has magic method
    omit 4863dec91f1c [SPARK-45380][CORE][SQL][ML][R][CONNECT] Replace `mutable.WrappedArray` with `mutable.ArraySeq`
    omit 0b68e41cf79c [SPARK-45382][BUILD] Upgrade Netty to 4.1.99.Final
    omit bb59b4892042 [SPARK-42205][CORE] Don't log accumulator values in stage / task start and getting result event logs
    omit c8b987709242 [SPARK-45379][INFRA] Allow the daily tests of branch-3.3 to use the new test group tags
    omit 2c1057b469b4 [MINOR][SQL] Remove `regression-test-SPARK-8489/test-2.11.jar`
    omit 8e6b1603a667 [SPARK-45227][CORE] Fix a subtle thread-safety issue with CoarseGrainedExecutorBackend
    omit 1d9905ab4289 [SPARK-45276][INFRA] Replace Java 8 and Java 11 installed in the Dockerfile with Java
    omit 5c4aef4d4caf [SPARK-45338][SQL][FOLLOWUP] Remove useless `toSeq`
    omit 0d6fda5bbee9 [SPARK-45057][CORE] Avoid acquire read lock when keepReadLock is false
    omit d52561cfb1f1 [SPARK-44937][CORE] Add test keys for SSL functionality
    omit 2a88feadd4b7 [SPARK-44937][CORE] Mark connection as timedOut in TransportClient.close
    omit 6341310711ee [SPARK-44895][CORE][UI] Add 'daemon', 'priority' for ThreadStackTrace
    omit b3d5bc0c1090 [SPARK-45362][PYTHON] Project out PARTITION BY expressions before Python UDTF 'eval' method consumes them
    omit b938ff9f520f [SPARK-45365][INFRA] Allow `branch-3.4` daily test to use the new test group tags
    omit ee21b12c395a [MINOR][SQL] Remove duplicate cases of escaping characters in string literals
    omit 2cf937f9bac2 [SPARK-45266][PYTHON] Refactor ResolveFunctions analyzer rule to delay making lateral join when table arguments are used
    omit 183a3d761f36 [SPARK-45360][SQL][CONNECT] Initialize spark session builder configuration from SPARK_REMOTE
    omit 187e9a851758 [SPARK-45364][INFRA][BUILD] Clean up the unnecessary Scala 2.12 logical in SparkBuild
    omit 2cc1ee4d3a05 [SPARK-45344][CORE][SQL] Remove all Scala version string check
    omit 6d2ffaa4ea87 [SPARK-45353][PYTHON][DOCS] Refine docstring of `create_map/slice/array_join`
    omit c5967310740e [SPARK-44442][MESOS] Remove Mesos support
    omit 6a9d35f766dc [SPARK-45354][SQL] Resolve functions bottom-up
    omit 9109d7037f44 [SPARK-44838][SQL] raise_error improvement
    omit 8399dd321afc [SPARK-44539][BUILD] Upgrade RoaringBitmap to 1.0.0
    omit ab92cae78e3c [SPARK-45343][DOCS] Clarify behavior of multiLine in CSV options
    omit e10459f9d2f6 [SPARK-44170][BUILD][TESTS] Migrate Junit 4 to Junit 5
    omit b7763a7eae2b [SPARK-45338][CORE][SQL] Replace `scala.collection.JavaConverters` to `scala.jdk.CollectionConverters`
    omit 9f817379c68e [SPARK-45341][CORE] Correct the title level in the comments of KVStore.java to make `sbt doc` run successfully with Java 17
    omit 2da6d1a87314 [SPARK-45337][CORE] Refactor `AbstractCommandBuilder#getScalaVersion` to remove the check for Scala 2.12
    omit 6e6ee613a0d3 [SPARK-45268][PYTHON][DOCS] python function categories should be consistent with SQL References
    omit b348cd12a331 [MINOR][PYTHON][CONNECT] Check `self._session` before call `self._session.client`
    omit f8c87f03297e [SPARK-43380][SQL] Fix Avro data type conversion issues without causing performance regression
    omit 1cbc424ae2ac [SPARK-45346][SQL] Parquet schema inference should respect case sensitive flag when merging schema
    omit 400d4ded64ef [SPARK-43662][PS][CONNECT] Support merge_asof in Spark Connect
    omit 0b057c807f19 [SPARK-45308][PS][TESTS] Enable `GroupbySplitApplyTests.test_split_apply_combine_on_series` for pandas 2.0.0
    omit 8cbc741320da [SPARK-45267][PS] Change the default value for numeric_only
    omit 644c8b25bfda [SPARK-45335][SQL][DOCS] Correct the group of `ElementAt` and `TryElementAt`
    omit 28dc555821b6 [SPARK-45329][PYTHON][CONNECT] DataFrame methods skip pandas conversion
    omit eff46ea77e9b [SPARK-45340][SQL] Remove the SQL config `spark.sql.hive.verifyPartitionPath`
    omit f6c6acbc00d2 [SPARK-44780][DOC] SQL temporary variables
    omit 1597d8174a54 [SPARK-43850][BUILD][GRAPHX] Remove the import for `scala.language.higherKinds` and delete the corresponding suppression rule
    omit 7e8aafd2c0f1 [SPARK-45334][SQL] Remove misleading comment in parquetSchemaConverter
    omit 17430fe47029 [SPARK-45302][PYTHON] Remove PID communication between Python workers when no demon is used
    omit 17881eb7eca4 [SPARK-45339][PYTHON][CONNECT] Pyspark should log errors it retries
    omit e6d1e9ed3843 [SPARK-44751][SQL][FOLLOWUP] Change `xmlExpressions.scala` package name
    omit 3c84c229d167 [SPARK-45328][SQL] Remove Hive support prior to 2.0.0
    omit ea3104fa71b0 [SPARK-45325][BUILD][FOLLOWUP] Update docs and sbt
    omit 13cd291c3549 [SPARK-44366][BUILD] Upgrade antlr4 to 4.13.1
    omit ff084d2852e6 [SPARK-44756][CORE] Executor hangs when RetryingBlockTransferor fails to initiate retry
    omit 60d02b444e22 [SPARK-45316][CORE][SQL] Add new parameters `ignoreCorruptFiles`/`ignoreMissingFiles` to `HadoopRDD` and `NewHadoopRDD`
    omit 2aa06fcf1607 [SPARK-45271][SQL] Merge _LEGACY_ERROR_TEMP_1113 into TABLE_OPERATION & delete some unused method in QueryCompilationErrors
    omit e54c866701dd [SPARK-45309][SQL] Remove all SystemUtils.isJavaVersionAtLeast with JDK 9/11/17
    omit 9933e9c2c54e [SPARK-45323][BUILD] Upgrade `snappy` to 1.1.10.4
    omit 2094e7121700 [SPARK-45321][TESTS] Clean up the unnecessary Scala 2.12 related binary files
    omit 73d17d52a8bc [MINOR][BUILD] Fix lint-js
    omit 9a44dc4f9ee0 [SPARK-45232][SQL][DOCS] Add missing function groups to SQL references
    omit 91b76070b05f [SPARK-45333][CORE] Fix one unit mistake related to spark.eventLog.buffer.kb
    omit 273a375cd314 [SPARK-45248][CORE] Set the timeout for spark ui server
    omit d49780037b51 [SPARK-45299][TESTS] Remove JDK 8 workaround in UtilsSuite
    omit 933f17d13f6e [SPARK-45300][SQL][TESTS] Remove JDK 8 workaround in TimestampFormatterSuite
    omit 7ffc0b71aa3e [SPARK-45182][CORE] Ignore task completion from old stage after retrying indeterminate stages
    omit 96b591abdba5 [SPARK-45318][SHELL][TESTS] Merge test cases from `SingletonRepl2Suite/Repl2Suite` back into `SingletonReplSuite/ReplSuite`
    omit b1a0d6703bf0 [SPARK-45312][SQL][UI] Support toggle display/hide plan svg on execution page
    omit 47bad35a4da5 [SPARK-45325][BUILD] Upgrade Avro to 1.11.3
    omit fc5342314c8d [SPARK-44126][CORE] Shuffle migration failure count should not increase when target executor decommissioned
    omit 8230f16164b1 [SPARK-45317][SQL][CONNECT] Handle null filename in stack traces of exceptions
    omit 7741fe79e201 [SPARK-44550][SQL] Enable correctness fixes for `null IN (empty list)` under ANSI
    omit c1b12bd56429 [SPARK-45322][CORE] Use ProcessHandle to get pid directly
    omit 36e626bc60af [SPARK-43711][SPARK-44372][CONNECT][PS][TESTS] Clear message for Spark ML dependent tests
    omit 0edeb605f2f6 [SPARK-45304][BUILD] Remove test classloader workaround for SBT build
    omit 42a6557172cd [MINOR][SQL][DOCS] Remove JDK 8 related information in the comemnts for aes_encrypt and aes_decrypt
    omit 772802d11656 [SPARK-45301][BUILD] Remove org.scala-lang scala-library added for JDK 11 workaround
    omit c1c1c9fa98f7 [SPARK-45305][SQL][TESTS] Remove JDK 8 workaround added TreeNodeSuite
    omit 4eb5577ece24 [SPARK-45303][CORE] Remove JDK 8/11 workaround in KryoSerializerBenchmark
    omit 30c73f6e1038 [SPARK-45313][CORE] Inline `Iterators#size` and remove `Iterators.scala`
    omit 3ff8977bf658 [SPARK-45320][SQL][TESTS] Update benchmark result for `InMemoryColumnarBenchmark`
    omit 6e16401fbff9 [SPARK-45307][INFRA] Use Zulu JDK in `benchmark` GitHub Action and Java 21
    omit a975a086093a [SPARK-45298][SPARK-31959][SQL][TESTS] Remove the workaround for JDK-8228469 in test
    omit 6e327858d3cc [SPARK-45247][PYTHON][TESTS][FOLLOW-UP] Deduplicate FrameReidexingTests.test_filter test
    omit f94cc9f7e085 [SPARK-45295][CORE][SQL] Remove Utils.isMemberClass workaround for JDK 8
    omit e9cc76a27f6f [SPARK-45297][SQL] Remove workaround for dateformatter added in SPARK-31827
    omit c326928edde3 [SPARK-45296][INFRA][BUILD] Comment out unused JDK 11 related in dev/run-tests.py
    omit 7e9666be15b5 [SPARK-45306][SQL][TESTS] Make `InMemoryColumnarBenchmark` use AQE-aware utils to collect plans
    omit 5d422155f1da [SPARK-45291][SQL][REST] Use unknown query execution id instead of no such app when id is invalid
    omit fb2bee37c964 [SPARK-42617][PS] Support `isocalendar` from the pandas 2.0.0
    omit bb0d287114fb [SPARK-45294][PYTHON][DOCS] Use JDK 17 in Binder integration for PySpark live notebooks
    omit f81f51467b85 [SPARK-45257][CORE][FOLLOWUP] Correct the from version in migration guide
    omit 4863be5632f3 [SPARK-45207][SQL][CONNECT] Implement Error Enrichment for Scala Client
    omit 913991046c6d [SPARK-45240][SQL][CONNECT] Implement Error Enrichment for Python Client
    omit 609552e19cfe [SPARK-45279][PYTHON][CONNECT] Attach plan_id for all logical plans
    omit a881438114ea [SPARK-45286][DOCS] Add back Matomo analytics
    omit 204d1c07b783 [MINOR][DOCS] Add Java 21 to index.md and building-spark.md
    omit b3e14d398542 [SPARK-44345][CORE] Lower `unknown shuffle map output` log level to WARN if shuffle migration is enabled
    omit 834d71b99046 [SPARK-45287][TESTS] Add Java 21 benchmark result and update Java 17 ones
    omit 4841ef97826b [SPARK-44119][K8S][DOCS] Drop K8s v1.25 and lower version support
    omit 94002494c286 [SPARK-45288][TESTS] Remove outdated benchmark result files, `*-jdk1[17]*results.txt`
    omit 51938fea36af [SPARK-45274][CORE][SQL][UI] Implementation of a new DAG drawing approach for job/stage/plan graphics to avoid fork
    omit 19532b32665d [SPARK-45285][CORE][TESTS] Remove deprecated `Runtime.getRuntime.exec(String)` API usage
    omit fc69614657f9 [SPARK-45284][R] Update SparkR minimum SystemRequirements to Java 17
    omit 06ccb6d43447 [SPARK-45281][CORE][TESTS] Update BenchmarkBase to use Java 17 as the base version
    omit 6960a50a0fdc [SPARK-45277][BUILD][INFRA] Install Java 17 to support SparkR testing on Windows
    omit f935331ff283 [SPARK-45256][SQL] DurationWriter fails when writing more values than initial capacity
    omit 4e75390e3583 [SPARK-45280][INFRA] Change Maven daily test use Java 17 for testing
    omit 4d435637351e [SPARK-45247][BUILD][PYTHON][PS] Upgrade Pandas to 2.1.1
    omit 46ebed2b3714 [SPARK-44112][BUILD][INFRA][DOCS] Drop support for Java 8 and Java 11
    omit 9cad22a78ec4 [SPARK-43655][CONNECT][PS][TESTS] Enable `NamespaceParityTests.test_get_index_map`
    omit 498b7e1a2f36 [SPARK-45272][BUILD][INFRA][DOCS] Remove Scala version specific comments, and scala-2.13 profile usage
    omit 670bb8e5fb1d [SPARK-45270][K8S][INFRA][DOCS] Upgrade `Volcano` to 1.8.0
    omit 25f8a44a5ebe [SPARK-45269][K8S] Use Java 21-jre in K8s Dockerfile
    omit 5f643ee1444d [MINOR][PYTHON][TESTS] Add init file to pyspark.ml.deepspeed.tests
    omit 33830d2694f6 [SPARK-44113][INFRA][FOLLOW-UP] Remove Scala 2.13 scheduled jobs
    omit 342920296c94 [SPARK-44113][BUILD][INFRA][DOCS] Drop support for Scala 2.12
    omit db02469434a5 [SPARK-45251][CONNECT] Add client_type field for FetchErrorDetails
    omit 6b747ab8cef4 [SPARK-45253][SQL][DOCS] Correct the group of `ShiftLeft` and `ArraySize`
    omit 0bf950ee4f77 [SPARK-43433][PS] Match `GroupBy.nth` behavior to the latest Pandas
    omit 555aece74d2a [SPARK-45093][CONNECT][PYTHON] Error reporting for addArtifacts query
    omit 8d5999728722 [SPARK-45257][CORE] Enable `spark.eventLog.compress` by default
    omit 7c3c7c5a4bd9 [SPARK-41086][SQL] Use DataFrame ID to semantically validate CollectMetrics
    omit ebfc5b55fc09 [SPARK-45244][TESTS] Correct spelling in VolcanoTestsSuite
    omit 47e8205628a0 [SPARK-45191][SQL] InMemoryTableScanExec simpleStringWithNodeId adds columnar info
    omit 39d6fdabb48c [SPARK-45163][SQL] Merge UNSUPPORTED_VIEW_OPERATION & UNSUPPORTED_TABLE_OPERATION & fix some issue
    omit f4f3b28aaacc [SPARK-45261][CORE][FOLLOWUP] Avoid `transform` of conf value
    omit fdedec1de54f [SPARK-45252][CORE] Escape the greater/less than symbols in the comments to make `sbt doc` execute successfully
    omit 39507b7f537d [SPARK-45263][CORE][TESTS] Make `EventLoggingListenerSuite` independent from `spark.eventLog.compress` conf
    omit 2f3cb36cb99a [SPARK-45261][CORE] Fix `EventLogFileWriters` to handle `none` as a codec
    omit 892fdc532696 [SPARK-45108][SQL] Improve the InjectRuntimeFilter for check probably shuffle
    omit 8b967e191b75 [SPARK-43254][SQL] Assign a name to the error _LEGACY_ERROR_TEMP_2018
    omit cf04bd93b752 [SPARK-45219][PYTHON][DOCS] Refine docstring of withColumn(s)Renamed
    omit a2bab5efc5b5 [SPARK-45235][CONNECT][PYTHON] Support map and array parameters by `sql()`
    omit 4531d710ba79 [SPARK-45138][SS] Define a new error class and apply it when checkpointing state to DFS fails
    omit a0ffb2e7f6a0 [MINOR][DOCS][CONNECT] Update notes about supported modules in PySpark API reference
    omit 14ba4fc47915 [SPARK-45216][SQL] Fix non-deterministic seeded Dataset APIs
    omit c1dc9ae1c10e [SPARK-45107][PYTHON][DOCS] Refine docstring of explode
    omit 76acd12a73cb [SPARK-45218][PYTHON][DOCS] Refine docstring of Column.isin
    omit e9e9320b878d [SPARK-45234][PYTHON][DOCS] Refine DocString of `regr_*` functions
    omit 324a07b534ac [SPARK-45241][INFRA] Use Zulu JDK in `build_and_test` and `build_java21` GitHub Action and Java 21
    omit f1bc0f938162 [SPARK-45237][DOCS] Change the default value of `spark.history.store.hybridStore.diskBackend` in `monitoring.md` to `ROCKSDB`
    omit 288a92b9f41e [SPARK-45112][SQL] Use UnresolvedFunction based resolution in SQL Dataset functions
    omit d92d6f60342c [SPARK-43979][SQL][FOLLOWUP] Handle non alias-only project case
    omit c89221b02bb3 [SPARK-45224][PYTHON] Add examples w/ map and array as parameters of `sql()`
    omit 27dc7ff4cb0e [SPARK-43498][PS][TESTS] Enable `StatsTests.test_axis_on_dataframe` for pandas 2.0.0
    omit ad18393d2f1e [SPARK-45226][PYTHON][DOCS] Refine docstring of `rand/randn`
    omit 92aa0f6e1d8c [SPARK-45229][CORE][UI] Show the number of drivers waiting in SUBMITTED status in MasterPage
    omit 60393202ccdb [SPARK-45225][SQL] XML: XSD file URL support
    omit 48b1a283a2eb [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC
    omit d62a8e63e86d [SPARK-44463][SS][CONNECT] Improve error handling for Connect steaming Python worker
    omit 227e26202522 [SPARK-45178][SS] Fallback to execute a single batch for Trigger.AvailableNow with unsupported sources rather than using wrapper
    omit 2325983c83fa [SPARK-45192][UI] Fix overdue lineInterpolate parameter for graphviz edge
    omit 1fdd46f173f7 [SPARK-44306][YARN] Group FileStatus with few RPC calls within Yarn Client
    omit c3a288281071 [SPARK-45215][SQL][TESTS] Combine HiveCatalogedDDLSuite and HiveDDLSuite
    omit 9fb44456b097 [SPARK-43453][PS] Ignore the `names` of `MultiIndex` when `axis=1` for `concat`
    omit eec090755aa5 [SPARK-45211][CONNECT] Eliminated ambiguous references in `CloseableIterator#apply` to fix Scala 2.13 daily test
    omit d202c7083e07 [SPARK-45212][INFRA] Install independent Python linter dependencies for branch-3.5
    omit 7e14c8cc33f0 [SPARK-44910][SQL] Encoders.bean does not support superclasses with generic type arguments
    omit 2697a9426e17 [SPARK-43654][CONNECT][PS][TESTS] Enable `InternalFrameParityTests.test_from_pandas`
    omit e818e9cefa01 [SPARK-45202][BUILD] Fix lint-js tool and js format
    omit 464a3c19f510 [SPARK-45113][PYTHON][DOCS][FOLLOWUP] Add sorting to the example of `collect_set/collect_list` to ensure stable results
    omit 8323c0c48de7 [SPARK-45133][CONNECT][TESTS][FOLLOWUP] Add test that queries transition to FINISHED
    omit f83e5ec202be [SPARK-45065][PYTHON][PS] Support Pandas 2.1.0
    omit 981312284f07 [SPARK-45188][SQL][DOCS] Update error messages related to parameterized `sql()`
    omit 29acdf755cd0 [SPARK-45153][BUILD][PS] Rebalance testing time for `pyspark-pandas-connect-part1`
    omit eadb591f37a1 [SPARK-45203][SQL][TEST] Join tests in KeyGroupedPartitioningSuite should use merge join hint
    omit 0dda75f824d7 [SPARK-45137][CONNECT] Support map/array parameters in parameterized `sql()`
    omit 8d363c6e2c84 [SPARK-45196][PYTHON][DOCS] Refine docstring of `array/array_contains/arrays_overlap`
    omit 02bbd2867be5 [SPARK-43628][SPARK-43629][CONNECT][PS][TESTS] Clear message for JVM dependent tests
    omit 1ae8e018164f [SPARK-45056][CONNECT][TESTS] Change the tests related to Python in `SparkConnectSessionHodlerSuite` to assume shouldTestPandasUDFs
    omit 82d0823c2f9b [SPARK-44788][PYTHON][DOCS][FOLLOW-UP] Move `from_xml`/`schema_of_xml` to `Xml Functions`
    omit c1c58698d3d6 [SPARK-45148][BUILD] Upgrade scalatest related dependencies to the 3.2.17 series
    omit 360adc2250bc [SPARK-45197][CORE] Make `StandaloneRestServer` add `JavaModuleOptions` to drivers
    omit 5b89bc156cb5 [SPARK-44788][PYTHON][DOCS][FOLLOW-UP] Add from_xml and schema_of_xml into PySpark documentation
    omit 5db824aa6f92 [SPARK-45193][PS][CONNECT][TESTS] Refactor `test_mode` to be compatible with Spark Connect
    omit 89041a4a8c7b [SPARK-44788][CONNECT][PYTHON][SQL] Add from_xml and schema_of_xml to pyspark, spark connect and sql function
    omit 5299e5423f43 [SPARK-44823][PYTHON] Update black to 23.9.1 and fix erroneous check
    omit 99a979df0de9 [SPARK-45130][CONNECT][ML][PYTHON] Avoid Spark connect ML model to change input pandas dataframe
    omit d5ff04da217d [SPARK-45167][CONNECT][PYTHON][FOLLOW-UP] Use lighter threading Rlock, and use the existing eventually util function
    omit 94b0ff59d84b [SPARK-45179][DOCS][FOLLOWUP] Add migration guide for Numpy minimum version upgrade
    omit 46e07245eafe [SPARK-45189][SQL] Creating UnresolvedRelation from TableIdentifier should include the catalog field
    omit fd1551892262 [SPARK-45184][SQL][DOCS][TESTS] Remove orphaned error class documents
    omit f5365d0dc590 [SPARK-45034][SQL] Support deterministic mode function
    omit f8f2735426ee [SPARK-45187][CORE] Fix `WorkerPage` to use the same pattern for `logPage` urls
    omit 65b12f95c149 [SPARK-45167][CONNECT][PYTHON] Python client must call `release_all`
    omit 5d19a51ae6cb [SPARK-45180][PS] Remove boolean inputs for `inclusive` parameter from `Series.between`
    omit e84c66db60c7 [SPARK-45078][SQL] Fix `array_insert` ImplicitCastInputTypes not work
    omit 21a9458c2d3e [SPARK-45168][PYTHON][FOLLOWUP] Add migration guide for Pandas minimum version upgrade
    omit 948fa0e3e6b6 [SPARK-43458][SPARK-43561][PS][TESTS] Enable `test_to_latex` for (Series|DataFrame) conversion
    omit 804f741453fb [SPARK-45127][DOCS] Exclude README.md from document build
    omit 33979829db9e [SPARK-45146][DOCS] Update the default value of 'spark.executor.logs.rolling.strategy'
    omit ab46dc048baf [SPARK-44872][CONNECT][FOLLOWUP] Deflake ReattachableExecuteSuite and increase retry buffer
    omit e5f69e45bb12 [SPARK-45181][BUILD] Upgrade buf to v1.26.1
    omit 55764a9ae8ef [SPARK-40497][BUILD] Re-upgrade Scala to 2.13.11
    omit 69ba1bd14768 [SPARK-45164][PS] Remove deprecated `Index` APIs
    omit 1b7dbf7d4aa0 [SPARK-45173][UI] Remove some unnecessary sourceMapping files in UI
    omit 3536f3343b57 [SPARK-45166][PYTHON][FOLLOWUP] Delete unused `pyarrow_version_less_than_minimum` from `pyspark.sql.pandas.utils`
    omit e35c748d43e7 [SPARK-43874][FOLLOWUP][TESTS] Enable `GroupbyIndexTests.test_groupby_multiindex_columns`
    omit 82ed1c9f8ad5 [SPARK-45179][PYTHON] Increase Numpy minimum version to 1.21
    omit 0d1f43cda135 [SPARK-45151][CORE][UI] Task Level Thread Dump Support
    omit 197229418ff7 [SPARK-45168][PYTHON][FOLLOWUP]  `test_missing_data.py` Code Cleanup
    omit 213ac1c946d8 [SPARK-45177][PS] Remove `col_space` parameter from `to_latex`
    omit 74d3d87171dd [SPARK-45166][PYTHON] Clean up unused code paths for pyarrow<4
    omit c490d8c17af9 [SPARK-45128][SQL] Support `CalendarIntervalType` in Arrow
    omit c3b676d818ed [SPARK-45168][PYTHON] Increase Pandas minimum version to 1.4.4
    omit c1c710e7da75 [SPARK-45143][PYTHON][CONNECT] Make PySpark compatible with PyArrow 13.0.0
    omit 5d4ca79aad11 [SPARK-45172][BUILD] Upgrade `commons-compress` to 1.24.0
    omit e097f916a276 [SPARK-45171][SQL] Initialize non-deterministic expressions in `GenerateExec`
    omit 5678dbecc29c [SPARK-45174][CORE] Support `spark.deploy.maxDrivers`
    omit 91ccc0f8e9fd [SPARK-45165][PS] Remove `inplace` parameter from `CategoricalIndex` APIs
    omit 822f58f0d26b [SPARK-45159][PYTHON] Handle named arguments only when necessary
    omit 25c624f269bf [SPARK-45084][SS] StateOperatorProgress to use accurate effective shuffle partition number
    omit db5f37c1fac3 [SPARK-45161][INFRA] Bump previousSparkVersion to 3.5.0
    omit 090fd18f3624 [SPARK-45118][PYTHON] Refactor converters for complex types to short cut when the element types don't need converters
    omit cb298e802196 [SPARK-45119][PYTHON][DOCS] Refine docstring of inline
    omit 73d3c49c97ae [SPARK-45088][PYTHON][CONNECT] Make `getitem` work with duplicated columns
    omit e1d2372b8916 [MINOR][PYTHON][DOCS] Fix default value of parameter `barrier` in MapInXXX
    omit 0e9ad5ad66a4 [SPARK-45056][PYTHON][SS][CONNECT][TESTS][FOLLOW-UP] Remove listeners only when they exists
    omit 6653f94d4893 [SPARK-45156][SQL] Wrap `inputName` by backticks in the `NON_FOLDABLE_INPUT` error class
    omit cd672b09ac69 [SPARK-45162][SQL] Support maps and array parameters constructed via `call_function`
    omit 99df9dda47f1 [SPARK-45120][SPARK-45150][UI] Upgrade d3 from v3 to v7(v7.8.5) and apply api changes in UI
    omit b6190a3db974 [SPARK-45056][PYTHON][SS][CONNECT] Termination tests for streamingQueryListener and foreachBatch
    omit 21c27d55b342 [SPARK-36191][SQL] Handle limit and order by in correlated scalar (lateral) subqueries
    omit 9798244ca647 [SPARK-45135][PYTHON][TESTS] Make `utils.eventually` a parameterized decorator
    omit d26e77aabd29 Revert "[SPARK-45120][UI] Upgrade d3 from v3 to v7(v7.8.5) and apply api changes in UI"
    omit c12eb2c93d76 [SPARK-45144][BUILD] Downgrade `scala-maven-plugin` to 4.7.1
    omit b568ba43f0dd [SPARK-45157][SQL] Avoid repeated `if` checks in `[On|Off|HeapColumnVector`
    omit 076cb7aabac2 [SPARK-45146][DOCS] Update the default value of 'spark.submit.deployMode'
    omit 61435b42fdc4 [SPARK-45142][INFRA] Specify the range for Spark Connect dependencies in pyspark base image
    omit 2710dbe6abf0 [SPARK-45139][SQL] Add DatabricksDialect to handle SQL type conversion
    omit 4f74fc581486 [SPARK-45147][CORE] Remove `System.setSecurityManager` usage
    omit e3d2dfa8b514 [SPARK-45141][PYTHON][INFRA][TESTS] Pin `pyarrow==12.0.1` in CI
    omit 66f42b5b292c [SPARK-45145][EXAMPLE] Add JavaSparkSQLCli example
    omit c119b8aec871 [SPARK-45133][CONNECT] Make Spark Connect queries be FINISHED when last result task is finished
    omit ef89b278f8e0 [SPARK-45096][INFRA] Optimize apt-get install in Dockerfile
    omit 74c78970cd9e [SPARK-45131][PYTHON][DOCS] Refine docstring of `ceil/ceiling/floor/round/bround`
    omit 4221235a6b27 [SPARK-45120][UI] Upgrade d3 from v3 to v7(v7.8.5) and apply api changes in UI
    omit d999f622dc68 [SPARK-45117][SQL] Implement missing otherCopyArgs for the MultiCommutativeOp expression
    omit f33c2051c01e Revert "[SPARK-45110][BUILD] Upgrade rocksdbjni to 8.5.3"
    omit 4b96add471d2 [SPARK-44872][CONNECT] Server testing infra and ReattachableExecuteSuite
    omit d8298bffd91d [SPARK-45081][SQL] Encoders.bean does no longer work with read-only properties
    omit 2b731565941a [MINOR][DOCS] Add errors.rst to .gitignore
    omit 1e03db36a939 [SPARK-44911][SQL] Create hive table with invalid column should return error class
    omit 4d213ff3dea4 [SPARK-45092][SQL][UI] Avoid analyzing twice for failed queries
    omit 994b6976b2a5 [SPARK-44915][CORE] Validate checksum of remounted PVC's shuffle data before recovery
    omit d8129f837c47 [SPARK-45085][SQL] Merge UNSUPPORTED_TEMP_VIEW_OPERATION into UNSUPPORTED_VIEW_OPERATION and refactor some logic
    omit 6565ae47cae1 [SPARK-43252][SQL] Replace the error class `_LEGACY_ERROR_TEMP_2016` with an internal error
    omit fa2bc21ba1e6 [SPARK-45110][BUILD] Upgrade rocksdbjni to 8.5.3
    omit d7845da6ddf2 [SPARK-45125][INFRA] Remove dev/github_jira_sync.py in favor of ASF jira_options
    omit 5d2d9155d24e [SPARK-45121][CONNECT][PS] Support `Series.empty` for Spark Connect
    omit 3148511a923b [SPARK-43123][PS] Raise `TypeError` for `DataFrame.interpolate` when all columns are object-dtype
    omit 47d801e5e9de [SPARK-45124][CONNET] Do not use local user ID for Local Relations
    omit 9520087409d5 [SPARK-44647][SQL] Support SPJ where join keys are less than cluster keys
    omit 6c3d9f5d89df [SPARK-45109][SQL][CONNECT][FOLLOWUP] Fix log function in Connect
    omit 169aa4bee950 [SPARK-45111][BUILD] Upgrade maven to 3.9.4
    omit c7ea3f7d53d5 [SPARK-43251][SQL] Replace the error class `_LEGACY_ERROR_TEMP_2015` with an internal error
    omit 2a10c8d93aa9 [SPARK-45069][SQL] SQL variable should always be resolved after outer reference
    omit 8ee22d9358e9 [SPARK-45114][PYTHON][DOCS] Adjust the `versionadded` and `versionchanged` information to the parameters
    omit f41dc28fbda6 [SPARK-45113][PYTHON][DOCS] Refine docstrings of `collect_list/collect_set`
    omit 3d119a528060 [SPARK-43295][PS] Support string type columns for `DataFrameGroupBy.sum`
    omit eb0b09f0f2b5 [SPARK-45044][PYTHON][DOCS] Refine docstring of `groupBy/rollup/cube`
    omit 10529608a390 [SPARK-45027][PYTHON] Hide internal functions/variables in `pyspark.sql.functions` from auto-completion
    omit 5e97c79ad829 [SPARK-45109][SQL][CONNECT] Fix aes_decrypt and ln functions in Connect
    omit 445c5417ea1d [SPARK-45105][DOCS] Make hyperlinks in documents clickable
    omit c8fa821a873b [SPARK-44866][SQL] Add `SnowflakeDialect` to handle BOOLEAN type correctly
    omit a663c0bf0c5b [SPARK-45104][UI] Upgrade `graphlib-dot.min.js` to 1.0.2
    omit 4dd4737b34b1 [SPARK-45075][SQL] Fix alter table with invalid default value will not report error
    omit fac236e1350d [SPARK-44805][SQL] getBytes/getShorts/getInts/etc. should work in a column vector that has a dictionary
    omit 2b4387fe9498 [SPARK-45106][SQL] `PercentileCont` should check user supplied input
    omit 81bc38ee8776 [SPARK-45098][DOCS] Custom jekyll-rediect-from redirect.html template to fix doc redirecting
    omit e4df5d14b224 [MINOR][DOCS] Change "filter" to "transform" in `transform` function docstring
    omit 4c8d39881e3f [SPARK-44986][DOCS] There should be a gap at the bottom of the HTML
    omit fd424caf6c46 [SPARK-45100][SQL] Fix an internal error from `reflect()`on `NULL` class and method
    omit aaf413ce351d [SPARK-44508][PYTHON][DOCS] Add user guide for Python user-defined table functions
    omit a37c265371dc [SPARK-44732][XML][FOLLOWUP] Partial backport of spark-xml "Shortcut common type inference cases to fail fast"
    omit b8b58e0b95bb [SPARK-45077][UI] Upgrade dagre-d3.js from 0.4.3 to 0.6.4
    omit 0e6e15ca6331 [SPARK-45080][SS] Explicitly call out support for columnar in DSv2 streaming data sources
    omit af1615d026ea [SPARK-45036][FOLLOWUP][SQL] SPJ: Make sure result partitions are sorted according to partition values
    omit 7818f47ca2fe [SPARK-45086][UI] Display hexadecimal for thread lock hash code
    omit 622bbf2e2926 [SPARK-44845][YARN][DEPLOY] Fix file system uri comparison function
    omit 56393b907232 [SPARK-45091][PYTHON][CONNECT][SQL] Function `floor/round/bround` accept Column type `scale`
    omit 66f89f35f0ea [SPARK-45090][PYTHON][CONNECT] DataFrame.{cube, rollup}` support column ordinals
    omit f13743de04e4 [SPARK-44835][CONNECT] Make INVALID_CURSOR.DISCONNECTED a retriable error
    omit a0a78b6081f1 [SPARK-45025][CORE] Allow block manager memory store iterator to handle thread interrupt and perform task completion gracefully
    omit e82805da8766 [SPARK-45050][SQL][CONNECT] Improve error message for UNKNOWN io.grpc.StatusRuntimeException
    omit 0777562c5197 [BUILD][TEST] Remove obsolete repo of DB2 JDBC driver
    omit cbe36b20a859 [SPARK-44801][SQL][FOLLOWUP] Remove overdue comments for generating plan info in SQLExecution
    omit 48872d4b3938 [SPARK-44833][CONNECT][PYTHON][FOLLOW-UP] Fix the type annotation for iterator at reattach.py
    omit 24b29adcf536 [SPARK-45079][SQL] Fix an internal error from `percentile_approx()`on `NULL` accuracy
    omit 637f16e7ff88 [SPARK-45070][SQL][DOCS] Describe the binary and datetime formats of `to_char`/`to_varchar`
    omit e4d17e9a1fb6 [SPARK-44833][CONNECT] Fix sending Reattach too fast after Execute
    omit 7e1dc10e19d7 [SPARK-45046][BUILD] Set `shadeTestJar` of `core` module to `false`
    omit 16e813cecd55 [SPARK-45071][SQL] Optimize the processing speed of `BinaryArithmetic#dataType` when processing multi-column data
    omit ba35140ad166 [SPARK-45083][PYTHON][DOCS] Refine the docstring of function `min`
    omit e0a6af988df3 [SPARK-45082][DOC] Review and fix issues in API docs for 3.5.0
    omit 40943c2748fd [SPARK-45072][CONNECT] Fix outer scopes for ammonite classes
    omit 5fff2427ccb8 [SPARK-43241][PS][FOLLOWUP] Add migration guide for behavior change
    omit 6c885a7cf57d [SPARK-45074][PYTHON][CONNECT] `DataFrame.{sort, sortWithinPartitions}` support column ordinals
    omit a2c247bc1c82 [SPARK-45076][PS] Switch to built-in `repeat` function
    omit 57c92fbab9bb [MINOR][PYTHON][TESTS] Add init file to pyspark.ml.tests.connect
    omit d7e827e52c45 [SPARK-43241][PS] `MultiIndex.append` not checking names for equality
    omit 4686c2733702 [SPARK-45073][PS][CONNECT] Replace `LastNotNull` with `Last(ignoreNulls=True)`
    omit 66fb2250224f [SPARK-45049][CONNECT][DOCS][TESTS] Refine docstrings of `coalesce/repartition/repartitionByRange`
    omit 8d358cdbd57e [SPARK-42619][PS] Add `show_counts` parameter for DataFrame.info
    omit ebdbc7b1d3e5 [SPARK-45064][PYTHON][CONNECT] Add the missing `scale` parameter in `ceil/ceiling`
    omit d17a8613a68a [SPARK-45047][PYTHON][CONNECT] `DataFrame.groupBy` support ordinals
    omit 96e1e86b0a8f [SPARK-45059][CONNECT][FOLLOWUP] Remove `try_reflect` problem filter rule in connect
    omit ba20eaa4c30a [SPARK-45066][SQL][PYTHON][CONNECT] Make function `repeat` accept column-type `n`
    omit 798fce3b5719 [SPARK-45063][PYTHON][DOCS] Refine docstring of `max_by/min_by`
    omit 792b37c1ab46 [SPARK-45031][INFRA] Choose the right merge code path and merge hash for reopened PRs
    omit 9e2aafb13739 [SPARK-45036][SQL] SPJ: Simplify the logic to handle partially clustered distribution
    omit 85d1c7f3a5dd [SPARK-45067][BUILD] Upgrade slf4j to 2.0.9
    omit ba1c2f3b3839 [SPARK-44890][BUILD] Update miswritten remarks
    omit 32a87f03da7e [SPARK-44846][SQL] Convert the lower redundant Aggregate to Project in RemoveRedundantAggregates
    omit 416207659aad [SPARK-45033][SQL] Support maps by parameterized `sql()`
    omit b0b7835bee28 [SPARK-45059][CONNECT][PYTHON] Add `try_reflect` functions to Scala and Python
    omit d03ebced0efa [SPARK-45060][SQL] Fix an internal error from `to_char()`on `NULL` format
    omit f2a6c97d7188 [SPARK-44876][PYTHON][FOLLOWUP] Fix Arrow-optimized Python UDF to delay wrapping the function with fail_on_stopiteration
    omit 60d8fc49bec5 [SPARK-45042][BUILD] Upgrade jetty to 9.4.52.v20230823
    omit 74c1f02531e7 [SPARK-44940][SQL] Improve performance of JSON parsing when "spark.sql.json.enablePartialResults" is enabled
    omit cf0a5cb472ef [SPARK-45045][SS] Revert back the behavior of idle progress for StreamingQuery API from SPARK-43183
    omit 5b609598503d [SPARK-45052][SQL][PYTHON][CONNECT] Make function aliases output column name consistent with SQL
    omit f258af5a98b8 [SPARK-45058][PYTHON][DOCS] Refine docstring of DataFrame.distinct
    omit 09802865fd75 [SPARK-45053][PYTHON][MINOR] Log improvement in python version mismatch
    omit 6d1be3a389e4 [SPARK-44667][INFRA][FOLLOWUP] Uninstall `deepspeed` libraries for non-ML jobs
    omit 3ca57ae7a9bc [SPARK-45038][PYTHON][DOCS] Refine docstring of `max`
    omit 79cc7f838f01 [SPARK-45032][CONNECT] Fix compilation warnings related to `Top-level wildcard is not allowed and will error under -Xsource:3`
    omit 7a01ba65b740 [SPARK-45061][SS][CONNECT] Clean up Running python StreamingQueryLIstener processes when session expires
    omit 3e22c8653d72 [SPARK-44640][PYTHON][FOLLOW-UP] Update UDTF error messages to include method name
    omit 967aac1171a4 [SPARK-44956][BUILD] Upgrade Jekyll to 4.3.2 & Webrick to 1.8.1
    omit 82d54fc89246 [SPARK-45043][BUILD] Upgrade `scalafmt` to 3.7.13
    omit f0fb434c268f [SPARK-45026][CONNECT][FOLLOW-UP] Code cleanup
    omit e9962e893358 [SPARK-45054][SQL] HiveExternalCatalog.listPartitions should restore partition statistics
    omit e4ebb372fa18 [SPARK-44901][SQL] Add API in Python UDTF 'analyze' method to return partitioning/ordering expressions
    omit df534c355d90 [SPARK-44952][SQL][PYTHON] Support named arguments in aggregate Pandas UDFs
    omit e86849a84323 [SPARK-45037][INFRA] Upload unit tests log files for timeouted cancel
    omit 8c27de68756d [SPARK-44750][PYTHON][CONNECT][TESTS][FOLLOW-UP] Avoid creating session twice in `SparkConnectSessionWithOptionsTest`
    omit 71d531300967 [SPARK-44942][INFRA] Use Jira notification options to sync with Github
    omit e4114f67e12a [SPARK-45048][CONNECT] Add additional tests for Python client and attachable execution
    omit 00f66994c802 [SPARK-44577][SQL] Fix INSERT BY NAME returns nonsensical error message
    omit 9d28bef2f70b [SPARK-44743][SQL] Add `try_reflect` function
    omit fd7acd32895e [SPARK-44238][CORE][SQL] Introduce a new `readFrom` method with byte array input for `BloomFilter`
    omit 74447ab5817b [SPARK-44162][CORE] Support G1GC in spark metrics
    omit b0f9978ec08c [SPARK-45026][CONNECT] `spark.sql` should support datatypes not compatible with arrow
    omit 10f31904636d [SPARK-45029][CONNECT][TESTS] Ignore `from_protobuf messageClassName/from_protobuf messageClassName options` in `PlanGenerationTestSuite`
    omit c783e5abb58a [SPARK-44990][SQL][FOLLOWUP] Remove lazy of `nullAsQuotedEmptyString`
    omit 4bb3aaeb4c3f [SPARK-45028][PYTHON][DOCS] Refine docstring of `DataFrame.drop`
    omit 63365e7c0f24 [SPARK-44994][PYTHON][DOCS] Refine docstring of DataFrame.filter
    omit e72ce91250a9 [SPARK-44987][SQL] Assign a name to the error class `_LEGACY_ERROR_TEMP_1100`
    omit 723a0aa30f9a Revert "[SPARK-43646][CONNECT][TESTS] Make both SBT and Maven use `spark-proto` uber jar to test the `connect` module"
    omit 9d84369ade67 [SPARK-45018][PYTHON][CONNECT] Add CalendarIntervalType to Python Client
    omit 9a023c479c6a [SPARK-45014][CONNECT] Clean up fileserver when cleaning up files, jars and archives in SparkContext
    omit ed03173f33aa [SPARK-45024][PYTHON][CONNECT] Filter out some configurations in Session Creation
    omit a45a3a3d60cb [SPARK-45012][SQL] CheckAnalysis should throw inlined plan in AnalysisException
    omit 00fb1851def2 [SPARK-45015][PYTHON][DOCS] Refine DocStrings of `try_{add, subtract, multiply, divide, avg, sum}`
    omit 171251518b46 [SPARK-44742][PYTHON][DOCS][FOLLOWUP] Upgrade `pydata_sphinx_theme` to 0.8.0 in `spark-rm` Dockerfile
    omit caceb888510a [SPARK-45016][PYTHON][CONNECT] Add missing `try_remote_functions` annotations
    omit 1c2f474e3516 [SPARK-45017][PYTHON] Add `CalendarIntervalType` to PySpark
    omit 8bdbaabc8d81 [SPARK-44997][DOCS] Align example order (Python -> Scala/Java -> R) in all Spark Doc Content
    omit dac750b855c3 [SPARK-44990][SQL] Reduce the frequency of get `spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv`
    omit c76e3f61dd7b [SPARK-45005][CONNECT][PS][TESTS] Reducing the CI time by splitting the slow tests
    omit 5df4e1654609 [SPARK-45021][BUILD] Remove `antlr4-maven-plugin` configuration from `sql/catalyst/pom.xml`
    omit edd210697272 [SPARK-43923][CONNECT][FOLLOW-UP] Propagate extra tags to SparkListenerConnectOperationFinished
    omit 27f9ac206753 [SPARK-44239][SQL] Free memory allocated by large vectors when vectors are reset
    omit 742a892d58e4 [SPARK-42304][SQL] Rename `_LEGACY_ERROR_TEMP_2189` to `GET_TABLES_BY_TYPE_UNSUPPORTED_BY_HIVE_VERSION`
    omit eb2d7f5f7547 [SPARK-45008][INFRA] Improve branch suggestion for backporting
    omit b23b6e64572f [SPARK-45010][INFRA] Limit GHA job execution time to up to 5 hours in `build_and_test.yml`
    omit 1ceb39ff99b3 [SPARK-45011][PYTHON][DOCS] Refine docstring of Column.between
    omit 548d8a9a1572 [SPARK-44728][PYTHON][DOCS] Add examples to approxQuantile docstring
    omit b939acd9072e [SPARK-45002][SS] Avoid uncaught exception from state store maintenance task thread on error to prevent executor being killed
    omit a7eef2116919 [SPARK-43438][SQL] Error on missing input columns in `INSERT`
    omit 8505084bc265 [SPARK-45003][PYTHON][DOCS] Refine docstring of `asc/desc`
    omit e5728e2fb706 [SPARK-44999][CORE] Refactor `ExternalSorter` to reduce checks on `shouldPartition` when calling `getPartition`
    omit 6c4ae354c53c [SPARK-45007][INFRA] Fix merged pull requests resolution
    omit 42ca7c57bc1a [SPARK-45006][UI] Use the same date format of other UI date elements for the x-axis of timelines
    omit ea57e7904fce [SPARK-45001][PYTHON][CONNECT] Implement DataFrame.foreachPartition
    omit 281f174304a5 [SPARK-44981][PYTHON][CONNECT][FOLLOW-UP] Explicitly pass runtime configurations only
    omit 0a8a29807b55 [SPARK-44967][SQL][CONNECT] Unit should be considered first before using Boolean for TreeNodeTag
    omit 76a32d3df0fe [SPARK-45000][PYTHON][CONNECT] Implement DataFrame.foreach
    omit 7315a046e22e [SPARK-44996][K8S] Use `lazy val` for `DefaultVolcanoClient`
    omit df63adf73437 [SPARK-43646][CONNECT][TESTS] Make both SBT and Maven use `spark-proto` uber jar to test the `connect` module
    omit 600f62e0edd9 [SPARK-44860][SQL] Add SESSION_USER function
    omit c39a82593c3b [SPARK-44965][PYTHON] Hide internal functions/variables from `pyspark.sql.functions`
    omit c596fceebb9e [SPARK-44995][K8S] Promote `SparkKubernetesClientFactory` to `DeveloperApi`
    omit 5db58f92538d [SPARK-44993][CORE] Add `ShuffleChecksumUtils.compareChecksums` by reusing `ShuffleChecksumTestHelp.compareChecksums`
    omit 8e779d1af75c [SPARK-44989][INFRA] Add a directional message to promote JIRA_ACCESS_TOKEN
    omit 49da438ece84 [SPARK-44868][SQL][FOLLOWUP] Invoke the `to_varchar` function in Scala API
    omit 50d9a56f824a [SPARK-44832][CONNECT] Make transitive dependencies work properly for Scala Client
    omit c8a209251374 [SPARK-44985][CORE] Use toString instead of stacktrace for task reaper threadDump
    omit 8523ee5d90f8 [SPARK-44972][INFRA] Eagerly check if the token is valid to align with the behavior of username/password auth
    omit d95e8f3c65e5 [SPARK-44867][CONNECT][DOCS] Refactor Spark Connect Docs to incorporate Scala setup
    omit 4946d025b620 [SPARK-44983][SQL] Convert binary to string by `to_char` for the formats: `hex`, `base64`, `utf-8`
    omit f0b04286022e [SPARK-44974][CONNECT] Null out SparkSession/Dataset/KeyValueGroupedDatset on serialization
    omit 474f64a88502 [SPARK-44984][PYTHON][CONNECT] Remove `_get_alias` from DataFrame
    omit 5b69dfd67e35 [SPARK-44982][CONNECT] Mark Spark Connect server configurations as static
    omit 64636aff61aa [SPARK-44981][PYTHON][CONNECT] Filter out static configurations used in local mode
    omit 421ff4e3c047 [MINOR][SQL][DOC] Fix incorrect link in sql menu and typo
    omit 5291c6c9274a [SPARK-44980][PYTHON][CONNECT] Fix inherited namedtuples to work in createDataFrame
    omit b7788f9b552d [SPARK-44960][UI] Unescape and consist error summary across UI pages
    omit de96a86d9dc3 [SPARK-44091][YARN][TESTS] Introduce `withResourceTypes` to `ResourceRequestTestHelper` to restore `resourceTypes` as default value after testing
    omit d40447b2234a [SPARK-44945][DOCS][PYTHON] Automate PySpark error class documentation
    omit 4a4856207d41 [SPARK-44897][SQL] Propagating local properties to subquery broadcast exec
    omit eacd234a8876 [SPARK-44963][PYTHON][ML][TESTS] Make PySpark (pyspark-ml module) tests passing without any optional dependency
    omit fb3b0684e9cb [SPARK-44978][SQL][TEST] Fix SQLQueryTestSuite unable create table normally
    omit 7866da9519a5 [SPARK-44497][WEBUI] Show task partition id in Task table
    omit 84a65bd1cf95 [SPARK-42944][PYTHON][FOLLOW-UP] Rename tests from foreachBatch to foreach_batch
    omit 9326615592ea [SPARK-44784][CONNECT] Make SBT testing hermetic
    omit 23ce9c46fa80 [SPARK-44964][ML][CONNECT][TESTS] Clean up pyspark.ml.connect.functions doctest
    omit 04339e30dbdd [SPARK-44975][SQL] Remove BinaryArithmetic useless override resolved
    omit 4f8a1991e793 [SPARK-44968][BUILD] Downgrade ivy from 2.5.2 to 2.5.1
    omit 50f20fbad36d [SPARK-44969][SQL] Reuse `ArrayInsert` in `ArrayAppend`
    omit 12f3c81c26ef [SPARK-44961][PYTHON][CONNECT][TESTS] Make PySpark (pyspark-connect module) tests passing without any dependency
    omit 632dd3319b56 [SPARK-44966][CORE][CONNECT] Change the never changed `var` to `val`
    omit 428a7a1f981b [SPARK-44865][SS] Make StreamingRelationV2 support metadata column
    omit 2583bd2c16a3 [SPARK-44958][PYTHON][CONNECT][TESTS] Add a test to validate the parity of functions
    omit 0b3a5823589b [SPARK-44839][SS][CONNECT] Better Error Logging when user tries to serialize spark session
    omit bf3bef1664f4 [SPARK-44840][SQL][FOLLOWUP] Change the version from 3.5.0 to 3.4.2 for `spark.sql.legacy.negativeIndexInArrayInsert`
    omit 3f7c36106124 [SPARK-44962][INFRA] Add Java 21 `Maven` build to `build_and_test.yml`
    omit c29b7e453aa6 [SPARK-42017][PYTHON][CONNECT][FOLLOWUP] Avoid double validation in `__getattr__ `
    omit 994389f42a40 [SPARK-44930][SQL] Deterministic ApplyFunctionExpression should be foldable
    omit fb45476d58c7 [SPARK-44957][PYTHON][SQL][TESTS] Make PySpark (pyspark-sql module) tests passing without any dependency
    omit 47555da2ae29 [SPARK-44547][CORE] Ignore fallback storage for cached RDD migration
    omit f8211120112c [SPARK-44863][UI][FOLLOWUP] Move Mima rules to v40excludes
    omit 9e9d66b1c6ef [SPARK-44944][INFRA] Auto grant contributor role to first-time contributors
    omit 492064dd95ea [SPARK-44822][PYTHON][FOLLOW-UP] Make Python UDTFs by default non-deterministic
    omit 4748d858b447 [SPARK-44820][DOCS] Switch languages consistently across docs for all code snippets
    omit 5dda2fa49255 [SPARK-44948][DOCS][TESTS][PYTHON] Update document & test related to `Int64Index`
    omit e79e8797e446 [SPARK-44827][PYTHON][TESTS] Fix test when ansi mode enabled
    omit ce6b5f3751e9 [SPARK-44918][SQL][PYTHON] Support named arguments in scalar Python/Pandas UDFs
    omit 4661275f0950 [SPARK-44938][SQL] Change default value of `spark.sql.maxSinglePartitionBytes` to 128m
    omit ce12f6dbad2d [SPARK-41471][SQL] Reduce Spark shuffle when only one side of a join is KeyGroupedPartitioning
    omit 517fcd37046c [SPARK-44801][SQL][UI] Capture analyzing failed queries in Listener and UI
    omit c6d4912341fd [SPARK-44941][SQL][TESTS] Turn off hive.conf.validation in tests
    omit 3b405948ee47 [SPARK-44934][SQL] Use outputSet instead of output to check if column pruning occurred in PushdownPredicateAndPruneColumnsForCTEDef
    omit 2d44848f12cf [SPARK-44435][SS][CONNECT] Tests for foreachBatch and Listener
    omit f7a0b3ab50d8 [SPARK-44863][UI] Add a button to download thread dump as a txt in Spark UI
    omit a824a6de89fd [SPARK-44121][CONNECT][TESTS] Renable Arrow-based connect tests in Java 21
    omit 897b87fa1288 [SPARK-44302][BUILD] Reenable PySpark test on the daily test of Java 21 after the new arrow version release
    omit 9be5ea74a9a3 [SPARK-44097][SPARK-44229][SQL][TESTS] Reenable PandasUDF and o.a.s.sql.execution.arrow tests in Java 21
    omit 17f27ab5c9ac [SPARK-44127][R][TESTS] Reenable test_sparkSQL_arrow.R in Java 21
    omit d4638dc398d2 [SPARK-44247][BUILD] Upgrade Arrow to 13.0.0
    omit c660eb367c0b [SPARK-43943][FOLLOWUP] Correct a function alias
    omit 0c1342ea0694 [SPARK-44939][R] Support Java 21 in SparkR SystemRequirements
    omit 830500150f7e [SPARK-44929][TESTS] Standardize log output for console appender in tests
    omit 4f8c5ede6fae [SPARK-44928][PYTHON][DOCS] Replace the module alias 'sf' instead of 'F' in pyspark.sql import functions
    omit 6a604a4e31af [SPARK-44936][CORE] Simplify the log when Spark HybridStore hits the memory limit
    omit b93b31f633fc [MINOR][DOCS] Fix typos in `pyspark_upgrade.rst`
    omit 6c3e5346d4ee [SPARK-44903][PYTHON][DOCS] Refine docstring of `approx_count_distinct`
    omit 9c68d03c3003 [SPARK-42017][PYTHON][CONNECT] df['col_name']` should validate the column name
    omit c2e3171f3d38 [SPARK-44750][PYTHON][CONNECT] Apply configuration to sparksession during creation
    omit d382c6b3aef2 [SPARK-44935][K8S] Fix `RELEASE` file to have the correct information in Docker images if exists
    omit 2a3aec1f9040 [SPARK-44906][K8S] Make `Kubernetes[Driver|Executor]Conf.annotations` substitute annotations instead of feature steps
    omit 2d0a0a00cb5d [SPARK-44816][CONNECT] Improve error message when UDF class is not found
    omit dd6cda5b614b [SPARK-44861][CONNECT] jsonignore SparkListenerConnectOperationStarted.planRequest
    omit 378b76e79781 [MINOR][PYTHON] Code cleanup: remove resolved todo items
    omit 75e6b731cced [SPARK-44549][SQL] Support window functions in correlated scalar subqueries
    omit 611e17e89260 [SPARK-44914][BUILD] Upgrade `Apache ivy` from 2.5.1 to 2.5.2
    omit 0d1b5975b2d3 [SPARK-44908][ML][CONNECT] Fix cross validator foldCol param functionality
    omit 4d90c597ec90 [SPARK-44923][PYTHON][BUILD] Some directories should be cleared when regenerating files
    omit 80668dc1a36a [SPARK-44909][ML] Skip starting torch distributor log streaming server when it is not available
    omit 00cd5e846b6b [SPARK-44899][PYTHON][DOCS] Refine the docstring of DataFrame.collect
    omit 3d83978fc4d2 [SPARK-44921][SQL] Remove SqlBaseLexer.tokens from codebase
    omit 2137606a6686 [SPARK-44920][CORE] Use await() instead of awaitUninterruptibly() in TransportClientFactory.createClient()
    omit 7b1a3494107b [SPARK-44925][K8S] K8s default service token file should not be materialized into token
    omit 34b51cc78447 [SPARK-44922][TESTS] Disable o.a.p.h.InternalParquetRecordWriter logs for tests
    omit a0c029c18b0e [SPARK-44878][SS] Disable strict limit for RocksDB write manager to avoid insertion exception on cache full
    omit 13588c10cbc3 [SPARK-44905][SQL] Stateful lastRegex causes NullPointerException on eval for regexp_replace
    omit 510e7c2359d0 [SPARK-43567][FOLLOWUP] Missing backtick from migration guide
    omit 49ba2443c118 [SPARK-44742][PYTHON][DOCS] Add Spark version drop down to the PySpark doc site
    omit 7e5250a797e3 [SPARK-44907][PYTHON][CONNECT] DataFrame.join` should throw IllegalArgumentException for invalid join types
    omit 9b757687e1e1 [SPARK-44916][DOCS][TESTS] Document Spark Driver Live Log UI
    omit ce50a563d311 [SPARK-44840][SQL] Make `array_insert()` 1-based for negative indexes
    omit 24293cab2de0 [SPARK-43780][SQL][FOLLOWUP] Fix the config doc `spark.sql.optimizer.decorrelateJoinPredicate.enabled`
    omit bd8fbf465796 [SPARK-44871][SQL] Fix percentile_disc behaviour
    omit 65b8ca2694c2 [MINOR][PYTHON][DOCS] Remove duplicated versionchanged per versionadded
    omit 16607a5fd03f [SPARK-44885][SQL] NullPointerException is thrown when column with ROWID type contains NULL values
    omit dc900b47556d [SPARK-44786][SQL][CONNECT] Convert common Spark exceptions
    omit 1a4cacf3f920 [SPARK-44813][INFRA][FOLLOWUP] Make the jira library optional
    omit 69f54bbe35fd [MINOR][SQL] Merge RefreshTable case on ResolveSessionCatalog
    omit 1569ab543d2b [SPARK-42768][SQL] Enable cached plan apply AQE by default
    omit 534f9ea31196 [SPARK-44904][PYTHON][DOCS] Correct the `versionadded` of `sql.functions.approx_percentile` to 3.5.0
    omit 9fdf65aefc55 [SPARK-44854][PYTHON] Python timedelta to DayTimeIntervalType edge case bug
    omit 206554f12790 [SPARK-44751][SQL] XML FileFormat Interface implementation
    omit b137b28f6f56 [SPARK-44214][CORE] Support Spark Driver Live Log UI
    omit 4646991abd7f [SPARK-44776][CONNECT] Add ProducedRowCount to SparkListenerConnectOperationFinished
    omit 8d4ca0aa580f [SPARK-43563][SPARK-43459][SPARK-43451][SPARK-43506] Remove `squeeze` from `read_csv` & enabling more tests
    omit 290b6327faad [SPARK-44891][PYTHON][CONNECT] Enable Doctests of `rand`, `randn` and `log`
    omit 7f3a439fe31e [SPARK-44748][SQL] Query execution for the PARTITION BY clause in UDTF TABLE arguments
    omit 109e9cd20d5d [SPARK-44898][BUILD] Upgrade `gcs-connector` to 2.2.17
    omit b9c4fa4bc312 Revert "[SPARK-44801][SQL][UI] Capture analyzing failed queries in Listener and UI"
    omit c34ec4112444 [SPARK-44818] Fix race for pending task kill issued before taskThread is initialized
    omit 72c62b6596d2 [SPARK-44889][PYTHON][CONNECT] Fix docstring of `monotonically_increasing_id`
    omit 3ab20641f884 [SPARK-44888][SQL][TESTS] Regenerate golden files of `SQLQueryTestSuite` for Java 21
    omit a21e19b6e7ac [SPARK-44841][FOLLOWUP] Add migration guide for the behavior change
    omit 1f92995249b3 [SPARK-44801][SQL][UI] Capture analyzing failed queries in Listener and UI
    omit 04024fd573a2 [SPARK-44887][DOCS] Fix wildcard import `from pyspark.sql.functions import *` in `Quick Start` Examples
    omit f351e7b4eddb [SPARK-43205][DOCS][SQL][FOLLOWUP] IDENTIFIER clause docs
    omit e73487c4348c [SPARK-44785][SQL][CONNECT] Convert common alreadyExistsExceptions and noSuchExceptions
    omit 48ab6553023d [SPARK-44868][SQL] Convert datetime to string by `to_char`/`to_varchar`
    omit 5151b5ba9630 [SPARK-44877][CONNECT][PYTHON] Support python protobuf functions for Spark Connect
    omit ef6327545e2a [SPARK-44842][SPARK-43812][PS] Support stat functions for pandas 2.0.0 and enabling tests
    omit 52c4673ed04f [SPARK-44858][PYTHON][DOCS] Refine dostring of DataFrame.isEmpty
    omit 3d5a7d907122 [SPARK-40178][SQL][COONECT] Support coalesce hints with ease for PySpark and R
    omit 536ac30d3ca4 [SPARK-44879][PYTHON][DOCS] Refine the docstring of spark.createDataFrame
    omit 75c0b8b61ca5 [SPARK-44876][PYTHON] Fix Arrow-optimized Python UDF on Spark Connect
    omit 169af1191fce [SPARK-44882][PYTHON][CONNECT] Remove function uuid/random/chr from PySpark
    omit 3f380b9ecc8b [SPARK-44880][UI] Remove unnecessary right curly brace at the end of the thread locks info
    omit 2e2f5e9c28b4 [SPARK-44782][INFRA] Adjust PR template to Generative Tooling Guidance recommendations
    omit c7d63eac48d3 [SPARK-44873] Support alter view with nested columns in Hive client
    omit 8fb799d47bbd [SPARK-44813][INFRA] The Jira Python misses our assignee when it searches users again
    omit 5d267fe79e01 Revert "[SPARK-44813][INFRA] The Jira Python misses our assignee when it searches users again"
    omit 54dd18b5e095 [SPARK-44875][INFRA] Fix spelling for commentator to test SPARK-44813
    omit fd8436ae785a [SPARK-44869][DOC] Add doc for insert by name statement
    omit 1292b5665502 [SPARK-44852][BUILD] Exclude `junit-jupiter-api` from `curator-test`
    omit 00255bc63b1a [SPARK-44813][INFRA] The Jira Python misses our assignee when it searches users again
    omit 68345e7f5d9b [SPARK-44289][FOLLOWUP] Cleanup doctest
    omit c88ced88af9a [SPARK-44729][PYTHON][DOCS] Add canonical links to the PySpark docs page
    omit b37daf5695e5 [SPARK-44740][CONNECT][FOLLOW] Fix metadata values for Artifacts
    omit fc0be7ebace3 [SPARK-44853][PYTHON][DOCS] Refine docstring of DataFrame.columns property
    omit bb41cd889efd [SPARK-44834][PYTHON][SQL][TESTS][FOLLOW-UP] Update the analyzer results of the udtf tests
    omit 960c027fb91b [SPARK-44289][SPARK-43874][SPARK-43869][SPARK-43607][PS] Support `indexer_between_time` for pandas 2.0.0 & enabling more tests
    omit 02a07cd6adc7 [SPARK-43205][DOC] identifier clause docs
    omit f807bd239aeb [SPARK-44857][CORE][UI] Fix `getBaseURI` error in Spark Worker LogPage UI buttons
    omit c321b3dd66f6 [SPARK-43875][PS][TESTS] Enabling Categorical tests for Pandas 2.0.0 and above
    omit 48faaa8ee73d [SPARK-44831][PYTHON][DOCS] Refine DocString of `DataFrame.{union, unionAll, unionByName}`
    omit 07f70227f8bf [SPARK-44859][SS] Fix incorrect property name in structured streaming doc
    omit 6771d9d757f9 [SPARK-44433][3.5X] Terminate foreach batch runner when streaming query terminates
    omit be04ac1ace91 [SPARK-44834][PYTHON][SQL][TESTS] Add SQL query tests for Python UDTFs
    omit 047b2247879c [SPARK-44849] Expose SparkConnectExecutionManager.listActiveExecutions
    omit 72a466835a44 [SPARK-44806][CONNECT] Move internal client spark-connect-common to be able to test real in-process server with a real RPC client
    omit 026aa4fdbdca [SPARK-43462][SPARK-43871][PS][TESTS] Enable `SeriesDateTimeTests` for pandas 2.0.0 and above
    omit a960e71905e3 [SPARK-44841][PS] Support `value_counts` for pandas 2.0.0 and above
    omit 4952a03fdc22 [SPARK-44721][CONNECT] Revamp retry logic and make retries run for 10 minutes
    omit fce83d499939 [SPARK-44822][PYTHON][SQL] Make Python UDTFs by default non-deterministic
    omit 06959e2b812b [SPARK-44836][PYTHON] Refactor Arrow Python UDTF
    omit 2a5e75818ffa [SPARK-43618][SPARK-43658][CONNECT][PS][TESTS] Enabling more tests
    omit 6ae4f4f751fe [SPARK-44829][CONNECT] Expose uploadAllArtifactClasses in ArtifactManager to `sql` package
    omit 11cbdc291c96 [SPARK-43509][PYTHON][CONNECT][FOLLOW-UP] Check SPARK_CONNECT_MODE_ENABLED when creating a session
    omit 80118e2c688c [SPARK-44807][CONNECT] Add Dataset.metadataColumn to Scala Client
    omit 86a04f4c5f94 [SPARK-44802][INFRA][FOLLOWUP] Fix to consider JIRA_ACCESS_TOKEN in precheck conditions
    omit 8c97b3c48a7e [SPARK-44828][BUILD] Upgrade ORC to 1.9.1
    omit e25a6541cf8f [SPARK-44821][BUILD][K8S] Upgrade `kubernetes-client` to 6.8.1
    omit cb8c653f3c7f [SPARK-44799][CONNECT] Fix outer scopes resolution on the executor side
    omit ed906e0f7b1d [SPARK-44795][CONNECT][TESTS][FOLLOWUP] Fix test case `REPL class in UDF` for Scala 2.13
    omit 7d96f12784de [SPARK-44824][CONNECT][TESTS] Reset `ammoniteOut` in the `afterEach` method of `ReplE2ESuite`
    omit 2be20e54a222 [SPARK-44809][SS] Remove unused RocksDB custom metrics for pause/writeBatch
    omit ac567b36db90 [SPARK-44705][FOLLOWUP] Fix Deprecation Version of ContetAwareIterator
    omit b9f11143d058 [SPARK-42664][CONNECT] Support `bloomFilter` function for `DataFrameStatFunctions`
    omit 2ab404ff179a [SPARK-44794][CONNECT] Make Streaming Queries work with Connect's artifact management
    omit afe188687ba7 [SPARK-44803][BUILD] Replace `publish` with `publishOrSkip` in SparkBuild to eliminate warnings
    omit 1ae1d89d08a7 [SPARK-43205][SQL][FOLLOWUP] IDENTIFIER clause should accept alias and RuntimeReplaceable
    omit afcccb42c96c [SPARK-44718][SQL] Match ColumnVector memory-mode config default to OffHeapMemoryMode config value
    omit a53a9e93bd75 [SPARK-44653][SQL][FOLLOWUP] ResolveUnion should not combine Unions
    omit 6c85ec3160cf [SPARK-44791][CONNECT][TESTS][FOLLOWUP] Fix test case `Collect REPL generated class` for Scala 2.13
    omit c36d54a569e5 [SPARK-44802][INFRA] Token based ASF JIRA authentication
    omit 9f897190bdb2 [SPARK-44798][BUILD] Fix Scala 2.13 mima check after SPARK-44705 merged
    omit dc5fdb4aa1cd [SPARK-44796][BUILD][CONNECT] Remove `grpc-java` plugin related configuration from the `connect/connect-client-jvm` module
    omit c9ff70253999 [SPARK-43885][SQL][FOLLOWUP] Instruction#dataType should not fail
    omit 46580ab4cb02 [SPARK-43380][SQL] Revert `Fix Avro data type conversion issues`
    omit 7e5216943357 [SPARK-43063][SQL][FOLLOWUP] Add a space between -> and value when first value is null
    omit 420e6878c68f [SPARK-43780][SQL] Support correlated references in join predicates for scalar and lateral subqueries
    omit f7002fb25ca9 [SPARK-44749][PYTHON][FOLLOWUP][TESTS] Add more tests for named arguments in Python UDTF
    omit 1d562904e4e7 [SPARK-44795][CONNECT] CodeGenerator Cache should be classloader specific
    omit d4629563492e [SPARK-44749][SQL][PYTHON] Support named arguments in Python UDTF
    omit 0fc85a24a153 [SPARK-44783][SQL][TESTS] Checks arrays as named and positional parameters
    omit a50170b888f4 [SPARK-44792][BUILD] Upgrade curator to 5.2.0
    omit 481546385a60 Revert "Add test"
    omit 3c1ea33a3bfd Revert "Add classloader Id to code generation cache."
    omit 8a0e0591c83f Revert "Refine solution"
    omit 890748873bd8 Refine solution
    omit 6d4891b32cee Add classloader Id to code generation cache.
    omit 2c465f681ada Add test
    omit 6bd95d0e0045 [SPARK-44775][PYTHON][DOCS] Add missing version information in DataFrame APIs
    omit dcf3d582293c [SPARK-44791][CONNECT] Make ArrowDeserializer work with REPL generated classes
    omit f496cd1ee2a7 [SPARK-44736][CONNECT] Add Dataset.explode to Spark Connect Scala Client
    omit 7070b3672d84 [MINOR][SQL] Rename shouldBroadcast to isDynamicPruning in InSubqueryExec
    omit 2f9c8ac25ba6 [SPARK-44242][CORE][FOLLOWUP] Use the `assertThrows` method to fix Java linter issue
    omit 295c615b16b8 [SPARK-44404][SQL] Assign names to the error class _LEGACY_ERROR_TEMP_[1009,1010,1013,1015,1016,1278]
    omit b9fc5c03ed69 [SPARK-44778][SQL] Add the alias `TIMEDIFF` for `TIMESTAMPDIFF`
    omit 7baf9dad1962 [SPARK-44719][SQL] Fix NoClassDefFoundError when using Hive UDF
    omit 567823eb81b7 [SPARK-44242] Improve Max Heap not set check
    omit 1d02ca499769 [SPARK-43987][SHUFFLE] Separate finalizeShuffleMerge Processing to Dedicated Thread Pools
    omit e38d06be77e0 [SPARK-44740][CONNECT][FOLLOW] Fix metadata values in Python client
    omit 3ed99206cc2a [SPARK-44422][FOLLOWUP][CONNECT] Fix typo in ProtoUtils
    omit 3c5f6a2ebf8a [SPARK-44625][CONNECT][FOLLOWUP] Make initialization of SparkConnectExecutionManager lazy
    omit f462b6319afc [SPARK-43979][SQL][FOLLOWUP] Fix the detection of alias-only project
    omit 84dbe846b30d [SPARK-44625][CONNECT] SparkConnectExecutionManager to track all executions
    omit 5a27dc9b6174 [SPARK-44761][CONNECT] Support DataStreamWriter.foreachBatch(VoidFunction2)
    omit cb16591f9b7a [SPARK-44760][INFRA] Fix list index out of range for JIRA resolution in merge_spark_pr
    omit 6729f497bc19 [SPARK-43476][SPARK-43477][SPARK-43478][PS] Support `StringMethods` for pandas 2.0.0 and above
    omit 73b0376ec152 [SPARK-44731][PYTHON][CONNECT] Make TimestampNTZ works with literals in Python Spark Connect
    omit 92f2d654bdcc [SPARK-44770][WEBUI] Add a displayOrder variable to WebUITab to specify the order in which tabs appear
    omit e3caa4737768 [SPARK-44727][CORE][DOCS] Improve docs and error message for dynamic allocation conditions
    omit 5bb9cd8e55ff [SPARK-44737][SQL][UI] Should not display json format errors on SQL page for non-SparkThrowables on SQL Tab
    omit b828ff0bdba8 [SPARK-44771][INFRA] Remove 'sudo' in 'pip install' suggestions of dev scripts
    omit 13732922cca4 [SPARK-44763][SQL] Fix a bug of promoting string as double in binary arithmetic with interval
    omit 88de5663cc8d Revert "[SPARK-44760][INFRA] Fix list index out of range for JIRA resolution in merge_spark_pr"
    omit 1c3f618ee388 [SPARK-43872][PS] Support `(DataFrame|Series).plot` with pandas 2.0.0 and above
    omit 8aaff5583949 [SPARK-44705][PYTHON] Make PythonRunner single-threaded
    omit 9bde882fcb39 [SPARK-44765][CONNECT] Simplify retries of ReleaseExecute
    omit 42eb42236286 [SPARK-43032][FOLLOWUP][SS][CONNECT] StreamingQueryManager bug fix
    omit 8fe50fcdb6a3 [SPARK-44766][PYTHON] Cache the pandas converter for reuse for Python UDTFs
    omit 5db87787d5cc [SPARK-43781][SQL] Fix IllegalStateException when cogrouping two datasets derived from the same source
    omit 3164ff51dd24 [SPARK-44760][INFRA] Fix list index out of range for JIRA resolution in merge_spark_pr
    omit 36b5429abc73 [SPARK-44741][CORE] Support regex-based MetricFilter in `StatsdSink`
    omit bd14d6412a31 [SPARK-43660][CONNECT][PS][FOLLOWUP] Remove JVM dependency for resample
    omit 2c9b25a90b57 [SPARK-43245][SPARK-43705][PS] Type match for `DatetimeIndex`/`TimedeltaIndex` with pandas 2
    omit d5f1f17278a7 [SPARK-44691][SQL][CONNECT] Move Subclasses of AnalysisException to sql/api
    omit 55b07b1367a7 [SPARK-44554][INFRA][FOLLOWUP] Install `IPython` for the Python linter check in the daily test for branch-3.3
    omit 5c94565e75cc [MINOR][BUILD] Skip `deepspeed` in requirements on MacOS
    omit 97cfe45c157e [SPARK-44701][PYTHON][TESTS][FOLLOWUP] Keep `torch` in 3.5 daily GA
    omit 0eeb6042859e [SPARK-44461][SS][PYTHON][CONNECT] Verify Python Version for spark connect streaming workers
    omit 1846991b20a0 [SPARK-44732][SQL] Built-in XML data source support
    omit 3b3e0fdc61cf [SPARK-42621][PS] Add inclusive parameter for pd.date_range
    omit d27496eb3bf9 [SPARK-44747][CONNECT] Add missing SparkSession.Builder methods
    omit 7af4e358f3f4 [SPARK-44740][CONNECT] Support specifying `session_id` in SPARK_REMOTE connection string
    omit 4db378fae307 [SPARK-44745][DOCS][K8S] Document shuffle data recovery from the remounted K8s PVCs
    omit 87298db43d9a [SPARK-44503][SQL] Project any PARTITION BY expressions not already returned from Python UDTF TABLE arguments
    omit be9ffb37585f [SPARK-44720][CONNECT] Make Dataset use Encoder instead of AgnosticEncoder
    omit 27c5a1f9f0e3 [SPARK-43429][CONNECT] Deflake SparkSessionSuite
    omit 0b757d3b610a [SPARK-42620][PS] Add `inclusive` parameter for (DataFrame|Series).between_time
    omit e5e0f389dad6 Revert "[SPARK-43660][CONNECT][PS][FOLLOWUP] Remove JVM dependency for resample"
    omit b19a83e236b2 [SPARK-43660][CONNECT][PS][FOLLOWUP] Remove JVM dependency for resample
    omit 4f25df1dc25c [SPARK-43979][SQL][FOLLOWUP] transformUpWithNewOutput` should only be used with new outputs
    omit c73660c3e727 [SPARK-44738][PYTHON][CONNECT] Add missing client metadata to calls
    omit 3ccb03ec5e7d [SPARK-42849][SQL] Session Variables
    omit f9058d69b2af [SPARK-44551][SQL] Fix behavior of null IN (empty list) in expression execution
    omit bdee74b43451 [SPARK-44726][CORE] Improve `HeartbeatReceiver` config validation error message
    omit e584ed4ad96a [SPARK-44581][YARN] Fix the bug that ShutdownHookManager gets wrong UGI from SecurityManager of ApplicationMaster
    omit 3cec5a4c7d8c [SPARK-43709][PS] Remove `closed` parameter from `ps.date_range` & enable test
    omit eccc045a2508 [MINOR][DOC] Fixed deprecated procedure syntax
    omit e05959e1900c [SPARK-44717][PYTHON][PS] Respect TimestampNTZ in resampling
    omit 9c9a4a844da3 [SPARK-43568][SPARK-43633][PS] Support `Categorical` APIs for pandas 2
    omit 66d8e6a3d83d [SPARK-44695][PYTHON] Improve error message for `DataFrame.toDF`
    omit 3920a4189ca3 [SPARK-44723][BUILD] Upgrade `gcs-connector` to 2.2.16
    omit 69bd8358827f [SPARK-44665][PYTHON] Add support for pandas DataFrame assertDataFrameEqual
    omit ea00ecd1f88b [SPARK-44722][CONNECT] ExecutePlanResponseReattachableIterator._call_iter: AttributeError: 'NoneType' object has no attribute 'message'
    omit 3af2e77bd20c [SPARK-44725][DOCS] Document `spark.network.timeoutInterval`
    omit 418bba5ad605 [SPARK-44709][CONNECT] Run ExecuteGrpcResponseSender in reattachable execute in new thread to fix flow control
    omit 8c444f497137 [SPARK-44715][CONNECT] Bring back callUdf and udf function
    omit 93af0848e467 [SPARK-44710][CONNECT] Add Dataset.dropDuplicatesWithinWatermark to Spark Connect Scala Client
    omit c46d4caa5986 [SPARK-44713][CONNECT][SQL] Move shared classes to sql/api
    omit 74fa07c57020 [SPARK-44236][SQL] Disable WholeStageCodegen when set `spark.sql.codegen.factoryMode` to NO_CODEGEN
    omit 29e8331681c6 [SPARK-44714] Ease restriction of LCA resolution regarding queries with having
    omit f9d417fc17a8 [SPARK-44657][CONNECT] Fix incorrect limit handling in ArrowBatchWithSchemaIterator and config parsing of CONNECT_GRPC_ARROW_MAX_BATCH_SIZE
    omit f7879b4c2500 [SPARK-44680][SQL] Improve the error for parameters in `DEFAULT`
    omit d2b60ff51fab [SPARK-43567][PS] Support `use_na_sentinel` for `factorize`
    omit b4b91212b1d4 [SPARK-44703][CORE] Log eventLog rewrite duration when compact old event log files
    omit 590b77f76284 [SPARK-44005][PYTHON] Improve error messages for regular Python UDTFs that return non-tuple values
    omit 25053d981864 [SPARK-44689][CONNECT] Make the exception handling of function `SparkConnectPlanner#unpackScalarScalaUDF` more universal
    omit 6dadd188f365 [SPARK-44554][INFRA] Make Python linter related checks pass of branch-3.3/3.4 daily testing
    omit aa1261dc1296 [SPARK-44641][SQL] Incorrect result in certain scenarios when SPJ is not triggered
    omit 7493c5764f96 [SPARK-43429][CONNECT] Add Default & Active SparkSession for Scala Client
    omit 630b1777904f [SPARK-44683][SS] Logging level isn't passed to RocksDB state store provider correctly
    omit 9368a0f0c100 [SPARK-44694][PYTHON][CONNECT] Refactor active sessions and expose them as an API
    omit f47a2560e6e3 [MINOR][UI] Increasing the number of significant digits for Fraction Cached of RDD
    omit 8911578020f8 [SPARK-44132][SQL] Materialize `Stream` of join column names to avoid codegen failure
    omit 4eea89d33964 [SPARK-44692][CONNECT][SQL] Move Trigger(s) to sql/api
    omit 42e5daddf3ba [SPARK-44575][SQL][CONNECT] Implement basic error translation
    omit d5d3f393f16d [SPARK-43606][PS] Remove `Int64Index` & `Float64Index`
    omit 726ccb532a1b [SPARK-44707][K8S] Use INFO log in `ExecutorPodsWatcher.onClose` if `SparkContext` is stopped
    omit f1a161cb3950 [SPARK-44561][PYTHON] Fix AssertionError when converting UDTF output to a complex type
    omit 2a23c7a18a0b [SPARK-38475][CORE] Use error class in org.apache.spark.serializer
    omit a3a32912be04 [SPARK-44697][CORE] Clean up the deprecated usage of `o.a.commons.lang3.RandomUtils`
    omit bf7654998fbb [SPARK-44686][CONNECT][SQL] Add the ability to create a RowEncoder in Encoders.scala
    omit 433fb2af8a3c [SPARK-44701][PYTHON][TESTS] Skip `ClassificationTestsOnConnect` when `torch` is not installed
    omit f139733b92d4 [SPARK-42321][SQL] Assign name to _LEGACY_ERROR_TEMP_2133
    omit 1f10cc4a5945 [SPARK-44628][SQL] Clear some unused codes in "***Errors" and extract some common logic
    omit df8e52d84d1e [SPARK-44682][PS] Make pandas error class message_parameters strings
    omit 7515061ec237 [SPARK-44264][PYTHON][ML][TESTS][FOLLOWUP] Adding Deepspeed To The Test Dockerfile
    omit a98b11274d95 [SPARK-44693][BUILD] Rename the `object Catalyst` in SparkBuild to `object SqlApi`
    omit 656bf36363c4 [SPARK-44690][SPARK-44376][BUILD] Downgrade Scala to 2.13.8
    omit 9462dcd0e996 [SPARK-41636][SQL] Make sure `selectFilters` returns predicates in deterministic order
    omit a640373fff38 [SPARK-44629][PYTHON][DOCS] Publish PySpark Test Guidelines webpage
    omit d6998979427b [SPARK-44634][SQL] Encoders.bean does no longer support nested beans with type arguments
    omit 74ae1e3434c3 [SPARK-42500][SQL] ConstantPropagation support more case
    omit 41a2a7daeee0 [SPARK-44650][CORE] `spark.executor.defaultJavaOptions` Check illegal java options
    omit d264ee37f316 [SPARK-44688][INFRA] Add a file existence check before executing `free_disk_space` and `free_disk_space_container`
    omit 8332f0b6d33a [SPARK-44687][BUILD] Fix mima check for Scala 2.13 after SPARK-44198 merged
    omit cf64008fce77 [SPARK-44433][PYTHON][CONNECT][SS][FOLLOWUP] Set back USE_DAEMON after creating streaming python processes
    omit 8e60a04d19ed [SPARK-44667][INFRA] Uninstall large ML libraries for non-ML jobs
    omit 8b53fed7ef0e [SPARK-44663][PYTHON] Disable arrow optimization by default for Python UDTFs
    omit 62415dc59627 [SPARK-44644][PYTHON] Improve error messages for Python UDTFs with pickling errors
    omit 780bae928399 [SPARK-44671][PYTHON][CONNECT] Retry ExecutePlan in case initial request didn't reach server in Python client
    omit 2d3bb4d5db71 [SPARK-44672][INFRA] Fix git ignore rules related to Antlr
    omit 4a344b6a37a8 [SPARK-44674][GRAPHX] Remove `BytecodeUtils` from `graphx` module
    omit 84ea6f242e49 [SPARK-44656][CONNECT] Make all iterators CloseableIterators
    omit 6073d7219330 [SPARK-44675][INFRA] Increase ReservedCodeCacheSize for release build
    omit a6f048a3d993 [SPARK-44655][SQL] Make the code cleaner about static and dynamic data/partition filters
    omit e019e8720fb5 [SPARK-44600][INFRA] Make `repl` module to pass Maven daily testing
    omit e4bae48d5aa3 [SPARK-44618][INFRA] Uninstall CodeQL/Go/Node in non-container jobs
    omit bd9dd3887eaf [SPARK-43967][SQL][PYTHON] Add memory limits for Python UDTF analyzer
    omit 2c2d6534bebe [SPARK-44582][SQL] Skip iterator on SMJ if it was cleaned up
    omit ce1fe57cdd70 [SPARK-44653][SQL] Non-trivial DataFrame unions should not break caching
    omit 52437bc73695 [SPARK-44624][CONNECT] Retry ExecutePlan in case initial request didn't reach server
    omit 492f6fac02a0 [SPARK-44664][PYTHON][CONNECT] Release the execute when closing the iterator in Python client
    omit 4ed59d932631 [SPARK-44619][INFRA] Free up disk space for container jobs
    omit 678f47264e08 [SPARK-43562][SPARK-43870][PS] Remove APIs from `DataFrame` and `Series`
    omit 26ed4fbc00dd [SPARK-43873][PS] Enabling `FrameDescribeTests`
    omit 380c0f2033fb [SPARK-44640][PYTHON] Improve error messages for Python UDTF returning non Iterable
    omit 5c36c5804772 [SPARK-44433][PYTHON][CONNECT][SS][FOLLOWUP] Terminate listener process with `removeListener` and improvements
    omit 52a9002fa238 [MINOR][DOC] Fix a typo in ResolveReferencesInUpdate scaladoc
    omit d2d43b888aeb [SPARK-44642][CONNECT] ReleaseExecute in ExecutePlanResponseReattachableIterator after it gets error from server
    omit 16b031eb144f [SPARK-44652] Raise error when only one df is None
    omit 9fbf0b4853c6 [SPARK-44661][CORE][TESTS] `getMapOutputLocation` should not throw NPE
    omit ed036a9d0aab [SPARK-44658][CORE] `ShuffleStatus.getMapStatus` should return `None` instead of `Some(null)`
    omit f824d058b14e [SPARK-42330][SQL] Assign the name `RULE_ID_NOT_FOUND` to the error class `_LEGACY_ERROR_TEMP_2175`
    omit fd624530e3b2 [SPARK-44572][INFRA] Clean up unused installers ASAP
    omit 445e3c395221 [SPARK-44645][PYTHON][DOCS] Update assertDataFrameEqual docs error example output
    omit 3b3e30113262 [SPARK-42730][CONNECT][DOCS] Update Spark Standalone Mode page
    omit c607f1843035 [SPARK-44488][SQL] Support deserializing long types when creating `Metadata` object from JObject
    omit b390c9c55a0c [SPARK-44059][SQL] Add better error messages for SQL named argumnts
    omit 8a8471811727 [SPARK-44643][SQL][PYTHON] Fix Row.__repr__ for the case the field is empty Row
    omit 1a89bdc60d55 [SPARK-44620][SQL][PS][CONNECT] Make `ResolvePivot` retain the `Plan_ID_TAG`
    omit 7b9057b90f93 [SPARK-44424][PYTHON][CONNECT][FOLLOW-UP] Import Connect related libraries after checking dependencies
    omit 68d8e65c7829 [SPARK-44424][CONNECT][PYTHON] Python client for reattaching to existing execute in Spark Connect
    omit 79938ee2a717 [SPARK-44626][SS][CONNECT] Followup on streaming query termination when client session is timed out for Spark Connect
    omit 784f1d0da7f9 [SPARK-44636][CONNECT] Leave no dangling iterators
    omit 6882482fcdfb [SPARK-44421][FOLLOWUP] Minor rename of ResponseComplete to ResultComplete
    omit 26c7e55f1999 [SPARK-44637][CONNECT] Synchronize accesses to ExecuteResponseObserver
    omit cfc75ec12aee [SPARK-44503][SQL] Add analysis and planning for PARTITION BY and ORDER BY clause after TABLE arguments for TVF calls
    omit e326b9b6565b [SPARK-41532][DOCS][FOLLOWUP] Regenerate `sql-error-conditions-connect-error-class.md`
    omit 886f7c8bea89 [SPARK-44562][SQL] Add OptimizeOneRowRelationSubquery in batch of Subquery
    omit 35d47651ed01 [SPARK-44631][CONNECT][CORE] Remove session-based directory when the isolated session cache is evicted
    omit 1bf2a589eb7b [SPARK-44618][INFRA][FOLLOWUPS] Make `free_disk_space` uninstall two unused packages completely
    omit e91c024cf3b7 [SPARK-44280][SQL] Add convertJavaTimestampToTimestamp in JDBCDialect API
    omit 4ec27c3801aa [SPARK-44555][SQL] Use checkError() to check Exception in command Suite & assign some error class names
    omit 47d9fbd407d1 [SPARK-41532][CONNECT][FOLLOWUP] Make the scala client using the same error class as python client
    omit c1c4a79ff172 [SPARK-44421][CONNECT][FOLLOWUP] Minor comment improvements
    omit ca274fe15cf9 [SPARK-44607][SQL] Remove unused function `containsNestedColumn` from `Filter`
    omit 0a9bc446163e [MINOR][CONNECT] Fix some typos in connect server module
    omit 6891f2c0ac8c [SPARK-42497][FOLLOWUPS][TESTS] Add missing UTs to `modules.py`
    omit 4f66f091ad5d [SPARK-44218][PYTHON] Customize diff log in assertDataFrameEqual error message format
    omit 6f6dd38ce542 [SPARK-44588][CORE] Fix double encryption issue for migrated shuffle blocks
    omit cdcf12b9fc70 [SPARK-44601][BUILD] Add `jackson-mapper-asl` as test dependency to `hive-thriftserver` module to make Maven test pass
    omit 9e42805f77c1 [SPARK-44480][SS] Use thread pool to perform maintenance activity for hdfs/rocksdb state store providers
    omit 6933d43cc63c [SPARK-44623][BUILD] Upgrade `commons-lang3` to 3.13.0
    omit f54b40202178 [SPARK-29497][CONNECT] Throw error when UDF is not deserializable
    omit 4f62f8a718e8 [SPARK-44613][CONNECT] Add Encoders object
    omit 5b8eaf8be508 [SPARK-44421][FOLLOWUP] Fix doc test in SparkThrowableSuite
    omit 821026bc730c [SPARK-44311][CONNECT][SQL] Improved support for UDFs on value classes
    omit 8053d5f16541 [SPARK-44604][BUILD] Upgrade Netty to 4.1.96.Final
    omit f64c72c35729 [SPARK-44567][INFRA] Add a new Daily testing GitHub Action job for Maven
    omit a011d16a24db [SPARK-44614][PYTHON][CONNECT] Add missing packages in setup.py
    omit 3f815953d658 [SPARK-44618][INFRA] Free up disk space for non-container jobs
    omit a40e46fe6dc3 [SPARK-44591][CONNECT][WEBUI] Use jobTags in SparkListenerSQLExecutionStart to link SQL Execution IDs for Spark UI Connect page
    omit 546e39c5dabc [SPARK-44490][WEBUI] Remove unused `TaskPagedTable` in StagePage
    omit 3e5203c64c06 [SPARK-44421][SPARK-44423][CONNECT] Reattachable execution in Spark Connect
    omit fd18c34a8513 [SPARK-44615][CONNECT][TESTS] rename spark connect client suites to avoid conflict
    omit 46491403bcc7 [SPARK-44586][INFRA][ML][PYTHON] TorchDistributor` should install cpu-only Torch for testing
    omit 772ebd312520 [SPARK-44583][DOC] `spark.*.io.connectionCreationTimeout` parameter documentation
    omit c858851fc6c2 [SPARK-44579][SQL] Support Interrupt On Cancel in SQLExecution
    omit 9a2abe42cf48 [SPARK-44611][CONNECT] Do not exclude scala-xml
    omit eb4152dfcd6e [SPARK-44617] Support comparison between lists of Rows
    omit 799ab87ebd23 [SPARK-42941][SS][CONNECT] Python StreamingQueryListener
    omit 5384f4601a4b [SPARK-44559][PYTHON] Improve error messages for Python UDTF arrow cast
    omit adf009d545d7 [SPARK-44610][SQL] DeduplicateRelations should retain Alias metadata when creating a new instance
    omit 1380a3df08e9 [SPARK-44603] Add pyspark.testing to setup.py
    omit 24f9b26cb108 [SPARK-43997][CONNECT] Add support for Java UDFs
    omit 8f44033dfce1 [SPARK-44605][CORE] Refine internal ShuffleWriteProcessor API
    omit 17c206fb71d0 [SPARK-44602][SQL][CONNECT][PS] Make `WidenSetOperationTypes` retain the `Plan_ID_TAG`
    omit ea3061beedf7 [SPARK-44340][SPARK-44341][SQL][PYTHON][FOLLOWUP] Set partition index correctly for WindowGroupLimitExec,WindowExec and WindowInPandasExec
    omit 32498b390db9 [SPARK-44542][CORE] Eagerly load SparkExitCode class in exception handler
    omit af6e1775185b [MINOR][TESTS] Rename ArrowParityTests to JobCancellationTests
    omit b8cb9d552aaa [SPARK-43402][SQL] FileSourceScanExec supports push down data filter with scalar subquery
    omit ff022e54c2ca [DOCS] Update concat and concat_ws documentation to point out unexpected behavior
    omit c93b410c81e9 [MINOR][TESTS] Clearing residual files after SparkSubmitSuite
    omit be95298f2466 [SPARK-44524][BUILD] Balancing pyspark-pandas-connect and pyspark-pandas-slow-connect GA testing time
    omit 55391f633a43 [SPARK-44587][SQL][CONNECT] Increase protobuf marshaller recursion limit
    omit 0f9cca5b419b [SPARK-44287][SQL][FOLLOWUP] Do not trigger execution too early
    omit 9f03f434358d [SPARK-44597][PYTHON][TESTS] Migrate test_sql assert_eq to use assertDataFrameEqual
    omit 5df1d79f2242 [SPARK-43744][CONNECT][FOLLOW-UP] Throw error from the constructor
    omit 994a3f14bc81 [SPARK-44593][INFRA] Make `breaking-changes-buf` cancelable
    omit d232e6e170d1 [SPARK-44590][SQL][CONNECT] Remove the arrow batch record limit for SqlCommandResult
    omit 6cd3ce496393 [SPARK-44584][CONNECT] Set client_type information for AddArtifactsRequest and ArtifactStatusesRequest in Scala Client
    omit a76ee1139dfd [SPARK-44596] Fix pandas-on-Spark type checks for assertDataFrameEqual
    omit 4d3fb62302bb [SPARK-44432][SS][CONNECT] Terminate streaming queries when a session times out in Spark Connect
    omit 37b571d2eade [SPARK-44591][CONNECT][SQL] Add jobTags to SparkListenerSQLExecutionStart
    omit f8786f01ad60 [SPARK-44394][CONNECT][WEBUI] Add a Spark UI page for Spark Connect
    omit 11b3b23737d8 [SPARK-43838][SQL][FOLLOWUP] Add missing aggregate in `renewDuplicatedRelations`
    omit 6d0fed9a18ff [SPARK-43744][CONNECT] Fix class loading problem caused by stub user classes not found on the server classpath
    omit 3761b7d65bd1 [SPARK-44574][SQL][CONNECT] Errors that moved into sq/api should also use AnalysisException
    omit 85a4d1e56e80 [SPARK-41400][CONNECT] Remove Connect Client Catalyst Dependency
    omit 72af2c0fbc66 [SPARK-44585][MLLIB] Fix warning condition in MLLib RankingMetrics ndcgAk
    omit 20bb6c0c5a84 [SPARK-37377][SQL][FOLLOWUP] Fix the partitioned join of one side test case not match
    omit 3dcee0ccf7e0 [SPARK-44538][CONNECT][SQL] Reinstate Row.jsonValue and friends
    omit 7ee9e43ec689 [SPARK-44552][SQL] Remove `private object ParseState` definition from `IntervalUtils`
    omit 7c1ad5bb60c8 [SPARK-44548][PYTHON] Add support for pandas-on-Spark DataFrame assertDataFrameEqual
    omit 3cf88cb6c42e [SPARK-44361][SQL][FOLLOWUP] Use PartitionEvaluator API in MapInBatchExec
    omit f235c9f622dd [SPARK-42098][SQL] Fix ResolveInlineTables can not handle with RuntimeReplaceable expression
    omit bf1bbc514ebf [SPARK-44287][SQL][FOLLOWUP] Set partition index correctly
    omit 9c7f613f0f10 [SPARK-44558][CONNECT][PYTHON] Export Spark Log Level
    omit 9ae33b82d78d [SPARK-44264][PYTHON] E2E Testing for Deepspeed
    omit 5fc90fbd4e32 [SPARK-44198][CORE] Support propagation of the log level to the executors
    omit a3bd477a6d8c [SPARK-44425][CONNECT] Validate that user provided sessionId is an UUID
    omit 5f6537409383 [SPARK-44560][PYTHON][CONNECT] Improve tests and documentation for Arrow Python UDF
    omit 7194ce9263fe [SPARK-43968][PYTHON] Improve error messages for Python UDTFs with wrong number of outputs
    omit 921fb289f003 [MINOR][DOCS] fix: some minor typos
    omit 01191c83f8c7 [SPARK-44505][SQL] Provide override for columnar support in Scan for DSv2
    omit 50b36d131326 [SPARK-44454][SQL][HIVE] HiveShim getTablesByType support fallback
    omit 9b40c0cdc5cf [SPARK-43611][PS][CONNECT][TESTS][FOLLOWUPS] Enable more tests
    omit f7947341ab29 [SPARK-44544][INFRA][FOLLOWUP] Force run `run_python_packaging_tests`
    omit 918d1581fb0f [SPARK-44536][BUILD] Upgrade sbt to 1.9.3
    omit deaa2f7200bb [SPARK-44482][CONNECT] Connect server should can specify the bind address
    omit 77aab6edb7c2 [SPARK-44557][INFRA] Clean up untracked/ignored files before running pip packaging test in GitHub Actions
    omit 8647b243deed [SPARK-44533][PYTHON] Add support for accumulator, broadcast, and Spark files in Python UDTF's analyze
    omit 29ef893eb2d4 [SPARK-44507][SQL][CONNECT] Move AnalysisException to sql/api
    omit 185a0a5a2395 [SPARK-43611][SQL][PS][CONNCECT] Make `ExtractWindowExpressions` retain the `PLAN_ID_TAG`
    omit 02e36dd0f077 [SPARK-44479][PYTHON] Fix ArrowStreamPandasUDFSerializer to accept no-column pandas DataFrame
    omit da359259b138 [SPARK-44457][CONNECT][TESTS] Add `truncatedTo(ChronoUnit.MICROS)` to make `ArrowEncoderSuite` in Java 17 daily test GA task pass
    omit 43b753a3530b [SPARK-44522][BUILD] Upgrade `scala-xml` to 2.2.0
    omit 91e97f92fe76 [SPARK-44528][CONNECT] Support proper usage of hasattr() for Connect dataframe
    omit 6b6216c01ef3 [SPARK-44537][BUILD] Upgrade kubernetes-client to 6.8.0
    omit d0d4aab43784 [SPARK-44154][SQL][FOLLOWUP] `BitmapCount` and `BitmapOrAgg` should use `DataTypeMismatch` to indicate unexpected input data type
    omit 071feabbd432 [SPARK-44531][CONNECT][SQL] Move encoder inference to sql/api
    omit a84e2b1eee3b [SPARK-44525][SQL] Improve error message when Invoke method is not found
    omit 748eaff4e217 [SPARK-44544][INFRA] Deduplicate `run_python_packaging_tests`
    omit ac8fe83af217 [SPARK-44264][PYTHON][DOCS] Added Example to Deepspeed Distributor
    omit 89c3472c5feb [SPARK-44264][PYTHON] Added Deepspeed Folder to setup.py
    omit 6ca45c52b741 [MINOR][SQL] Fix comment in KeyGroupedPartitioning not match with parameter
    omit 03d0ecc68365 [SPARK-44535][CONNECT][SQL] Move required Streaming API to sql/api
    omit fddd0c2c6b8b [SPARK-44509][CONNECT][FOLLOW-UP] Add inheritable support in new job cancellation API
    omit 35187995bf9e [SPARK-44530][CORE][CONNECT] Move SparkBuildInfo to common/util
    omit a8b25f8b5ba2 [SPARK-44503][SQL][FOLLOWUP] Simplify the test case for PARTITION BY and ORDER BY clause after TABLE arguments for TVF calls
    omit 307e46cc4dfd [SPARK-44532][CONNECT][SQL] Move ArrowUtils to sql/api
    omit 4a75a0260e10 [SPARK-44541][SQL] Remove useless function `hasRangeExprAgainstEventTimeCol` from `UnsupportedOperationChecker`
    omit da84f81b1e2e [SPARK-44355][SQL] Move WithCTE into command queries
    omit 304d6ac0564b [SPARK-44540][UI] Remove unused stylesheet and javascript files of jsonFormatter
    omit b7d10505e524 [SPARK-44534][K8S] Handle only shuffle files in KubernetesLocalDiskShuffleExecutorComponents
    omit 8bca01e30ae2 [SPARK-44523][SQL] Filter's maxRows/maxRowsPerPartition is 0 if condition is FalseLiteral
    omit b930f7a1e814 [SPARK-43964][PYTHON][TESTS][FOLLOWUP] Skip a test using pandas when pandas is not available
    omit 7f730f65b65f [SPARK-44466][SQL] Exclude configs starting with `SPARK_DRIVER_PREFIX` and `SPARK_EXECUTOR_PREFIX` from modifiedConfigs
    omit 5ee462d4cb60 [SPARK-44509][PYTHON][CONNECT] Add job cancellation API set in Spark Connect Python client
    omit 2f7a9a1e26f4 [SPARK-44520][SQL] Replace the term UNSUPPORTED_DATA_SOURCE_FOR_DIRECT_QUERY with UNSUPPORTED_DATASOURCE_FOR_DIRECT_QUERY and disclosure root AE
    omit d97a4e214c7e [SPARK-44131][SQL][PYTHON][CONNECT][FOLLOWUP] Support qualified function name for call_function
    omit a6ac63d14b56 [SPARK-44449][CONNECT] Upcasting for direct Arrow Deserialization
    omit 78b3345fa17c [SPARK-44486][PYTHON][CONNECT] Implement PyArrow `self_destruct` feature for `toPandas`
    omit 58e5d86cc076 [SPARK-44503][SQL] Add SQL grammar for PARTITION BY and ORDER BY clause after TABLE arguments for TVF calls
    omit cfc279c6c0fd [SPARK-42944][SS][PYTHON][CONNECT][FOLLOWUP] Streaming ForeachBatch in Python followups
    omit 6992f8b7db8d [SPARK-44455][SQL] Quote identifiers with backticks in SHOW CREATE TABLE result
    omit 4644344f443f [SPARK-44519][CONNECT] SparkConnectServerUtils generated incorrect parameters for jars
    omit 2441bdc49f18 [SPARK-44521][CONNECT][TESTS] Use `Utils.createTempDir` to make the dirs generated by `SparkConnectServiceSuite` is cleaned up after testing
    omit ecfb49c57e5d [SPARK-44513][BUILD] Upgrade snappy-java to 1.1.10.3
    omit 515dfc166a95 [SPARK-44496][SQL][CONNECT] Move Interfaces needed by SCSC to sql/api
    omit 46440a4a5421 [MINOR][UI] Simplify columnDefs in stagepage.js
    omit 49401c7a679c [MINOR][CONNECT] Remove redundant type cast in `ArtifactManager`
    omit 784b942196bb [SPARK-44380][PYTHON][FOLLOWUP] Set __doc__ for analyze static method when Arrow is enabled
    omit 09c44fd05d10 [SPARK-44485][SQL] Optimize TreeNode.generateTreeString
    omit 6409284963d6 [SPARK-44510][UI] Update dataTables to 1.13.5 and remove some unreached png files
    omit bdeae8706745 [SPARK-44341][SQL][PYTHON][FOLLOWUP] Move the base trait WindowEvaluatorFactoryBase to a single file
    omit e6649ebfd44f [SPARK-44506][BUILD] Upgrade mima-core & sbt-mima-plugin to 1.1.3
    omit dda37841189a [SPARK-44422][CONNECT] Spark Connect fine grained interrupt
    omit 679ea56dc8be [SPARK-39634][SQL] Allow file splitting in combination with row index generation
    omit 7606ad32d616 [SPARK-44484][SS] Add batchDuration to StreamingQueryProgress json method
    omit 8c26c014f11b [SPARK-44504][SS] Unload provider thereby forcing DB instance close and releasing resources on maintenance task error
    omit 1fb3e16a48d8 [SPARK-43966][SQL][PYTHON] Support non-deterministic table-valued functions
    omit bd014b398204 [SPARK-42898][SQL] Mark that string/date casts do not need time zone id
    omit f1e310b84d5f [SPARK-44502][DOC][SS][PYTHON] Add missing versionchanged field to streaming functions
    omit 99f9df564ef3 [SPARK-43952][SQL][FOLLOWUP] Correct AQE cancel broadcast job tag
    omit 06db44a3b3d9 [SPARK-44487][TEST] Fix KubernetesSuite report NPE when not set spark.kubernetes.test.unpackSparkDir
    omit 9619ada35059 [SPARK-44477][SQL] Treat TYPE_CHECK_FAILURE_WITH_HINT as an error subclass
    omit 325888bc521e [SPARK-44292][SQL][FOLLOWUP] Make TYPE_CHECK_FAILURE_WITH_HINT use correct name
    omit bde87d66d0d2 [SPARK-44252][SS] Define a new error class and apply for the case where loading state from DFS fails
    omit ca822d5f25d5 [SPARK-44501][K8S] Ignore checksum files in KubernetesLocalDiskShuffleExecutorComponents
    omit 57b985c5d57e [SPARK-44494][INFRA] Use `minikube` v1.30.1 for `k8s-integration-tests`
    omit 6700f3ce8b10 [SPARK-43839][SQL][FOLLOWUP] Convert _LEGACY_ERROR_TEMP_1337 to UNSUPPORTED_FEATURE.TIME_TRAVEL
    omit 8ff6b7a04cba [SPARK-44475][SQL][CONNECT] Relocate DataType and Parser to sql/api
    omit 24bf76a088be [SPARK-44491][INFRA] Add `branch-3.5` to `publish_snapshot` GitHub Action job
    omit 399705512a41 [SPARK-44367][SQL][UI] Show error message on UI for each failed query
    omit 964f7555a7f0 [SPARK-44380][SQL][PYTHON] Support for Python UDTF to analyze in Python
    omit db357edb7b21 [SPARK-44431][SQL] Fix behavior of null IN (empty list) in optimization rules
    omit e0c79c637c16 [SPARK-43838][SQL] Fix subquery on single table with having clause can't be optimized
    omit 11e2e4245df1 [SPARK-44481][CONNECT][PYTHON] Make pyspark.sql.is_remote an API
    omit fa9725fac0c0 [SPARK-44278][CONNECT] Implement a GRPC server interceptor that cleans up thread local properties
    omit 5939b75b5fe7 [SPARK-44396][CONNECT] Direct Arrow Deserialization
    omit 047fad5861da [MINOR][INFRA] Update the labeler for CORE and CONNECT
    omit bca28f87ae12 [SPARK-44361][SQL][FOLLOW-UP] Remove unused variables and fix import statements
    omit 392f8d80c8cc [SPARK-44264][ML][PYTHON] Support Distributed Training of Functions Using Deepspeed
    omit 0879a25c4c27 [SPARK-44272][YARN] Path Inconsistency when Operating statCache within Yarn Client
    omit 9b43a9f3ea55 [SPARK-44361][SQL] Use PartitionEvaluator API in MapInBatchExec
    omit a1eea7f206b9 [SPARK-43965][PYTHON][CONNECT][FOLLOWUP] Include test_parity_udtf in spark test module
    omit fb8c74b8d6e4 [SPARK-44411][SQL] Use PartitionEvaluator API in ArrowEvalPythonExec and  BatchEvalPythonExec
    omit d7128c32d645 [SPARK-44375][SQL] Use PartitionEvaluator API in DebugExec
    omit 8d779272cea9 [MINOR] Move `spark.stage.maxConsecutiveAttempts` to config
    omit cf99e6c00c3c [SPARK-44474][CONNECT] Reenable "Test observe response" at SparkConnectServiceSuite
    omit ee0e687b2bf5 [SPARK-44264][ML][PYTHON] Refactoring TorchDistributor To Allow for Custom "run_training_on_file" Function Pointer
    omit bdea4c5f5283 [SPARK-44401][PYTHON][DOCS] Arrow Python UDF Use Guide
    omit f35c814f9b3c [SPARK-44448][SQL] Fix wrong results bug from DenseRankLimitIterator and InferWindowGroupLimit
    omit ede82a951924 [SPARK-44324][SQL][CONNECT] Move CaseInsensitiveMap to sql/api
    omit 227e28cf84c0 [SPARK-43755][CONNECT][MINOR] Open `AdaptiveSparkPlanHelper.allChildren` instead of using copy in `MetricGenerator`
    omit 98de9a0ad460 [SPARK-44476][CORE][CONNECT] Fix population of artifacts for a JobArtifactState with no associated artifacts
    omit 2ab70576d68d [SPARK-44264][ML][PYTHON] Incorporating FunctionPickler Into TorchDistributor
    omit d93f6d145142 [SPARK-42944][SS][PYTHON] Streaming ForeachBatch in Python
    omit 6cebd97f4670 [SPARK-44464][SS] Implement applyInPandasWithState in PySpark
    omit ea354339f3aa [SPARK-44465][BUILD] Upgrade zstd-jni to 1.5.5-5
    omit 9c52c8295df7 [SPARK-44467][BUILD] Setting version to 4.0.0-SNAPSHOT
    omit 704131b13f88 [SPARK-43203][SQL][FOLLOWUP] V2SessionCatalog.dropTable should handle null table
    omit 32b7b58114d8 [SPARK-43923][CONNECT][FOLLOW-UP][TESTS] Skip "Test observe response" at SparkConnectServiceSuite
    omit 90a60d45e1df [SPARK-44468][BUILD] Add daily test GA task for branch3.5
    omit 4c427d54785a [SPARK-44471][INFRA] Change branches in build_and_test.yml for master branch
    omit 7fe873b9c8bf [SPARK-44451][BUILD] Make built document downloadable
    omit 73c645584c56 [SPARK-43967][PYTHON] Support regular Python UDTFs with empty return values
    omit 6b726d7352b7 [SPARK-44348][CONNECT][FOLLOW-UP] Avoid double slashes in the URI
    omit 95dc829c92ae [SPARK-44341][SQL][PYTHON] Define the computing logic through PartitionEvaluator API and use it in WindowExec and WindowInPandasExec
    omit df550cfc3ad8 [SPARK-44154][SQL] Added more unit tests to BitmapExpressionUtilsSuite and made minor improvements to Bitmap Aggregate Expressions
    omit 228b5dbfd768 [SPARK-44059] Add analyzer support of named arguments for built-in functions
    omit a367fde24de0 [SPARK-40770][PYTHON][FOLLOW-UP] Improved error messages for mapInPandas for schema mismatch
    omit b44e6054dac5 [SPARK-43923][CONNECT] Post listenerBus events during ExecutePlanRequest
    omit d574dbcc8596 [SPARK-44413][PYTHON] Clarify error for unsupported arg data type in assertDataFrameEqual
    omit 4b10e4b8921a Revert "[SPARK-44129][INFRA] Use "3.5.0" for "master" branch until creating `branch-3.5`"
    omit aa688106df02 [SPARK-44453][PYTHON] Use difflib to display errors in assertDataFrameEqual
    omit e578d466d4ea [SPARK-44446][PYTHON] Add checks for expected list type special cases
    omit 85d8d62216d3 [SPARK-44409][SQL] Handle char/varchar in Dataset.to to keep consistent with others
    omit 9a18c066d7e5 [MINOR][DOCS] Fix a typo in dev/merge_spark_pr.py
    omit 61b9236654d8 [SPARK-44439][CONNECT][SS] Fixed listListeners to only send ids back to client
    omit 3ef146cfca35 [SPARK-44429][SQL][TESTS] Make `MsSqlServerIntegrationSuite` robust in ANSI mode
    omit 80ca81eeafdd [SPARK-44430][SQL] Add cause to `AnalysisException` when option is invalid
    omit 001e93d916fb [SPARK-44264][PYTHON][ML][FOLLOW-UP] Clean Up Deepspeed Code
    omit a00a32a2f967 [SPARK-44216][PYTHON] Make assertSchemaEqual API public
    omit ce53fdfa32f9 [SPARK-44410][PYTHON][CONNECT] Set active session in create, not just getOrCreate
    omit d91904e62710 [SPARK-43965][PYTHON][CONNECT] Support Python UDTF in Spark Connect
    omit aec344512978 [SPARK-44441][BUILD] Upgrade `bcprov-jdk15on` and `bcpkix-jdk15on` to 1.70
    omit 5f73b405b251 [SPARK-44438][SS] Shutdown scheduled executor used for maintenance task if an error is reported
    omit e5e2b914de6a [MINOR][SS][DOCS] Fix typos in the Scaladoc and make the semantic of getCurrentWatermarkMs explicit
    omit c63ba6c4f43f [SPARK-42454][SQL] SPJ: encapsulate all SPJ related parameters in BatchScanExec
    omit 0a0c367299bf [SPARK-44415][BUILD] Upgrade snappy-java to 1.1.10.2
    omit 8a2e2f081e8f [SPARK-44021][SQL][FOLLOW-UP] Fix log messages when the number of partition exceeds maxPartitionNum
    omit e000cb868ccb [SPARK-44264][PYTHON][ML] FunctionPickler Class
    omit 18d0a276c501 [SPARK-43755][CONNECT] Move execution out of SparkExecutePlanStreamHandler and to a different thread
    omit 5f8d41b39446 [SPARK-44416][CONNECT][BUILD] Upgrade buf to v1.24.0
    omit 3e0b83f3e200 [SPARK-44418][PYTHON][CONNECT] Upgrade protobuf from 3.19.5 to 3.20.3
    omit ad1f1085b1b6 [SPARK-44382][BUILD] Upgrade protobuf-java to 3.23.4
    omit 1b8e9c6a957d [SPARK-44217][PYTHON] Allow custom precision for fp approx equality
    omit de59caa83af1 [SPARK-44222][BUILD][PYTHON] Upgrade grpc to 1.56.0 with lower/upperbound
    omit a47bde28d9a2 Revert "[SPARK-44222][BUILD][PYTHON] Upgrade `grpc` to 1.56.0"
    omit 7ecdad5c59ce [SPARK-43995][SPARK-43996][CONNECT] Add support for UDFRegistration to the Connect Scala Client
    omit d1d97604fbec [SPARK-44180][SQL] DistributionAndOrderingUtils should apply ResolveTimeZone
    omit fd518dac8d9f [SPARK-44201][CONNECT][SS] Add support for Streaming Listener in Scala for Spark Connect
    omit bac7050cf0ad Revert "[SPARK-41811][PYTHON][CONNECT] Implement SparkSession.sql's string formatter"
    omit a6db459328d7 [MINOR] Removing redundant parentheses from SQL function docs
    omit 61030de02f57 [SPARK-44364] [PYTHON] Add support for List[Row] data type for expected
    omit 9f07e4a747b0 [SPARK-43389][SQL] Added a null check for lineSep option
    omit 63cef16ef779 [SPARK-44395][SQL] Update TVF arguments to require parentheses around identifier after TABLE keyword
    omit d35fda69e49b [SPARK-44279][BUILD] Upgrade `optionator`  to ^0.9.3
    omit 4771853c9bc2 [SPARK-44398][CONNECT] Scala foreachBatch API
    omit 7e0679be8aa1 [SPARK-44338][SQL] Fix view schema mismatch error message
    omit af27400bd1f6 [SPARK-44402][SQL][TESTS] Add tests for schema pruning in delta-based DELETEs
    omit efed39516c0c [SPARK-42309][SQL] Introduce `INCOMPATIBLE_DATA_TO_TABLE` and sub classes
    omit 4dedb4ad2c9b [SPARK-44384][SQL][TESTS] Use checkError() to check Exception in *View*Suite, *Namespace*Suite, *DataSource*Suite
    omit 8bb07388ea66 [SPARK-44145][SQL] Callback when ready for execution
    omit 9aa42a970c4b [SPARK-41811][PYTHON][CONNECT] Implement SparkSession.sql's string formatter
    omit 357bcac823c0 [SPARK-44388][CONNECT] Fix protobuf cast issue when UDF instance is updated
    omit 3e82ac6ea3d9 [SPARK-44391][SQL] Check the number of argument types in `InvokeLike`
    omit fbb85ac1bbba [SPARK-43321][CONNECT][FOLLOWUP] Better names for APIs used in Scala Client joinWith
    omit 27a3dccfa049 [SPARK-44390][CORE][SQL] Rename `SparkSerDerseUtils` to `SparkSerDeUtils`
    omit 3dbcd8339831 [SPARK-43964][SQL][PYTHON] Support arrow-optimized Python UDTFs
    omit 7976f9771389 [SPARK-44348][TESTS][PYTHON][FOLLOW-UP] Reduces the memory used in local-cluster tests
    omit f4aa7989d875 [MINOR][PYTHON] Fix PySpark error class get_error_message
    omit 1bee2f6d701f [SPARK-44387][CONNECT][TESTS] Configure a custom `RetryPolicy` when `Test encryption` initializing `SparkConnectClient`
    omit 432aea3b4027 [MINOR][CONNECT] Fix compilation warning related to `Top-level wildcard is not allowed and will error under -Xsource:3`
    omit 11d8e2eaea30 [SPARK-44393][BUILD] Upgrade `H2` from 2.1.214 to 2.2.220
    omit f2b16275daf8 [SPARK-44399][PYHTON][CONNECT] Import SparkSession in Python UDF only when useArrow is None
    omit e8ba0d41a7f8 [SPARK-44397][PYTHON] Expose assertDataFrameEqual in pyspark.testing.utils
    omit caa3df48d94f [SPARK-44348][CORE][CONNECT][PYTHON] Reenable test_artifact with relevant changes
    omit 947164879b01 [SPARK-42981][CONNECT] Add direct arrow serialization
    omit db160e8a1660 [SPARK-44392][SQL][TESTS] Add tests for schema pruning in delta-based UPDATEs
    omit d9248e83bbb3 [SPARK-43974][CONNECT][BUILD] Upgrade buf to v1.23.1
    omit 8fd6d4d69d56 [SPARK-44349][R] Add math functions to SparkR
    omit 482497c22ac1 [SPARK-44131][SQL][FOLLOWUP] Revert the deprecation message
    omit 6161bf44f40f [SPARK-44353][CONNECT][SQL] Remove StructType.toAttributes
    omit fdffc8399fb1 [SPARK-44373][SQL] Wrap withActive for Dataset API w/ parse logic to make parser related configuration work
    omit 7101fdcee653 [SPARK-44334][SQL][UI] Status in the REST API response for a failed DDL/DML with no jobs should be FAILED rather than COMPLETED
    omit c5c4ec7d9cc1 [SPARK-44370][CONNECT] Migrate Buf remote generation alpha to remote plugins
    omit 6c02bd0198d2 [SPARK-44340][SQL] Define the computing logic through PartitionEvaluator API and use it in WindowGroupLimitExec
    omit daa984430739 [SPARK-43665][CONNECT][PS] Enable PandasSQLStringFormatter.vformat to work with Spark Connect
    omit ce359bcf32bd [SPARK-44325][SQL] Use PartitionEvaluator API in SortMergeJoinExec
    omit 443b49ef089a [SPARK-44377][BUILD] Exclude Junit5 dependencies from `jersey-test-framework-provider-simple`
    omit 054f94af95f3 [SPARK-44374][PYTHON][ML] Add example code for distributed ML for spark connect
    omit 0d90f2a8ea0c [SPARK-44264][ML][PYTHON] Write a Deepspeed Distributed Learning Class DeepspeedTorchDistributor
    omit 37aa62f629e6 [SPARK-44360][SQL] Support schema pruning in delta-based MERGE operations
    omit 0f6a4a737ee9 [SPARK-38476][CORE] Use error class in org.apache.spark.storage
    omit eb0711094439 [SPARK-44263][CONNECT] Custom Interceptors Support
    omit c5a23e9c23f7 [SPARK-44320][SQL] Assign names to the error class _LEGACY_ERROR_TEMP_[1067,1150,1220,1265,1277]
    omit d7bc6f5c7efd [SPARK-43974][CONNECT][BUILD] Revert "[] Upgrade buf to v1.23.1"
    omit c3bfc345191d [SPARK-44357][PYTHON] Add pyspark_testing module for GHA tests
    omit 9accb5c539f6 [SPARK-44363][PYTHON] Display percent of unequal rows in DataFrame comparison
    omit 7a27bc68c849 [SPARK-44251][SQL] Set nullable correctly on coalesced join key in full outer USING join
    omit b0b12cf30283 [SPARK-43974][CONNECT][BUILD] Upgrade buf to v1.23.1
    omit 9717d74d0726 [SPARK-44332][CORE][WEBUI] Fix the sorting error of Executor ID Column on Executors UI Page
    omit 01918bb90170 [SPARK-43983][PYTHON][ML][CONNECT] Implement cross validator estimator
    omit 453300b418bc [SPARK-44352][CONNECT] Put back sameType and friends in DataType
    omit f6866d1c1ab9 [SPARK-44343][CONNECT] Prepare ScalaReflection to the move to SQL/API
    omit f1ec99b10caf [SPARK-44350][BUILD] Upgrade sbt to 1.9.2
    omit 6835b30eb21d [SPARK-44351][SQL] Make some syntactic simplification
    omit c37d7dec8aa4 [SPARK-44271][SQL] Move default values functions from StructType to ResolveDefaultColumns
    omit 46dd3aa94250 [SPARK-44131][SQL] Add call_function and deprecate call_udf for Scala API
    omit b1af89de13f3 [SPARK-44194][DOCS][FOLLOWUP] Add missing `versionadded` annotations for JobTag APIs
    omit 5dbd6ff6aa71 [SPARK-44267][PS][INFRA] Upgrade `pandas` to 2.0.3
    omit 77a4aa057209 [SPARK-44337][PROTOBUF] Any fields set to 'Any.getDefaultInstance' cause parse errors
    omit b7c6c846c085 [SPARK-44328][SQL] Assign names to the error class _LEGACY_ERROR_TEMP_[2325-2328]
    omit 990affdd503f [SPARK-44290][CONNECT][FOLLOW-UP] Skip flaky tests, and fix a typo in session UUID together
    omit 81ef113d068b [SPARK-43710][PS][FOLLOWUP] Fix `date_part` invocations
    omit 4bec3114c3cb [SPARK-44342][SQL] Replace SQLContext with SparkSession for GenTPCDSData
    omit 71703d6d6647 [MINOR][DOCS] clarify array_position return value
    omit ea6bacfcbd5d [SPARK-44329][CONNECT][PYTHON] Add hll_sketch_agg, hll_union_agg, to_varchar, try_aes_decrypt to Scala and Python
    omit 32d170fb026b [SPARK-44331][CONNECT][PYTHON] Add bitmap functions to Scala and Python
    omit 5e31f4dfc20c [SPARK-38477][CORE] Use error class in org.apache.spark.shuffle
    omit fdeb8d8551e3 [SPARK-44321][CONNECT] Decouple ParseException from AnalysisException
    omit 57bbb4c6e730 [SPARK-44290][CONNECT] Session-based files and archives in Spark Connect
    omit e4a356a6d4f2 [SPARK-44061][PYTHON] Add assertDataFrameEqual util function
    omit 56b9f6cd46b1 [SPARK-44287][SQL] Use PartitionEvaluator API in RowToColumnarExec & ColumnarToRowExec SQL operators
    omit 7bfbeb62cb1d [SPARK-44326][SQL][CONNECT] Move utils that are used from Scala client to the common modules
    omit f7d47e83ce52 [SPARK-44333][CONNECT][SQL] Move EnhancedLogicalPlan out of ParserUtils
    omit d73bb9d51c99 [SPARK-44322][CONNECT] Make parser use SqlApiConf instead of SQLConf
    omit 05bc73a83921 [SPARK-44309][UI] Display Add/Remove Time of Executors on Executors Tab
    omit 7e9b027038e3 [SPARK-44327][SQL][CONNECT] Add functions `any` and `len` to Scala
    omit d4277b8e7834 [SPARK-44268][CORE][TEST][FOLLOWUP] Add test to generate `sql-error-conditions` doc automatic
    omit 485e8bc44b4b [SPARK-44312][CONNECT][PYTHON] Allow to set a user agent with an environment variable
    omit e3c2bf565959 [SPARK-44275][CONNECT] Add configurable retry mechanism to Scala Spark Connect
    omit 17fac569b4e4 [SPARK-43660][CONNECT][PS] Enable `resample` with Spark Connect
    omit 259d3e3b589d [SPARK-44315][SQL][CONNECT] Move DefinedByConstructorParams to sql/api
    omit 9c7978d80b8a [SPARK-43321][CONNECT] Dataset#Joinwith
    omit f6e0b3906d53 [SPARK-44316][BUILD] Upgrade Jersey to 2.40
    omit d12bec39cb14 [SPARK-44314][BUILD][CORE][TESTS] Add a new checkstyle rule to prohibit the use of `@Test(expected = ExpectedException.class)`
    omit 7bd135308404 [SPARK-44283][CONNECT] Move Origin to SQL/API
    omit 1adf28669150 [SPARK-44303][SQL] Assign names to the error class _LEGACY_ERROR_TEMP_[2320-2324]
    omit 1fbb94b87c0b [SPARK-44284][CONNECT] Create simple conf system for sql/api
    omit a68e362dca10 Revert "[SPARK-43851][SQL] Support LCA in grouping expressions"
    omit 5d840eb45535 [SPARK-44299][SQL] Assign names to the error class _LEGACY_ERROR_TEMP_227[4-6,8]
    omit f0e18284d0c8 [SPARK-44313][SQL] Fix generated column expression validation when there is a char/varchar column in the schema
    omit 128a991ba2d8 [SPARK-44154][SQL][FOLLOWUP] Add `INVALID_BITMAP_POSITION` error into doc
    omit 29e09473a816 [SPARK-44154] Implement bitmap functions
    omit 1e2522a9ff1c [SPARK-44310][CONNECT] The Connect Server startup log should display the hostname and port
    omit 783880b2ff79 [SPARK-44281][SQL] Move QueryCompilation error that used by DataType to sql/api as DataTypeErrors
    omit d1af925f6e12 [SPARK-44282][CONNECT] Prepare DataType parsing for use in Spark Connect Scala Client
    omit d9c2d38f14a4 [SPARK-44291][SPARK-43416][CONNECT] Fix incorrect schema for range query
    omit 9f87e19a5f18 [SPARK-42554][CONNECT] Implement GRPC exceptions interception for conversion
    omit 4b77aad824d6 [SPARK-44294][UI] Fix HeapHistogram column shows unexpectedly w/ select-all-box
    omit 5b81de9d1c3f [SPARK-44259][CONNECT][TESTS][FOLLOWUP] No longer initializing `Ammonite` for Java 21
    omit f30ddff123d8 [SPARK-44277][BUILD] Upgrade to Avro 1.11.2
    omit 531ec8bddc8d [SPARK-44300][CONNECT][BUG-FIX] Fix artifact cleanup to limit deletion scope to session specific artifacts
    omit 756b4ea7d6a5 [SPARK-44298][BUILD] Disable PySpark test on the daily test of Java 21 before the new arrow version release
    omit 82e091ef2707 Revert "[SPARK-43476][PYTHON][TESTS] Enable SeriesStringTests.test_string_replace for pandas 2.0.0"
    omit e88f5ec54bf1 [SPARK-44245][PYTHON] pyspark.sql.dataframe doctests behave differently
    omit 3e72806bb421 [SPARK-44215][SHUFFLE] If num chunks are 0, then server should throw a RuntimeException
    omit c08700f05f96 [SPARK-44293][CONNECT] Fix invalid URI for custom JARs in Spark Connect
    omit d53585c91b2e [SPARK-44292][SQL] Assign names to the error class _LEGACY_ERROR_TEMP_[2315-2319]
    omit 68862589a0cc [SPARK-44296][BUILD] Upgrade dropwizard metrics 4.2.19
    omit 16295a338220 [SPARK-44297][CORE][TESTS] Make `ClassLoaderIsolationSuite` test pass with Scala 2.13
    omit 0e8a95ea99f3 [SPARK-44295][BUILD] Upgrade `scala-parser-combinators` to 2.3.0
    omit 7fcabef28743 [SPARK-44250][ML][PYTHON][CONNECT] Implement classification evaluator
    omit 7bc28d54f832 [SPARK-44269][SQL] Assign names to the error class _LEGACY_ERROR_TEMP_[2310-2314]
    omit b573cca90ea8 [SPARK-44288][SS] Set the column family options before passing to DBOptions in RocksDB state store provider
    omit 4fcecfe17f2d [SPARK-44194][PYTHON][CORE] Add JobTag APIs to PySpark SparkContext
    omit 4c2ee76d2afa [SPARK-44274][CONNECT] Move out util functions used by ArtifactManager to common/utils
    omit 4cdb6d487ed1 [SPARK-44266][SQL] Move Util.truncatedString to sql/api
    omit 9cb557d12ab3 [SPARK-44150][PYTHON][CONNECT] Explicit Arrow casting for mismatched return type in Arrow Python UDF
    omit d9ce141c5018 [SPARK-44199] CacheManager refreshes the fileIndex unnecessarily
    omit f7e8da3fb6db [SPARK-44200][SQL][FOLLOWUP] Add `TABLE_VALUED_FUNCTION_TOO_MANY_TABLE_ARGUMENTS` error into doc
    omit 8390b03df62e [SPARK-44200][SQL] Support TABLE argument parser rule for TableValuedFunction
    omit 45ae9c5cc67d [SPARK-42169][SQL] Implement code generation for to_csv function (StructsToCsv)
    omit 356eada314e8 [SPARK-42828][PYTHON][SQL] More explicit Python type annotations for GroupedData
    omit 442fdb8be427 [SPARK-43476][PYTHON][TESTS] Enable SeriesStringTests.test_string_replace for pandas 2.0.0
    omit 44ffe9089bce [MINOR][SQL][TESTS] Use SystemUtils.isJavaVersionAtMost for java version check
    omit 8483191dfbb2 [SPARK-44185][SQL] Fix inconsistent path qualifying between catalog and data operations
    omit 4569bf63659f [SPARK-40731][DSTREAM] Make `streaming` pass on Java 21
    omit 8daa2553c92b [SPARK-44195][R] Add JobTag APIs to SparkR SparkContext
    omit e3970cc31af3 [SPARK-41822][CONNECT][TESTS][FOLLOW-UP] Remove the need of a fixed port to allow parallel running of tests
    omit ad3059f23c40 [SPARK-44246][CONNECT][FOLLOW-UP] Miscellaneous cleanups for Spark Connect Jar/Classfile Isolation
    omit 1fbb38ec5bd4 [SPARK-44249][SQL][PYTHON] Refactor PythonUDTFRunner to send its return type separately
    omit 2cb1d3bf8780 [SPARK-44211][PYTHON][CONNECT] Implement SparkSession.is_stopped
    omit b557f5752afc [SPARK-44268][CORE][TEST] Add tests to ensure error-classes.json and docs are in sync
    omit cf852b284d55 [SPARK-44254][SQL] Move QueryExecutionErrors that used by DataType to sql/api as DataTypeErrors
    omit f985e3e84a23 [SPARK-44255][SQL] Relocate StorageLevel to common/utils
    omit 3baf7f7b7106 [SPARK-44244][SQL] Assign names to the error class _LEGACY_ERROR_TEMP_[2305-2309]
    omit f406b54b2a89 [SPARK-44044][SS] Improve Error message for Window functions with streaming
    omit 9353d67f9290 [SPARK-43851][SQL] Support LCA in grouping expressions
    omit 2db8cfb3bd9b [SPARK-44060][SQL] Code-gen for build side outer shuffled hash join
    omit 35d51571a803 [SPARK-42784] should still create subDir when the number of subDir in merge dir is less than conf
    omit 482e852b8534 [MINOR] Fix Typo in `build/mvn` script
    omit 8c635a0fa558 [SPARK-44259][CONNECT][TESTS] Make `connect-client-jvm` pass on Java 21 except `RemoteSparkSession`-based tests
    omit 6d8d11be6a45 [SPARK-44256][BUILD] Upgrade rocksdbjni to 8.3.2
    omit 706829d97312 [SPARK-41487][SQL] Assign name to _LEGACY_ERROR_TEMP_1020
    omit ab67f4619873 [SPARK-43986][SQL] Create error classes for HyperLogLog function call failures
    omit 3fb9a2c6135d [SPARK-44260][SQL] Assign names to the error class _LEGACY_ERROR_TEMP_[1215-1245-2329] & Use checkError() to check Exception in *CharVarchar*Suite
    omit 05f5dccbd342 [SPARK-44257][BUILD] Update some maven plugins & scalafmt to newest version
    omit 7971e1c6a7c0 [SPARK-41599] Memory leak in FileSystem.CACHE when submitting apps to secure cluster using InProcessLauncher
    omit 38645fa470b5 [SPARK-44241][CORE] Mistakenly set io.connectionTimeout/connectionCreationTimeout to zero or negative will cause incessant executor cons/destructions
    omit 91c458125203 [SPARK-43922][SQL] Add named parameter support in parser for function calls
    omit 618b52097c07 [SPARK-44030][SQL][FOLLOW-UP] Move unapply from AnyTimestampType to AnyTimestampTypeExpression
    omit de8ec74f2826 [SPARK-43926][CONNECT][PYTHON] Add array_agg, array_size, cardinality, count_min_sketch,mask,named_struct,json_* to Scala and Python
    omit e505244460ba [SPARK-44150][PYTHON][FOLLOW-UP] Revert commits
    omit 7142e04639c1 [SPARK-44248][SS][SQL][KAFKA] Add preferred location in kafka source v2
    omit 414bc75ac5b9 [SPARK-44150][PYTHON][FOLLOW-UP] Fix ArrowStreamPandasSerializer to set arguments properly
    omit 6e56cfeaca88 [SPARK-44150][PYTHON][CONNECT] Explicit Arrow casting for mismatched return type in Arrow Python UDF
    omit c9734008401c [SPARK-43474][SS][CONNECT] Add a spark connect access to runtime Dataframes by ID
    omit 0ae86c0492cd [SPARK-44227][SQL] Extract SchemaUtils from StructField
    omit a9129defc0eb [SPARK-44208][CORE][SQL] Assign clear error class names for some logic that directly uses exceptions
    omit 6511a3e90206 [MINOR][TESTS] Fix potential bug for AlterTableTest
    omit ffbd1a3b5b17 [SPARK-44169][SQL] Assign names to the error class _LEGACY_ERROR_TEMP_[2300-2304]
    omit 70f34278cbfa [SPARK-44079][SQL] Fix `ArrayIndexOutOfBoundsException` when parse array as struct using PERMISSIVE mode with corrupt record
    omit af5364595015 [SPARK-44237][CORE] Simplify DirectByteBuffer constructor lookup logic
    omit e5a5921968c8 [SPARK-44220][SQL] Move StringConcat to sql/api
    omit a1592fa7ff26 [SPARK-43265][CORE][FOLLOW-UP] Move Error framework to a common utils module
    omit 1a5b9f295a62 [SPARK-44231][BUILD] Update ORC to 1.9.0
    omit cd6aa5819f3e [SPARK-44230][SQL][TESTS] Make `sql` module passes in Java 21
    omit 97a94b744960 [SPARK-44205][SQL] Extract Catalyst Code from DecimalType
    omit d2054fc358d4 [SPARK-44128][BUILD] Upgrade netty to 4.1.93
    omit 1fcd537a37b2 [SPARK-43757][CONNECT] Change client compatibility from allow list to deny list
    omit d14a6ecd9e15 [SPARK-40850][SQL] Fix test case interpreted queries may execute Codegen
    omit f26bdb7bfde5 [SPARK-44222][BUILD][PYTHON] Upgrade `grpc` to 1.56.0
    omit 1c8c47cb55da [SPARK-43914][SQL] Assign names to the error class _LEGACY_ERROR_TEMP_[2433-2437]
    omit 3cd486070bf4 [SPARK-44221][BUILD] Upgrade RoaringBitmap from 0.9.44 to 0.9.45
    omit f00de6f77a80 [SPARK-44182][DOCS] Use Spark version variables in Python and Spark Connect installation docs
    omit 1a1ae1f8cf6c [SPARK-44206][SQL] DataSet.selectExpr scope Session.active
    omit c8c51d047411 [SPARK-44039][CONNECT][TESTS] Improve for PlanGenerationTestSuite & ProtoToParsedPlanTestSuite
    omit 05fc3497f00a [SPARK-44161][CONNECT] Handle Row input for UDFs
    omit 20a8fc87d67c [SPARK-43631][CONNECT][PS] Enable Series.interpolate with Spark Connect
    omit 954987f19dca [SPARK-43979][SQL][FOLLOW-UP] CollectedMetrics should be treated as the same one for self-join
    omit be8b07a15348 [SPARK-44171][SQL] Assign names to the error class _LEGACY_ERROR_TEMP_[2279-2282] & delete some unused error classes
    omit 24960d84a9da [SPARK-44197][BUILD][FOLLOWUP] Update `IsolatedClientLoader` hadoop version
    omit a758d6a0f9df [SPARK-43081][FOLLOW-UP][ML][CONNECT] Make torch dataloader support torch 1.x
    omit 43f7a86a05ad [SPARK-44204][SQL][HIVE] Add missing recordHiveCall for getPartitionNames
    omit 8229feab9795 [SPARK-44192][BUILD][R] Support R 4.3.1
    omit 201d2b1e1278 [SPARK-44202][CORE] Add JobTag APIs to JavaSparkContext
    omit a9199c3d815b [SPARK-43136][CONNECT][FOLLOWUP] Adding tests for KeyAs
    omit 686e37e640d0 [SPARK-44030][SQL] Implement DataTypeExpression to offer Unapply for expression
    omit b02ea4cd370c [SPARK-44146][CONNECT] Isolate Spark Connect Session jars and classfiles
    omit 9f3011a73eff [SPARK-43952][FOLLOWUP] Don't set JobGroup in Spark Connect after all
    omit 9e41de2b1f23 [SPARK-43624][FOLLOWUP][PS][CONNECT] Moving `ewm` into Spark functions
    omit cf3820bf48f0 [SPARK-44197][BUILD] Upgrade Hadoop to 3.3.6
    omit cb9f6bf74b47 [SPARK-44183][PYTHON] Increate `PyArrow` minimum version to 4.0.0
    omit c47f09f23e3d [SPARK-43470][CORE] Add OS, Java, Python version information to application log
    omit 680ca2e56f2c [SPARK-44026] Allow providing initial value for SQLMetrics
    omit e96687cc8470 [SPARK-44188][CORE] Remove useless `resetAllPartitions` method in `ActiveJob`
    omit e98987220ae1 [SPARK-44189][CONNECT][PYTHON] Support positional parameters by `sql()`
    omit fd84fd3c29a7 [SPARK-44187][INFRA][BUILD] Enable benchmark on github to support Java 21
    omit e5cc340788a0 [SPARK-44186][BUILD] Upgrade sbt to 1.9.1
    omit 6a0ae19ee509 [SPARK-43924][CONNECT][PYTHON] Add misc functions to Scala and Python
    omit 2994560d204c [SPARK-44164][SQL] Extract toAttribute method from StructField to Util class
    omit 67abc4301405 [SPARK-44143][SQL][TESTS] Use checkError() to check Exception in *DDL*Suite
    omit 37c898c63b1f [MINOR][CONNECT][TESTS] Check named parameters in `sql()`
    omit 646388ee5f3f [SPARK-44153][CORE][UI][FOLLOWUP] Use JAVA_HOME and stderr
    omit 00e7c08606d0 [SPARK-44184][PYTHON][DOCS] Remove a wrong doc about `ARROW_PRE_0_15_IPC_FORMAT`
    omit 703e819510a7 [SPARK-44064][CORE][SQL] Add a new `apply` function to `NonFateSharingCache`
    omit 2407183cb863 [SPARK-44181][BUILD] Upgrade `Apache Kafka` to 3.4.1
    omit a903f8e9dbd6 [SPARK-44178][CONNECT] Support positional parameters in `sql()`
    omit b482d693c6de [SPARK-44174][SQL][TESTS] Fix `QueryExecutionErrorsSuite` for Java 21
    omit 0ab6a0e17b27 [SPARK-43969][SQL][TESTS][FOLLOWUP] Update `numeric.sql.out.java21`
    omit a8ea35f7c2fb [SPARK-39740][UI] Upgrade vis timeline to 7.7.2 to fix CVE-2020-28487
    omit 6590e7db5212 [SPARK-44158][K8S] Remove unused `spark.kubernetes.executor.lostCheckmaxAttempts`
    omit 0b9c76c9eb2b [SPARK-44151][BUILD] Upgrade `commons-codec` to 1.16.0
    omit cd69d4dd18cf [SPARK-44153][CORE][UI] Support `Heap Histogram` column in `Executors` tab
    omit 8eda2d8472c4 [SPARK-44135][PYTHON][CONNECT][DOCS] Document Spark Connect only API in PySpark
    omit 7398e93a2be2 [SPARK-44139][SQL] Discard completely pushed down filters in group-based MERGE operations
    omit cf6e90c205bd [SPARK-44134][CORE] Fix setting resources (GPU/FPGA) to 0 when they are set in spark-defaults.conf
    omit 3165407f85c4 [SPARK-43990][BUILD] Upgrade `kubernetes-client` to 6.7.2
    omit dcaa10a520ff [SPARK-44110][BUILD] Propagate proxy settings to forked JVMs
    omit 5feab1072ba0 [SPARK-44142][PYTHON] Replace type with tpe in utility to convert python types to spark types
    omit 744ba23027ed [SPARK-43876][SQL][FOLLOWUP] Add a unit test for fast hashmap for distinct queries
    omit 87bc64b23571 [SPARK-44138][SQL] Prohibit non-deterministic expressions, subqueries and aggregates in MERGE conditions
    omit 99337926496a [SPARK-44147][DOCS] Add `spark.ui.allowFramingFrom` doc to `Security` page
    omit 532a8325f5a3 [SPARK-44140][SQL][PYTHON] Support positional parameters in Python `sql()`
    omit 1b4048bf62dd [SPARK-44066][SQL] Support positional parameters in Scala/Java `sql()`
    omit 9d031ba8c995 [SPARK-44078][CONNECT][CORE] Add support for classloader/resource isolation
    omit fb4d6d9db27d [SPARK-44136][SS] Fixed an issue that StateManager may get materialized in executor instead of driver in FlatMapGroupsWithStateExec
    omit bc20e85b0e1d [SPARK-43710][PS][CONNECT] Support `functions.date_part` for Spark Connect
    omit 10751dc285c5 [SPARK-44133][PYTHON] Upgrade MyPy from 0.920 to 0.982
    omit bbcc438e5b3a [SPARK-43915][SQL] Assign names to the error class _LEGACY_ERROR_TEMP_[2438-2445]
    omit a4fb7cceb441 [SPARK-43205][SQL][FOLLOWUP] remove unnecessary abstraction for `withIdentClause`
    omit 6165f3160634 [SPARK-44056][SQL] Include UDF name in UDF execution failure error message when available
    omit 9b1e124b0bd0 [SPARK-43742][SQL][FOLLOWUP] Do not use null literal as default value for non-nullable columns
    omit 6a70f756dd11 [SPARK-44109][CORE] Remove duplicate preferred locations of each RDD partition
    omit ac1e22231055 [SPARK-43903][PYTHON][CONNECT] Improve ArrayType input support in Arrow Python UDF
    omit 94031ead7868 [SPARK-44004][SQL] Assign name & improve error message for frequent LEGACY errors
    omit 5facaece4dfa [SPARK-44106][PYTHON][CONNECT] Add `__repr__` for `GroupedData`
    omit 8a16aed9a172 [SPARK-43511][CONNECT][SS] Implemented MapGroupsWithState and FlatMapGroupsWithState APIs for Spark Connect
    omit 692ec99bc788 [SPARK-44129][INFRA] Use "3.5.0" for "master" branch until creating `branch-3.5`
    omit 5a55061df25a Revert "[SPARK-44129][INFRA] Use "3.5.0" for "master" branch until creating `branch-3.5`"
    omit 3d52ba49946c [SPARK-44107][CONNECT][PYTHON] Hide unsupported Column methods from auto-completion
    omit 95f071cf5f34 [SPARK-44125][R] Support Java 21 in SparkR
    omit 526831b8f072 [SPARK-44103][INFRA] Make pending container jobs cancelable
    omit 6cc63cbccfca [SPARK-44129][INFRA] Use "3.5.0" for "master" branch until creating `branch-3.5`
    omit 6bfc01188e96 [SPARK-42941][SS][CONNECT] 1/2] StreamingQueryListener - Event Serde in JSON format
    omit cafbea5b1362 [SPARK-44122][CONNECT][TESTS] Make `connect` module pass except Arrow-related ones in Java 21
    omit 607469b2fd2e [SPARK-43952][CORE][CONNECT][SQL] Add SparkContext APIs for query cancellation by tag
    omit ef8f22e53d31 [SPARK-44012][SS] KafkaDataConsumer to print some read status
    omit 66f25e314032 [SPARK-43876][SQL] Enable fast hashmap for distinct queries
    omit 68b30053f786 [SPARK-43939][CONNECT][PYTHON] Add try_* functions to Scala and Python
    omit 27e1e56d54b1 [SPARK-44022][BUILD] Enforce max bytecode version on Maven dependencies
    omit 4c15fa784dc2 [SPARK-44105][SQL] `LastNonNull` should be lazily resolved
    omit bfad72b0a868 [SPARK-44104][BUILD] Enabled `protobuf` module mima check for Spark 3.5.0
    omit 8dc02863b926 [SPARK-43929][SPARK-44073][SQL][PYTHON][CONNECT][FOLLOWUP] Add extract, date_part, datepart to Scala, Python and Connect API
    omit 35b3a18ff043 [SPARK-44100][ML][CONNECT][PYTHON] Move namespace from `pyspark.mlv2` to `pyspark.ml.connect`
    omit 0865c0db923e [SPARK-43944][SPARK-43942][SQL][FOLLOWUP] Directly leverage `UnresolvedFunction` for functions `startswith`/`endswith`/`contains`
    omit 6b36a9368d6e [SPARK-43944][SQL][CONNECT][PYTHON][FOLLOW-UP] Make `startswith` & `endswith` support binary type
    omit 7a8f2ec53fde [SPARK-44073][SQL][PYTHON][CONNECT] Add date time functions to Scala, Python and Connect - part 2
    omit 0df4266ff615 [SPARK-44074][CORE][SQL][TESTS] Fix loglevel restore behavior of `SparkFunSuite#withLogAppender` and re-enable UT `Logging plan changes for execution`
    omit d6380ad4d02c [SPARK-43942][SQL][CONNECT][PYTHON][FOLLOW-UP] Make contains support binary type
    omit 469197279dc2 [SPARK-44099][INFRA] Support both java-8-openjdk-amd64 and java-8-openjdk-arm64 in `spark-rm` Dockerfile
    omit 4dc0e98367a1 [SPARK-44036][FOLLOWUP][CONNECT][TESTS] Consolidate remaining tickets
    omit b4f4c3723175 [SPARK-43624][PS][CONNECT] Add `EWM` to SparkConnectPlanner
    omit f3db20c17dfd [SPARK-43969][SQL] Refactor & Assign names to the error class _LEGACY_ERROR_TEMP_1170
    omit 74185cf34a20 [SPARK-44054][CORE][TESTS] Make test cases inherit `SparkFunSuite` have a default timeout
    omit 1008d64461ef [SPARK-40732][SQL][TESTS] Make `(SQL|ThriftServer)QueryTestSuite` pass except Pandas UDF tests in Java 21
    omit 6c0c226d9019 [SPARK-43982][ML][PYTHON][CONNECT] Implement pipeline estimator for ML on spark connect
    omit 32a5db49e3e6 [SPARK-43203][SQL] Move all Drop Table case to DataSource V2
    omit 476c58ed26a1 [SPARK-43942][CONNECT][PYTHON] Add string functions to Scala and Python - part 1
    omit 0fc7eeb39aad [SPARK-44096][PYTHOM][DOCS] Make examples copy-pastable by adding a newline in all modules
    omit f21303fb17f9 [SPARK-40497][BUILD] Upgrade Scala to 2.13.11
    omit 8c84d2c9349d [SPARK-44018][SQL] Improve the hashCode and toString for some DS V2 Expression
    omit 8ccacb84306d [MINOR][PYTHON][DOCS] Make mapInPandas example copy-pastable
    omit 25a14c313bfd [SPARK-43929][SQL][PYTHON][CONNECT] Add date time functions to Scala, Python and Connect API - part 1
    omit 3a9185964a0d [SPARK-44092][CORE] Add `Utils.isJavaVersionAtLeast21` and make `core` module pass with Java 21
    omit d1c53208b67b [SPARK-44095][SQL][TESTS] Make HiveExternalCatalogVersionsSuite skip old Spark versions on Java 21
    omit 7dde27e387eb [MINOR][PYTHON][PS][TESTS] Rename `k_res` to `ps_res` (drop Koalas reference)
    omit 53dae3d0440f [SPARK-43009][PYTHON][FOLLOWUP] Parameterized `sql_formatter.sql()` with Any constants
    omit 1379680be8d5 [SPARK-44034][TESTS] Add a new test group for sql module
    omit b676a2d9a67c [SPARK-43979][SQL][FOLLOW-UP] Simplify metrics plan should replace nodes by new attributes
    omit 9054e46b12cb [SPARK-44094][BUILD] Upgrade Apache Arrow to 12.0.1
    omit cbfc920c2d75 [SPARK-44093][SQL][TESTS] Make `catalyst` module passes in Java 21
    omit d05091eb8f0f [SPARK-44089][SQL][TESTS] Remove the `@ignore` identifier from `AlterTableRenamePartitionSuite`
    omit ad6cd60ca740 [SPARK-44024][SQL] Change to use `map` when `unzip` only used to extract a single element
    omit 4503efcaadd8 [SPARK-43944][CONNECT][PYTHON] Add string functions to Scala and Python - part 2
    omit c27ebc99278d [SPARK-44085][BUILD] Update some maven plugins to newest version
    omit d6c8b0bd41db [SPARK-44088][BUILD] Remove `jdk.incubator.foreign` usage in Java 21
    omit 8dbe53c41f7a [SPARK-43835][INFRA] Add `build_java21.yml` Github Action Daily Job
    omit ffba5972f694 [SPARK-44086][SQL][TESTS] Move `DataFrameFunctionsSuite` to `sql - slow`
    omit e38c586b92c4 [SPARK-43832][BUILD] Upgrade Scala to 2.12.18
    omit 4e0bd3c57175 [SPARK-43097][FOLLOW-UP][ML] Improve logistic regression model saving
    omit 61a195d33936 [SPARK-43928][SQL][PYTHON][CONNECT][FOLLOWUP] Update `excludedSqlFunctions` of `DataFrameFunctionsSuite`
    omit 1ff670488c3b [MINOR][K8S][DOCS] Fix all dead links for K8s doc
    omit b97ce8b9a99c [SPARK-44052][CONNECT][PS] Add util to get proper Column or DataFrame class for Spark Connect
    omit 07ac4e91b30e [MINOR][DOCS] Remove double quote to fix link
    omit 02beb50ed7cd [SPARK-43928][SQL][PYTHON][CONNECT] Add bit operations to Scala, Python and Connect API
    omit 747953eb5c46 [SPARK-44071] Define and use Unresolved[Leaf|Unary]Node traits
    omit 8754e9a6e839 [SPARK-44081] Simplify PartitionedFileUtil API a little
    omit d73eafcf5806 [SPARK-42618][PYTHON][PS] Warning for the pandas-related behavior changes in next major release
    omit aff416fba532 [SPARK-44041][BUILD] Upgrade Ammonite to 2.5.9
    omit 0502a42dda4d [SPARK-44070][BUILD] Bump snappy-java 1.1.10.1
    omit 01d068b06d5e [SPARK-43474][SS][CONNECT] Add SessionHolder to SparkConnectPlanner
    omit fb1ee25a89e8 [SPARK-43290][SQL] Adds support for aes_encrypt IVs and AAD
    omit 8212d6f87c65 [SPARK-44075][CONNECT] Make `transformStatCorr` lazy
    omit 1b12094d2d18 [SPARK-43925][SQL][PYTHON][CONNECT] Add some, bool_or,bool_and,every to Scala, Python and Connect
    omit 55ba63c257b6 [SPARK-44040][SQL] Fix compute stats when AggregateExec node above QueryStageExec
    omit 34f2c39bf753 [SPARK-44072][DOCS] Update insert table docs for incorrect sql example
    omit 06b2b1a8c30f [SPARK-43937][CONNECT][PYTHON] Add ifnull,isnotnull,equal_null,nullif,nvl,nvl2 to Scala and Python
    omit 49dc2dbcbb1d [SPARK-42750][SQL][FOLLOWUP] Add INSERT OVERWRITE BY NAME statement
    omit fe4417c082d2 [SPARK-44055][CORE] Remove redundant `override` functions from `CheckpointRDD`
    omit fe787efca78d [SPARK-43941][PYTHON][TESTS][FOLLOWUP] Doctest clean up
    omit f44d9238e404 [SPARK-43627][SPARK-43626][PS][CONNECT] Enable `pyspark.pandas.spark.functions.{kurt, skew}` in Spark Connect
    omit 79100ce225d8 [SPARK-43941][SQL][PYTHON][CONNECT] Add any_value, approx_percentile, count_if, first_value, histogram_numeric, last_value, reduce to Scala, Python and Connect API
    omit 1491e7f628d5 [SPARK-43659][CONNECT][PS][TESTS] Enable `OpsOnDiffFramesEnabledSlowParityTests.test_series_eq`
    omit 209722817e27 [SPARK-43975][SQL] DataSource V2: Handle UPDATE commands for group-based sources
    omit 0be4b9799dad [SPARK-44057][SQL][TESTS] Mark all `local-cluster`-based tests as `ExtendedSQLTest`
    omit 05869bb2397d [SPARK-39979][SQL][FOLLOW-UP] Make `spark.sql.execution.arrow.useLargeVarTypes` as an internal configuration
    omit fff0b71d24bf [SPARK-44053][BUILD] Update ORC to 1.8.4
    omit b4766ff65c20 [SPARK-44047][CONNECT][BUILD] Upgrade google guava for connect from 31.0.1-jre to 32.0.1-jre
    omit 55d7f91cf80a [SPARK-44051][TESTS][PS][CONNECT] Split `pyspark.pandas.tests.connect.data_type_ops.test_parity_num_ops`
    omit a6567fefbd4b [SPARK-43931][SQL][PYTHON][CONNECT] Add make_* functions to Scala and Python
    omit ac7d7d835b0c [SPARK-43645][SPARK-43622][PS][CONNECT] Enable `pyspark.pandas.spark.functions.{var, stddev}` in Spark Connect
    omit d355c29ae309 [SPARK-44035][PS][CONNECT][TESTS] Split `pyspark.pandas.tests.connect.test_parity_ops_on_diff_frames_slow`
    omit c933dc90f847 [SPARK-44048][DOCS] Remove sql-migration-old.md
    omit 305506eceeba [SPARK-43963][SQL] DataSource V2: Handle MERGE commands for group-based sources
    omit 3adbce2734b3 [SPARK-44049][K8S][TESTS] Fix KubernetesSuite to use `inNamespace` for validating driver pod cleanup
    omit d8c9acfbf3dc [SPARK-43932][SQL][PYTHON][CONNECT] Add `current` like functions to Scala and Python
    omit a5d3bea04eb8 [SPARK-43981][PYTHON][ML] Basic saving / loading implementation for ML on spark connect
    omit 75291d60d732 [SPARK-43663][CONNECT][PS] Enable `SeriesParityTests.test_compare`
    omit b174aaa42e6d [SPARK-44021][SQL] Add spark.sql.files.maxPartitionNum
    omit 0086f5427410 [SPARK-43934][SQL][PYTHON][CONNECT] Add regexp_* functions to Scala and Python
    omit 9c8592058cd1 [SPARK-43684][SPARK-43685][SPARK-43686][SPARK-43691][CONNECT][PS] Fix `(NullOps|NumOps).(eq|ne)` for Spark Connect
    omit e8aa23a7056c [SPARK-44045][SQL][TESTS] Mark `WholeStageCodegenSparkSubmitSuite` as `ExtendedSQLTest`
    omit 7e94f2a54337 [SPARK-43529][SQL][FOLLOWUP] Code cleanup in UnresolvedTableSpec and related plans
    omit dfd40a409464 [SPARK-44016][CONNECT] Prevent processing of artifacts that contain invalid paths (absolute, parent, sibling, nephew)
    omit a02275ac16c8 [SPARK-43380][SQL] Fix conversion of Avro logical timestamp type to Long
    omit 3d7b6067e959 [SPARK-44028][BUILD] Upgrade Apache `commons-io` to 2.13.0
    omit 2d250085f0a2 [SPARK-43888][CONNECT][FOLLOW-UP] Spark Connect client should depend on common-utils explicitly
    omit bed53f855724 [SPARK-44036][CONNECT][PS] Cleanup & consolidate tickets to simplify the tasks
    omit 223c196d242f [SPARK-44038][DOCS][K8S] Update YuniKorn docs with v1.3
    omit 70186f58a14b [SPARK-43798][PYTHON][FOLLOW-UP] Sets the batch size as 1 to send individual group
    omit daee47a43647 [SPARK-43885][SQL] DataSource V2: Handle MERGE commands for delta-based sources
    omit 04d84df87e33 [SPARK-44031][BUILD] Upgrade silencer to 1.7.13
    omit 80588e4ddfac [SPARK-32559][SQL] Fix the trim logic did't handle ASCII control characters correctly
    omit 37ab190dc5bf [SPARK-43798][SQL][PYTHON] Support Python user-defined table functions
    omit 933dfd9cd7ee [SPARK-44023][SQL][TESTS] Add `System.gc` at `beforeEach` in `PruneFileSourcePartitionsSuite`
    omit c41be4ec0ad9 [SPARK-42298][SQL] Assign name to _LEGACY_ERROR_TEMP_2132
    omit 7107742a381c [SPARK-43398][CORE] Executor timeout should be max of idle shuffle and rdd timeout
    omit bf75dd6e8ef4 [SPARK-43617][PS][CONNECT] Enable `pyspark.pandas.spark.functions.product` in Spark Connect
    omit 7250941ab5b8 [SPARK-43529][SQL] Support general constant expressions as CREATE/REPLACE TABLE OPTIONS values
    omit 4da9348160f5 [SPARK-43938][CONNECT][PYTHON] Add to_* functions to Scala and Python
    omit 45ad044042f7 [SPARK-43179][FOLLOW-UP] Use the secret ByteBuffer instead of the String
    omit 4f2e2d7e0383 [SPARK-43772][BUILD][CONNECT] Move version configuration in `connect` module to parent
    omit 616c492aa1f4 [SPARK-43884] Param markers in DDL
    omit 0bdafb362da0 [SPARK-44017][PS][CONNECT][TESTS] Reorganize `ps.Groupby` unit tests
    omit 3ec27a94af8e [SPARK-44020][ML][PYTHON][TESTS] Split `pyspark.ml.tests.test_tuning`
    omit f74646e23da7 [SPARK-43621][PS][CONNECT] Enable `pyspark.pandas.spark.functions.repeat` in Spark Connect
    omit 5ea8ce4b5374 [SPARK-43594][SQL] Add LocalDateTime to anyToMicros
    omit c30b411e412a [SPARK-43916][CONNECT][TESTS] Adds the test case that percentile with frequency for Scala API
    omit 0dc2bd45eeed [SPARK-43616][PS][CONNECT] Enable `pyspark.pandas.spark.functions.mode` in Spark Connect
    omit 611908afb4fd [SPARK-43610][CONNECT][PS] Enable `InternalFrame.attach_distributed_column` in Spark Connect
    omit 93e0acbf7d9f [SPARK-43971][CONNECT][PYTHON] Support Python's createDataFrame in streaming manner
    omit c8fe1940b7f3 [SPARK-43943][SPARK-43935][SPARK-43930][DOCS][FOLLOW-UP] Add missing `versionadded` annotations
    omit 83e7fae4187f [SPARK-43648][CONNECT][TESTS] Move `interrupt all` related test to a new test file to pass Maven test
    omit efa1eb98ef30 [SPARK-44010][PYTHON][SS][MINOR] Python StreamingQueryProgress rowsPerSecond type fix
    omit 3f8155c45c97 [SPARK-43803][SS][CONNECT] Improve awaitTermination() to handle client disconnects
    omit f4742fd6efbd [SPARK-43133] Scala Client DataStreamWriter Foreach support
    omit 1089bc43975a [SPARK-43613][PS][CONNECT] Enable `pyspark.pandas.spark.functions.covar` in Spark Connect
    omit e6c6d444ae07 [SPARK-43700][SPARK-43701][CONNECT][PS] Enable `TimedeltaOps.(sub|rsub)` with Spark Connect
    omit 37f9beaba7e8 [SPARK-43943][SQL][TESTS][FOLLOW] Fix `DataFrame function and SQL function parity` in `DataFrameFunctionsSuite`
    omit 4168e1ac3c1b [SPARK-42290][SQL] Fix the OOM error can't be reported when AQE on
    omit 07cc04d3e3f3 [MINOR][SQL][TESTS] Move ResolveDefaultColumnsSuite to 'o.a.s.sql'
    omit 958b85418034 [SPARK-44006][CONNECT][PYTHON] Support cache artifacts
    omit 3cae38b4f109 [SPARK-43612][PYTHON][CONNECT][FOLLOW-UP] Copy dependent data files to data directory
    omit f1cca85f5132 [SPARK-43943][SQL][PYTHON][CONNECT] Add SQL math functions to Scala and Python
    omit 4ddf83f7f42a [SPARK-43916][SQL][PYTHON][CONNECT] Add percentile to Scala and Python API
    omit cbbc06147eca [SPARK-43992][SQL][PYTHON][CONNECT] Add optional pattern for Catalog.listFunctions
    omit fa4aff827912 [SPARK-43994][BUILD] Upgrade zstd-jni to 1.5.5-4
    omit 4c7eff269622 [SPARK-43273][SQL] Support `lz4raw` compression codec for Parquet
    omit 0c025a1cb944 [SPARK-43712][SPARK-43713][CONNECT][PS] Enable parity test: `test_line_plot`, `test_pie_plot`
    omit bd270830614a [SPARK-43935][SQL][PYTHON][CONNECT] Add xpath_* functions to Scala and Python
    omit 2adb8e12f73f [SPARK-44000][SQL][FOLLOWUP] Add comments when picking build side for BNLJ
    omit d88633ada5eb [SPARK-44000][SQL] Add hint to disable broadcasting and replicating one side of join
    omit caf905d7b371 Revert "[SPARK-43840][INFRA] Switch `scala-213` GitHub Action Job to `scala-212`"
    omit 15202e531957 [SPARK-43920][SQL][CONNECT] Create sql/api module
    omit e324d0ac86a2 [SPARK-43897][TESTS][PS][CONNECT] Reorganize ps.Series unit tests
    omit 22297345e453 [SPARK-44002][CONNECT] Fix the artifact statuses handler
    omit 0c7b4306c7c5 [SPARK-43540][K8S][CORE] Add working directory into classpath on the driver in K8S cluster mode
    omit 2057eb7e203c [SPARK-43933][SQL][PYTHON][CONNECT] Add linear regression aggregate functions to Scala and Python
    omit 4ffa2722d135 [SPARK-43984][SQL][PROTOBUF] Change to use `foreach` when `map` doesn't produce results
    omit 89de4f79e7f5 [SPARK-43790][PYTHON][CONNECT][ML] Add `copyFromLocalToFs` API
    omit fead8a7962a7 [SPARK-43993][SQL][TESTS] Add tests for cache artifacts
    omit 1a754913603b [SPARK-43921][PROTOBUF] Generate Protobuf descriptor files at build time
    omit e5b006f89d9e [MINOR][PS][DOCS] Add `DataFrame.{from_dict, to_orc}` to API references
    omit 64855fa55821 [SPARK-43961][SQL][PYTHON][CONNECT] Add optional pattern for Catalog.listTables
    omit 41fd030145a1 [SPARK-43376][SQL][FOLLOWUP] lazy construct subquery to improve reuse subquery
    omit e4c27f53e266 [SPARK-43979][SQL] CollectedMetrics should be treated as the same one for self-join
    omit 5021638ee147 [SPARK-43717][CONNECT] Scala client reduce agg cannot handle null partitions for scala primitive inputs
    omit 0ed48feab65f [SPARK-43985][PROTOBUF] spark protobuf: fix enums as ints bug
    omit d2f72a216777 [SPARK-43901][SQL] Avro to Support custom decimal type backed by Long
    omit e6adc67d43d6 [SPARK-42750][SQL] Support Insert By Name statement
    omit 54304493e0ac [SPARK-43615][TESTS][PS][CONNECT] Enable unit test `test_eval`
    omit 747db6675da8 [SPARK-43930][SQL][PYTHON][CONNECT] Add unix_* functions to Scala and Python
    omit 1e197b5bc225 [SPARK-43356][K8S] Migrate deprecated createOrReplace to serverSideApply
    omit 6fd1c649c72d [SPARK-43906][PYTHON][CONNECT] Implement the file support in SparkSession.addArtifacts
    omit e3957ce87186 [SPARK-43970][PYTHON][CONNECT] Hide unsupported dataframe methods from auto-completion
    omit 94098853592b [SPARK-43893][PYTHON][CONNECT] Non-atomic data type support in Arrow-optimized Python UDF
    omit 957f0e4c9f2d [SPARK-43959][SQL][TESTS] Make RowLevelOperationSuiteBase and AlignAssignmentsSuite abstract
    omit b4ab34bf9b22 [SPARK-43976][CORE] Handle the case where modifiedConfigs doesn't exist in event logs
    omit 8c6a54d70a79 [SPARK-43919][SQL] Extract JSON functionality out of Row
    omit ae8aaec9cdeb [SPARK-43977][CONNECT] Fix unexpected check result of `dev/connect-jvm-client-mima-check`
    omit 2a82b42bcb10 [SPARK-43097][ML] New pyspark ML logistic regression estimator implemented on top of distributor
    omit 0a4f226280c8 [SPARK-43205][SQL][FOLLOWUP] add ExpressionWithUnresolvedIdentifier to simplify code
    omit 89d44d092af4 [SPARK-43510][YARN] Fix YarnAllocator internal state when adding running executor after processing completed containers
    omit 0cd5ca5a7b31 [SPARK-43913][SQL] Assign names to the error class _LEGACY_ERROR_TEMP_[2426-2432]
    omit 61e6227fb62c [SPARK-43962][SQL] Improve error messages: `CANNOT_DECODE_URL`, `CANNOT_MERGE_INCOMPATIBLE_DATA_TYPE`, `CANNOT_PARSE_DECIMAL`, `CANNOT_READ_FILE_FOOTER`, `CANNOT_RECOGNIZE_HIVE_TYPE`
    omit d0fe6d4b7967 [SPARK-43948][SQL] Assign names to the error class _LEGACY_ERROR_TEMP_[0050|0057|0058|0059]
    omit 1df1d7661a3d [SPARK-43516][ML][PYTHON] Update MLv2 Transformer interfaces
    omit 51a919ea8d65 [SPARK-43973][SS][UI] Structured Streaming UI should display failed queries correctly
    omit 62338ed6cd9f [SPARK-42626][CONNECT] Add Destructive Iterator for SparkResult
    omit fc3489d8bb63 [SPARK-43783][SPARK-43784][SPARK-43788][ML] Make MLv2 (ML on spark connect) supports pandas >= 2.0
    omit c618dabc96a7 Revert "[SPARK-43911][SQL] Use toSet to deduplicate the iterator data to prevent the creation of large Array"
    omit 16ee478a9deb [SPARK-42299] Assign name to _LEGACY_ERROR_TEMP_2206
    omit b00210bd0320 [SPARK-43960][PS][TESTS] DataFrameConversionTestsMixin is not tested properly
    omit 7c51618cc262 [SPARK-43881][SQL][PYTHON][CONNECT] Add optional pattern for Catalog.listDatabases
    omit 8ae957247218 [SPARK-43955][BUILD] Upgrade `scalafmt` from 3.7.3 to 3.7.4
    omit 595ad30e6259 [SPARK-43911][SQL] Use toSet to deduplicate the iterator data to prevent the creation of large Array
    omit af61526bea7d [SPARK-43917][PS][INFRA] Upgrade `pandas` to 2.0.2
    omit 7b25b241368e [SPARK-43953][CONNECT] Remove `pass`
    omit cf6ef1173577 [SPARK-43954][BUILD] Upgrade sbt from 1.8.3 to 1.9.0
    omit 90ec7eaddb66 [SPARK-43957][SQL][TESTS] Use `checkError()` to check `Exception` in `*Insert*Suite`
    omit 898ad77900d8 [SPARK-43956][SQL] Fix the bug doesn't display column's sql for Percentile[Cont|Disc]
    omit 14b00cfc6c2e [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results
    omit 0effbec16edc [SPARK-36612][SQL] Support left outer join build left or right outer join build right in shuffled hash join
    omit 41d34e562ffe [SPARK-43904][BUILD] Upgrade jackson to 2.15.2
    omit 592857945c45 [SPARK-43950][BUILD] Upgrade `kubernetes-client` to 6.7.0
    omit 18b9bd9dcb0b [SPARK-43945][SQL][TESTS] Fix bug for `SQLQueryTestSuite` when run on local env
    omit c3b62708cd63 [SPARK-43516][ML][FOLLOW-UP] Drop vector type support in Distributed ML for spark connect
    omit 085dfeb2bed6 [SPARK-43949][PYTHON] Upgrade cloudpickle to 2.2.1
    omit fed51b9eeaf5 [SPARK-43063][SQL][FOLLOWUP] Add a space between `->` and value
    omit 74b04eeffdc4 [SPARK-43936][SQL] Fix bug for toSQLId
    omit 6f593beaf2f5 [SPARK-43267][JDBC] Handle postgres unknown user-defined column as string in array
    omit f2f6272dd977 [SPARK-43892][PYTHON] Add autocomplete support for `df[|]` in `pyspark.sql.dataframe.DataFrame`
    omit 869bd30bd97a [SPARK-43899][BUILD] Upgrade protobuf-java to 3.23.2
    omit 480b14f4b45a [SPARK-41593][FOLLOW-UP][ML] Torch distributor log streaming server: Avoid duplicated log to stdout redirection
    omit adabbb50053d [SPARK-43910][SQL] Strip `__auto_generated_subquery_name` from ids in errors
    omit 40298e906329 [SPARK-43030][SQL][FOLLOWUP] Fix DeduplicateRelations with duplicate aliases
    omit c8c6b0c33084 Revert [SPARK-43836][SPARK-43845][SPARK-43858][SPARK-43858] to keep Scala 2.12 for Spark 3.x
    omit c02c8be43c64 [SPARK-43421][SS] Implement Changelog based Checkpointing for RocksDB State Store Provider
    omit 10ee643c6746 [SPARK-43504][K8S] Mounts the hadoop config map on the executor pod
    omit 398435fcd6eb [SPARK-43898][CORE] Automatically register `immutable.ArraySeq$ofRef` to `KryoSerializer` for Scala 2.13
    omit 33a76811b23f [SPARK-43516][ML][FOLLOW-UP] Make `pyspark.mlv2` module supports python < 3.9
    omit e3eba0b8877c [SPARK-43896][TESTS][PS][CONNECT] Enable `test_iterrows` and `test_itertuples` on Connect
    omit 8f2afb88d42a [SPARK-43333][SQL] Allow Avro to convert union type to SQL with field name stable with type
    omit 419f46ab621e [SPARK-43895][CONNECT][GO] Prepare the go package path
    omit 47ddac13144b [SPARK-43804][PYTHON][TESTS] Test on nested structs support in Pandas UDF
    omit a88934227523 [SPARK-43867][SQL] Improve suggested candidates for unresolved attribute
    omit 518888359b3e [SPARK-43888][CORE] Relocate Logging to common/utils
    omit d3f76c6ca07a [SPARK-43894][PYTHON] Fix bug in df.cache()
    omit fead25ac4d60 [SPARK-43775][SQL] DataSource V2: Allow representing updates as deletes and inserts
... 36927 lines suppressed ...


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


(spark) 01/01: [SPARK-55305][SQL][TESTS] Use `ParquetFooterReader.readFooter` uniformly in test code to read the footer

Posted by ya...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yao pushed a commit to branch feature/crossjoin-array-contains-benchmark
in repository https://gitbox.apache.org/repos/asf/spark.git

commit bd84f87b7e3aec49461e8a86010fa9afe4df7936
Author: yangjie01 <ya...@baidu.com>
AuthorDate: Mon Feb 2 16:01:25 2026 +0800

    [SPARK-55305][SQL][TESTS] Use `ParquetFooterReader.readFooter` uniformly in test code to read the footer
    
    ### What changes were proposed in this pull request?
    This pr change to use `ParquetFooterReader.readFooter` uniformly in Spark test code to read the footer.
    
    ### Why are the changes needed?
    Use the methods encapsulated by Spark and avoid using deprecated APIs in `ParquetFileReader` (specifically, `ParquetFileReader#readFooter(Configuration, Path)` has been deprecated).
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    Pass Github Actions
    
    ### Was this patch authored or co-authored using generative AI tooling?
    No
    
    Closes #54086 from LuciferYang/use-ParquetFooterReader-readFooter-in-tests.
    
    Authored-by: yangjie01 <ya...@baidu.com>
    Signed-off-by: yangjie01 <ya...@baidu.com>
---
 .asf.yaml                                          |    41 +
 .gitattributes                                     |     7 +
 .github/PULL_REQUEST_TEMPLATE                      |    59 +
 .github/labeler.yml                                |   234 +
 .github/workflows/benchmark.yml                    |   230 +
 .github/workflows/build_and_test.yml               |  1464 +
 .github/workflows/build_branch35.yml               |    53 +
 .github/workflows/build_branch35_python.yml        |    47 +
 .github/workflows/build_branch40.yml               |    53 +
 .github/workflows/build_branch40_java21.yml        |    57 +
 .github/workflows/build_branch40_maven.yml         |    35 +
 .github/workflows/build_branch40_maven_java21.yml  |    36 +
 .github/workflows/build_branch40_non_ansi.yml      |    53 +
 .github/workflows/build_branch40_python.yml        |    47 +
 .../workflows/build_branch40_python_pypy3.10.yml   |    47 +
 .github/workflows/build_branch41.yml               |    53 +
 .github/workflows/build_branch41_java21.yml        |    57 +
 .github/workflows/build_branch41_maven.yml         |    35 +
 .github/workflows/build_branch41_maven_java21.yml  |    36 +
 .github/workflows/build_branch41_non_ansi.yml      |    53 +
 .github/workflows/build_branch41_python.yml        |    47 +
 .github/workflows/build_branch41_python_3.14.yml   |    47 +
 .../workflows/build_branch41_python_pypy3.10.yml   |    47 +
 .github/workflows/build_coverage.yml               |    51 +
 .github/workflows/build_infra_images_cache.yml     |   273 +
 .github/workflows/build_java21.yml                 |    57 +
 .github/workflows/build_main.yml                   |    34 +
 .github/workflows/build_maven.yml                  |    33 +
 .github/workflows/build_maven_java21.yml           |    35 +
 .github/workflows/build_maven_java21_arm.yml       |    37 +
 .github/workflows/build_maven_java21_macos26.yml   |    44 +
 .github/workflows/build_non_ansi.yml               |    55 +
 .github/workflows/build_python_3.10.yml            |    47 +
 .github/workflows/build_python_3.11.yml            |    47 +
 .github/workflows/build_python_3.12_arm.yml        |    35 +
 .../workflows/build_python_3.12_classic_only.yml   |    47 +
 .github/workflows/build_python_3.12_macos26.yml    |    35 +
 .github/workflows/build_python_3.12_pandas_3.yml   |    47 +
 .github/workflows/build_python_3.13.yml            |    47 +
 .github/workflows/build_python_3.13_nogil.yml      |    48 +
 .github/workflows/build_python_3.14.yml            |    47 +
 .github/workflows/build_python_3.14_nogil.yml      |    48 +
 .github/workflows/build_python_connect.yml         |   140 +
 .github/workflows/build_python_connect40.yml       |   120 +
 .github/workflows/build_python_minimum.yml         |    46 +
 .github/workflows/build_python_ps_minimum.yml      |    47 +
 .github/workflows/build_python_pypy3.10.yml        |    47 +
 .github/workflows/build_python_pypy3.11.yml        |    47 +
 .github/workflows/build_rockdb_as_ui_backend.yml   |    50 +
 .github/workflows/build_sparkr_window.yml          |    93 +
 .github/workflows/build_uds.yml                    |    53 +
 .../workflows/images/workflow-enable-button.png    |   Bin 0 -> 79807 bytes
 .github/workflows/labeler.yml                      |   192 +
 .github/workflows/maven_test.yml                   |   251 +
 .github/workflows/notify_test_workflow.yml         |   168 +
 .github/workflows/pages.yml                        |    98 +
 .github/workflows/publish_snapshot.yml             |    75 +
 .github/workflows/python_hosted_runner_test.yml    |   194 +
 .github/workflows/release.yml                      |   295 +
 .github/workflows/stale.yml                        |    44 +
 .github/workflows/test_report.yml                  |    48 +
 .github/workflows/update_build_status.yml          |   108 +
 .gitignore                                         |   140 +
 .mvn/extensions.xml                                |    24 +
 .mvn/jvm.config                                    |     4 +
 .mvn/maven.config                                  |     1 +
 .nojekyll                                          |     0
 .pre-commit-config.yaml                            |    33 +
 .sbtopts                                           |    20 +
 CONTRIBUTING.md                                    |    16 +
 LICENSE                                            |   267 +
 LICENSE-binary                                     |   546 +
 NOTICE                                             |    41 +
 NOTICE-binary                                      |  1172 +
 R/.gitignore                                       |     8 +
 R/CRAN_RELEASE.md                                  |   109 +
 R/DOCUMENTATION.md                                 |    30 +
 R/WINDOWS.md                                       |    63 +
 R/check-cran.sh                                    |    80 +
 R/create-docs.sh                                   |    70 +
 R/create-rd.sh                                     |    37 +
 R/find-r.sh                                        |    34 +
 R/install-dev.bat                                  |    39 +
 R/install-dev.sh                                   |    50 +
 R/install-source-package.sh                        |    57 +
 R/log4j2.properties                                |    31 +
 R/pkg/.Rbuildignore                                |    12 +
 R/pkg/.gitignore                                   |     1 +
 R/pkg/.lintr                                       |     2 +
 R/pkg/DESCRIPTION                                  |    64 +
 R/pkg/NAMESPACE                                    |   585 +
 R/pkg/R/DataFrame.R                                |  4335 +++
 R/pkg/R/RDD.R                                      |  1734 ++
 R/pkg/R/SQLContext.R                               |   737 +
 R/pkg/R/WindowSpec.R                               |   253 +
 R/pkg/R/backend.R                                  |   134 +
 R/pkg/R/broadcast.R                                |    84 +
 R/pkg/R/catalog.R                                  |   750 +
 R/pkg/R/client.R                                   |   125 +
 R/pkg/R/column.R                                   |   462 +
 R/pkg/R/context.R                                  |   462 +
 R/pkg/R/deserialize.R                              |   265 +
 R/pkg/R/functions.R                                |  5105 ++++
 R/pkg/R/generics.R                                 |  1747 ++
 R/pkg/R/group.R                                    |   257 +
 R/pkg/R/install.R                                  |   335 +
 R/pkg/R/jobj.R                                     |   110 +
 R/pkg/R/jvm.R                                      |   114 +
 R/pkg/R/mllib_classification.R                     |   808 +
 R/pkg/R/mllib_clustering.R                         |   671 +
 R/pkg/R/mllib_fpm.R                                |   223 +
 R/pkg/R/mllib_recommendation.R                     |   165 +
 R/pkg/R/mllib_regression.R                         |   865 +
 R/pkg/R/mllib_stat.R                               |   122 +
 R/pkg/R/mllib_tree.R                               |   762 +
 R/pkg/R/mllib_utils.R                              |   135 +
 R/pkg/R/pairRDD.R                                  |   954 +
 R/pkg/R/schema.R                                   |   245 +
 R/pkg/R/serialize.R                                |   242 +
 R/pkg/R/sparkR.R                                   |   839 +
 R/pkg/R/stats.R                                    |   221 +
 R/pkg/R/streaming.R                                |   205 +
 R/pkg/R/types.R                                    |   114 +
 R/pkg/R/utils.R                                    |   939 +
 R/pkg/R/window.R                                   |   112 +
 R/pkg/R/zzz.R                                      |    30 +
 R/pkg/README.md                                    |    73 +
 R/pkg/inst/profile/general.R                       |    23 +
 R/pkg/inst/profile/shell.R                         |    49 +
 R/pkg/inst/tests/testthat/test_basic.R             |   100 +
 R/pkg/inst/worker/daemon.R                         |   102 +
 R/pkg/inst/worker/worker.R                         |   292 +
 R/pkg/pkgdown/_pkgdown_template.yml                |   321 +
 R/pkg/pkgdown/extra.css                            |    48 +
 R/pkg/src-native/Makefile                          |    27 +
 R/pkg/src-native/Makefile.win                      |    27 +
 R/pkg/src-native/string_hash_code.c                |    49 +
 R/pkg/tests/fulltests/data/test_utils_utf.json     |     4 +
 R/pkg/tests/fulltests/jarTest.R                    |    32 +
 R/pkg/tests/fulltests/packageInAJarTest.R          |    30 +
 R/pkg/tests/fulltests/test_Serde.R                 |   158 +
 R/pkg/tests/fulltests/test_Windows.R               |    28 +
 R/pkg/tests/fulltests/test_binaryFile.R            |    92 +
 R/pkg/tests/fulltests/test_binary_function.R       |   104 +
 R/pkg/tests/fulltests/test_broadcast.R             |    51 +
 R/pkg/tests/fulltests/test_client.R                |    43 +
 R/pkg/tests/fulltests/test_context.R               |   260 +
 R/pkg/tests/fulltests/test_includePackage.R        |    62 +
 R/pkg/tests/fulltests/test_jvm_api.R               |    36 +
 R/pkg/tests/fulltests/test_mllib_classification.R  |   525 +
 R/pkg/tests/fulltests/test_mllib_clustering.R      |   335 +
 R/pkg/tests/fulltests/test_mllib_fpm.R             |   104 +
 R/pkg/tests/fulltests/test_mllib_recommendation.R  |    68 +
 R/pkg/tests/fulltests/test_mllib_regression.R      |   613 +
 R/pkg/tests/fulltests/test_mllib_stat.R            |    53 +
 R/pkg/tests/fulltests/test_mllib_tree.R            |   365 +
 R/pkg/tests/fulltests/test_parallelize_collect.R   |   112 +
 R/pkg/tests/fulltests/test_rdd.R                   |   804 +
 R/pkg/tests/fulltests/test_shuffle.R               |   224 +
 R/pkg/tests/fulltests/test_sparkR.R                |    46 +
 R/pkg/tests/fulltests/test_sparkSQL.R              |  4336 +++
 R/pkg/tests/fulltests/test_sparkSQL_arrow.R        |   267 +
 R/pkg/tests/fulltests/test_sparkSQL_eager.R        |    72 +
 R/pkg/tests/fulltests/test_streaming.R             |   284 +
 R/pkg/tests/fulltests/test_take.R                  |    69 +
 R/pkg/tests/fulltests/test_textFile.R              |   165 +
 R/pkg/tests/fulltests/test_utils.R                 |   281 +
 R/pkg/tests/run-all.R                              |    98 +
 R/pkg/vignettes/sparkr-vignettes.Rmd               |  1296 +
 R/run-tests.sh                                     |    75 +
 README.md                                          |   174 +
 assembly/pom.xml                                   |   383 +
 bin/beeline                                        |    32 +
 bin/beeline.cmd                                    |    22 +
 bin/docker-image-tool.sh                           |   336 +
 bin/find-spark-home                                |    41 +
 bin/find-spark-home.cmd                            |    60 +
 bin/load-spark-env.cmd                             |    62 +
 bin/load-spark-env.sh                              |    70 +
 bin/pyspark                                        |   102 +
 bin/pyspark.cmd                                    |    25 +
 bin/pyspark2.cmd                                   |    38 +
 bin/run-example                                    |    25 +
 bin/run-example.cmd                                |    27 +
 bin/spark-class                                    |   112 +
 bin/spark-class.cmd                                |    25 +
 bin/spark-class2.cmd                               |    89 +
 bin/spark-connect-shell                            |    26 +
 bin/spark-pipelines                                |    40 +
 bin/spark-shell                                    |   101 +
 bin/spark-shell.cmd                                |    25 +
 bin/spark-shell2.cmd                               |    43 +
 bin/spark-sql                                      |    26 +
 bin/spark-sql.cmd                                  |    25 +
 bin/spark-sql2.cmd                                 |    27 +
 bin/spark-submit                                   |    27 +
 bin/spark-submit.cmd                               |    25 +
 bin/spark-submit2.cmd                              |    27 +
 bin/sparkR                                         |    27 +
 bin/sparkR.cmd                                     |    25 +
 bin/sparkR2.cmd                                    |    27 +
 binder/Dockerfile                                  |    43 +
 binder/postBuild                                   |    55 +
 build/mvn                                          |   175 +
 build/sbt                                          |   149 +
 build/sbt-launch-lib.bash                          |   193 +
 build/spark-build-info                             |    39 +
 build/spark-build-info.ps1                         |    47 +
 build/util.sh                                      |    38 +
 common/kvstore/pom.xml                             |   120 +
 .../apache/spark/util/kvstore/ArrayWrappers.java   |   214 +
 .../apache/spark/util/kvstore/InMemoryStore.java   |   501 +
 .../org/apache/spark/util/kvstore/KVIndex.java     |    85 +
 .../org/apache/spark/util/kvstore/KVStore.java     |   135 +
 .../apache/spark/util/kvstore/KVStoreIterator.java |    51 +
 .../spark/util/kvstore/KVStoreSerializer.java      |    83 +
 .../org/apache/spark/util/kvstore/KVStoreView.java |   121 +
 .../org/apache/spark/util/kvstore/KVTypeInfo.java  |   168 +
 .../org/apache/spark/util/kvstore/LevelDB.java     |   432 +
 .../apache/spark/util/kvstore/LevelDBIterator.java |   339 +
 .../apache/spark/util/kvstore/LevelDBTypeInfo.java |   510 +
 .../org/apache/spark/util/kvstore/RocksDB.java     |   464 +
 .../apache/spark/util/kvstore/RocksDBIterator.java |   321 +
 .../apache/spark/util/kvstore/RocksDBTypeInfo.java |   512 +
 .../kvstore/UnsupportedStoreVersionException.java  |    30 +
 .../spark/util/kvstore/ArrayKeyIndexType.java      |    43 +
 .../spark/util/kvstore/ArrayWrappersSuite.java     |    59 +
 .../org/apache/spark/util/kvstore/CustomType1.java |    55 +
 .../org/apache/spark/util/kvstore/CustomType2.java |    49 +
 .../apache/spark/util/kvstore/DBIteratorSuite.java |   514 +
 .../spark/util/kvstore/InMemoryIteratorSuite.java  |    27 +
 .../spark/util/kvstore/InMemoryStoreSuite.java     |   240 +
 .../org/apache/spark/util/kvstore/IntKeyType.java  |    46 +
 .../spark/util/kvstore/LevelDBBenchmark.java       |   290 +
 .../spark/util/kvstore/LevelDBIteratorSuite.java   |    52 +
 .../apache/spark/util/kvstore/LevelDBSuite.java    |   493 +
 .../spark/util/kvstore/LevelDBTypeInfoSuite.java   |   213 +
 .../spark/util/kvstore/RocksDBIteratorSuite.java   |    49 +
 .../apache/spark/util/kvstore/RocksDBSuite.java    |   491 +
 .../spark/util/kvstore/RocksDBTypeInfoSuite.java   |   213 +
 .../kvstore/src/test/resources/log4j2.properties   |    31 +
 common/network-common/pom.xml                      |   229 +
 .../org/apache/spark/network/TransportContext.java |   320 +
 .../network/buffer/FileSegmentManagedBuffer.java   |   155 +
 .../apache/spark/network/buffer/ManagedBuffer.java |    92 +
 .../spark/network/buffer/NettyManagedBuffer.java   |    78 +
 .../spark/network/buffer/NioManagedBuffer.java     |    77 +
 .../spark/network/client/BaseResponseCallback.java |    31 +
 .../network/client/ChunkFetchFailureException.java |    31 +
 .../network/client/ChunkReceivedCallback.java      |    47 +
 .../client/MergedBlockMetaResponseCallback.java    |    41 +
 .../spark/network/client/RpcResponseCallback.java  |    34 +
 .../spark/network/client/StreamCallback.java       |    40 +
 .../spark/network/client/StreamCallbackWithID.java |    32 +
 .../spark/network/client/StreamInterceptor.java    |    95 +
 .../spark/network/client/TransportClient.java      |   399 +
 .../network/client/TransportClientBootstrap.java   |    34 +
 .../network/client/TransportClientFactory.java     |   376 +
 .../network/client/TransportResponseHandler.java   |   301 +
 .../spark/network/crypto/AuthClientBootstrap.java  |   125 +
 .../apache/spark/network/crypto/AuthEngine.java    |   262 +
 .../apache/spark/network/crypto/AuthMessage.java   |    65 +
 .../spark/network/crypto/AuthRpcHandler.java       |   144 +
 .../spark/network/crypto/AuthServerBootstrap.java  |    56 +
 .../spark/network/crypto/CtrTransportCipher.java   |   381 +
 .../spark/network/crypto/GcmTransportCipher.java   |   413 +
 .../java/org/apache/spark/network/crypto/README.md |   116 +
 .../spark/network/crypto/TransportCipher.java      |    28 +
 .../spark/network/crypto/TransportCipherUtil.java  |    41 +
 .../spark/network/protocol/AbstractMessage.java    |    54 +
 .../network/protocol/AbstractResponseMessage.java  |    32 +
 .../spark/network/protocol/ChunkFetchFailure.java  |    73 +
 .../spark/network/protocol/ChunkFetchRequest.java  |    67 +
 .../spark/network/protocol/ChunkFetchSuccess.java  |    86 +
 .../apache/spark/network/protocol/Encodable.java   |    41 +
 .../apache/spark/network/protocol/Encoders.java    |   217 +
 .../protocol/EncryptedMessageWithHeader.java       |   146 +
 .../network/protocol/MergedBlockMetaRequest.java   |    99 +
 .../network/protocol/MergedBlockMetaSuccess.java   |    90 +
 .../org/apache/spark/network/protocol/Message.java |    77 +
 .../spark/network/protocol/MessageDecoder.java     |    70 +
 .../spark/network/protocol/MessageEncoder.java     |   100 +
 .../spark/network/protocol/MessageWithHeader.java  |   204 +
 .../spark/network/protocol/OneWayMessage.java      |    77 +
 .../spark/network/protocol/RequestMessage.java     |    23 +
 .../spark/network/protocol/ResponseMessage.java    |    23 +
 .../apache/spark/network/protocol/RpcFailure.java  |    71 +
 .../apache/spark/network/protocol/RpcRequest.java  |    83 +
 .../apache/spark/network/protocol/RpcResponse.java |    83 +
 .../spark/network/protocol/SslMessageEncoder.java  |   110 +
 .../spark/network/protocol/StreamChunkId.java      |    64 +
 .../spark/network/protocol/StreamFailure.java      |    74 +
 .../spark/network/protocol/StreamRequest.java      |    73 +
 .../spark/network/protocol/StreamResponse.java     |    88 +
 .../spark/network/protocol/UploadStream.java       |   102 +
 .../spark/network/sasl/SaslClientBootstrap.java    |   110 +
 .../apache/spark/network/sasl/SaslEncryption.java  |   329 +
 .../spark/network/sasl/SaslEncryptionBackend.java  |    33 +
 .../org/apache/spark/network/sasl/SaslMessage.java |    79 +
 .../apache/spark/network/sasl/SaslRpcHandler.java  |   145 +
 .../spark/network/sasl/SaslServerBootstrap.java    |    50 +
 .../spark/network/sasl/SaslTimeoutException.java   |    35 +
 .../apache/spark/network/sasl/SecretKeyHolder.java |    35 +
 .../apache/spark/network/sasl/SparkSaslClient.java |   158 +
 .../apache/spark/network/sasl/SparkSaslServer.java |   213 +
 .../network/server/AbstractAuthRpcHandler.java     |   112 +
 .../network/server/BlockPushNonFatalFailure.java   |   176 +
 .../network/server/ChunkFetchRequestHandler.java   |   162 +
 .../spark/network/server/MessageHandler.java       |    39 +
 .../spark/network/server/NoOpRpcHandler.java       |    40 +
 .../network/server/OneForOneStreamManager.java     |   251 +
 .../apache/spark/network/server/RpcHandler.java    |   175 +
 .../apache/spark/network/server/StreamManager.java |   103 +
 .../network/server/TransportChannelHandler.java    |   207 +
 .../network/server/TransportRequestHandler.java    |   335 +
 .../spark/network/server/TransportServer.java      |   181 +
 .../network/server/TransportServerBootstrap.java   |    36 +
 .../org/apache/spark/network/shuffledb/DB.java     |    50 +
 .../apache/spark/network/shuffledb/DBBackend.java  |    42 +
 .../apache/spark/network/shuffledb/DBIterator.java |    38 +
 .../apache/spark/network/shuffledb/LevelDB.java    |    53 +
 .../spark/network/shuffledb/LevelDBIterator.java   |    86 +
 .../apache/spark/network/shuffledb/RocksDB.java    |    70 +
 .../spark/network/shuffledb/RocksDBIterator.java   |    94 +
 .../spark/network/shuffledb/StoreVersion.java      |    62 +
 .../network/ssl/ReloadingX509TrustManager.java     |   225 +
 .../org/apache/spark/network/ssl/SSLFactory.java   |   483 +
 .../spark/network/util/AbstractFileRegion.java     |    53 +
 .../network/util/ByteArrayReadableChannel.java     |    62 +
 .../network/util/ByteArrayWritableChannel.java     |    69 +
 .../network/util/ByteBufferWriteableChannel.java   |    59 +
 .../apache/spark/network/util/ConfigProvider.java  |    57 +
 .../org/apache/spark/network/util/CryptoUtils.java |    49 +
 .../org/apache/spark/network/util/DBProvider.java  |    69 +
 .../java/org/apache/spark/network/util/IOMode.java |    40 +
 .../apache/spark/network/util/LevelDBProvider.java |   136 +
 .../spark/network/util/LimitedInputStream.java     |   109 +
 .../spark/network/util/MapConfigProvider.java      |    56 +
 .../org/apache/spark/network/util/NettyLogger.java |    77 +
 .../spark/network/util/NettyMemoryMetrics.java     |   145 +
 .../org/apache/spark/network/util/NettyUtils.java  |   219 +
 .../apache/spark/network/util/RocksDBProvider.java |   176 +
 .../network/util/TimerWithCustomTimeUnit.java      |   124 +
 .../apache/spark/network/util/TransportConf.java   |   606 +
 .../spark/network/util/TransportFrameDecoder.java  |   286 +
 .../spark/network/ChunkFetchIntegrationSuite.java  |   250 +
 .../network/ChunkFetchRequestHandlerSuite.java     |   110 +
 .../spark/network/ExtendedChannelPromise.java      |    69 +
 .../org/apache/spark/network/ProtocolSuite.java    |   128 +
 .../network/RequestTimeoutIntegrationSuite.java    |   277 +
 .../apache/spark/network/RpcIntegrationSuite.java  |   467 +
 .../network/SslChunkFetchIntegrationSuite.java     |    32 +
 .../java/org/apache/spark/network/StreamSuite.java |   305 +
 .../org/apache/spark/network/StreamTestHelper.java |    92 +
 .../apache/spark/network/TestManagedBuffer.java    |   114 +
 .../java/org/apache/spark/network/TestUtils.java   |    32 +
 .../apache/spark/network/TransportConfSuite.java   |   110 +
 .../network/TransportRequestHandlerSuite.java      |   166 +
 .../network/TransportResponseHandlerSuite.java     |   209 +
 .../client/SslTransportClientFactorySuite.java     |    39 +
 .../client/TransportClientFactorySuite.java        |   268 +
 .../spark/network/crypto/AuthEngineSuite.java      |   175 +
 .../spark/network/crypto/AuthIntegrationSuite.java |   275 +
 .../spark/network/crypto/AuthMessagesSuite.java    |    53 +
 .../spark/network/crypto/CtrAuthEngineSuite.java   |   177 +
 .../spark/network/crypto/GcmAuthEngineSuite.java   |   339 +
 .../spark/network/crypto/TransportCipherSuite.java |    80 +
 .../spark/network/protocol/EncodersSuite.java      |    69 +
 .../protocol/EncryptedMessageWithHeaderSuite.java  |   154 +
 .../protocol/MergedBlockMetaSuccessSuite.java      |   101 +
 .../network/protocol/MessageWithHeaderSuite.java   |   186 +
 .../apache/spark/network/sasl/SparkSaslSuite.java  |   459 +
 .../server/OneForOneStreamManagerSuite.java        |   165 +
 .../ssl/ReloadingX509TrustManagerSuite.java        |   314 +
 .../apache/spark/network/ssl/SSLFactorySuite.java  |   104 +
 .../apache/spark/network/ssl/SslSampleConfigs.java |   251 +
 .../spark/network/util/CryptoUtilsSuite.java       |    50 +
 .../apache/spark/network/util/DBProviderSuite.java |    61 +
 .../apache/spark/network/util/JavaUtilsSuite.java  |    88 +
 .../network/util/NettyMemoryMetricsSuite.java      |   169 +
 .../network/util/TimerWithCustomUnitSuite.java     |   109 +
 .../network/util/TransportFrameDecoderSuite.java   |   312 +
 .../src/test/resources/certchain.pem               |    17 +
 common/network-common/src/test/resources/key.pem   |    29 +
 common/network-common/src/test/resources/keystore  |   Bin 0 -> 2247 bytes
 .../src/test/resources/log4j2.properties           |    31 +
 .../network-common/src/test/resources/truststore   |   Bin 0 -> 957 bytes
 .../src/test/resources/unencrypted-certchain.pem   |    21 +
 .../src/test/resources/unencrypted-key.pem         |    28 +
 .../src/test/resources/untrusted-keystore          |   Bin 0 -> 2246 bytes
 common/network-shuffle/pom.xml                     |   136 +
 .../spark/network/sasl/ShuffleSecretManager.java   |    96 +
 .../network/shuffle/AppsWithRecoveryDisabled.java  |    62 +
 .../network/shuffle/BlockFetchingListener.java     |    49 +
 .../network/shuffle/BlockPushingListener.java      |    54 +
 .../spark/network/shuffle/BlockStoreClient.java    |   274 +
 .../network/shuffle/BlockTransferListener.java     |    44 +
 .../apache/spark/network/shuffle/Constants.java    |    31 +
 .../apache/spark/network/shuffle/DownloadFile.java |    47 +
 .../spark/network/shuffle/DownloadFileManager.java |    36 +
 .../shuffle/DownloadFileWritableChannel.java       |    31 +
 .../apache/spark/network/shuffle/ErrorHandler.java |   115 +
 .../spark/network/shuffle/ExecutorDiskUtils.java   |    45 +
 .../network/shuffle/ExternalBlockHandler.java      |   613 +
 .../network/shuffle/ExternalBlockStoreClient.java  |   350 +
 .../shuffle/ExternalShuffleBlockResolver.java      |   496 +
 .../network/shuffle/MergeFinalizerListener.java    |    43 +
 .../spark/network/shuffle/MergedBlockMeta.java     |    66 +
 .../network/shuffle/MergedBlocksMetaListener.java  |    50 +
 .../network/shuffle/MergedShuffleFileManager.java  |   151 +
 .../shuffle/NoOpMergedShuffleFileManager.java      |    93 +
 .../network/shuffle/OneForOneBlockFetcher.java     |   382 +
 .../network/shuffle/OneForOneBlockPusher.java      |   171 +
 .../network/shuffle/RemoteBlockPushResolver.java   |  2262 ++
 .../network/shuffle/RetryingBlockTransferor.java   |   373 +
 .../network/shuffle/ShuffleIndexInformation.java   |    71 +
 .../spark/network/shuffle/ShuffleIndexRecord.java  |    24 +
 .../network/shuffle/ShuffleTransportContext.java   |   187 +
 .../spark/network/shuffle/SimpleDownloadFile.java  |    92 +
 .../spark/network/shuffle/checksum/Cause.java      |    25 +
 .../shuffle/checksum/ShuffleChecksumHelper.java    |   176 +
 .../protocol/AbstractFetchShuffleBlocks.java       |    80 +
 .../shuffle/protocol/BlockPushReturnCode.java      |    88 +
 .../shuffle/protocol/BlockTransferMessage.java     |   102 +
 .../network/shuffle/protocol/BlocksRemoved.java    |    70 +
 .../network/shuffle/protocol/CorruptionCause.java  |    70 +
 .../shuffle/protocol/DiagnoseCorruption.java       |   123 +
 .../shuffle/protocol/ExecutorShuffleInfo.java      |    95 +
 .../shuffle/protocol/FetchShuffleBlockChunks.java  |   140 +
 .../shuffle/protocol/FetchShuffleBlocks.java       |   140 +
 .../shuffle/protocol/FinalizeShuffleMerge.java     |    96 +
 .../shuffle/protocol/GetLocalDirsForExecutors.java |    77 +
 .../shuffle/protocol/LocalDirsForExecutors.java    |   113 +
 .../network/shuffle/protocol/MergeStatuses.java    |   129 +
 .../spark/network/shuffle/protocol/OpenBlocks.java |    86 +
 .../network/shuffle/protocol/PushBlockStream.java  |   121 +
 .../network/shuffle/protocol/RegisterExecutor.java |    91 +
 .../network/shuffle/protocol/RemoveBlocks.java     |    86 +
 .../shuffle/protocol/RemoveShuffleMerge.java       |    96 +
 .../network/shuffle/protocol/StreamHandle.java     |    78 +
 .../network/shuffle/protocol/UploadBlock.java      |   111 +
 .../shuffle/protocol/UploadBlockStream.java        |    85 +
 .../spark/network/sasl/SaslIntegrationSuite.java   |   158 +
 .../network/sasl/ShuffleSecretManagerSuite.java    |    55 +
 .../spark/network/shuffle/AppIsolationSuite.java   |   184 +
 .../shuffle/BlockTransferMessagesSuite.java        |    72 +
 .../CleanupNonShuffleServiceServedFilesSuite.java  |   236 +
 .../spark/network/shuffle/ErrorHandlerSuite.java   |    70 +
 .../network/shuffle/ExternalBlockHandlerSuite.java |   485 +
 .../shuffle/ExternalShuffleBlockResolverSuite.java |   123 +
 .../shuffle/ExternalShuffleCleanupSuite.java       |   147 +
 .../shuffle/ExternalShuffleIntegrationSuite.java   |   393 +
 .../shuffle/ExternalShuffleSecuritySuite.java      |   142 +
 .../shuffle/OneForOneBlockFetcherSuite.java        |   341 +
 .../network/shuffle/OneForOneBlockPusherSuite.java |   163 +
 .../shuffle/RemoteBlockPushResolverSuite.java      |  1665 ++
 .../shuffle/RetryingBlockTransferorSuite.java      |   475 +
 .../shuffle/ShuffleIndexInformationSuite.java      |    71 +
 .../shuffle/ShuffleTransportContextSuite.java      |   147 +
 .../network/shuffle/SimpleDownloadFileSuite.java   |    48 +
 .../SslExternalShuffleIntegrationSuite.java        |    44 +
 .../shuffle/SslExternalShuffleSecuritySuite.java   |    43 +
 .../shuffle/SslShuffleTransportContextSuite.java   |    38 +
 .../network/shuffle/TestShuffleDataContext.java    |   148 +
 .../protocol/FetchShuffleBlockChunksSuite.java     |    42 +
 .../shuffle/protocol/FetchShuffleBlocksSuite.java  |    42 +
 .../src/test/resources/certchain.pem               |    17 +
 common/network-shuffle/src/test/resources/key.pem  |    29 +
 common/network-shuffle/src/test/resources/keystore |   Bin 0 -> 2247 bytes
 .../src/test/resources/log4j2.properties           |    27 +
 .../network-shuffle/src/test/resources/truststore  |   Bin 0 -> 957 bytes
 .../src/test/resources/unencrypted-certchain.pem   |    21 +
 .../src/test/resources/unencrypted-key.pem         |    28 +
 .../src/test/resources/untrusted-keystore          |   Bin 0 -> 2246 bytes
 common/network-yarn/pom.xml                        |   249 +
 .../spark/network/yarn/YarnShuffleService.java     |   619 +
 .../network/yarn/YarnShuffleServiceMetrics.java    |   154 +
 .../network/yarn/util/HadoopConfigProvider.java    |    55 +
 common/sketch/pom.xml                              |    66 +
 .../org/apache/spark/util/sketch/BitArray.java     |   126 +
 .../org/apache/spark/util/sketch/BloomFilter.java  |   317 +
 .../apache/spark/util/sketch/BloomFilterBase.java  |   199 +
 .../apache/spark/util/sketch/BloomFilterImpl.java  |   134 +
 .../spark/util/sketch/BloomFilterImplV2.java       |   160 +
 .../apache/spark/util/sketch/CountMinSketch.java   |   222 +
 .../spark/util/sketch/CountMinSketchImpl.java      |   367 +
 .../util/sketch/IncompatibleMergeException.java    |    24 +
 .../apache/spark/util/sketch/Murmur3_x86_32.java   |   148 +
 .../org/apache/spark/util/sketch/Platform.java     |   172 +
 .../java/org/apache/spark/util/sketch/Utils.java   |    44 +
 .../spark/util/sketch/SparkBloomFilterSuite.java   |   396 +
 .../apache/spark/util/sketch/BitArraySuite.scala   |    77 +
 .../spark/util/sketch/BloomFilterSuite.scala       |   170 +
 .../spark/util/sketch/CountMinSketchSuite.scala    |   152 +
 common/tags/README.md                              |     1 +
 common/tags/pom.xml                                |    49 +
 .../apache/spark/annotation/AlphaComponent.java    |    33 +
 .../org/apache/spark/annotation/ClassicOnly.java   |    28 +
 .../org/apache/spark/annotation/DeveloperApi.java  |    36 +
 .../java/org/apache/spark/annotation/Evolving.java |    30 +
 .../org/apache/spark/annotation/Experimental.java  |    37 +
 .../java/org/apache/spark/annotation/Private.java  |    39 +
 .../java/org/apache/spark/annotation/Stable.java   |    31 +
 .../java/org/apache/spark/annotation/Unstable.java |    30 +
 .../scala/org/apache/spark/annotation/Since.scala  |    30 +
 .../org/apache/spark/annotation/package-info.java  |    23 +
 .../org/apache/spark/annotation/package.scala      |    25 +
 .../java/org/apache/spark/tags/AmmoniteTest.java   |    30 +
 .../java/org/apache/spark/tags/ChromeUITest.java   |    27 +
 .../java/org/apache/spark/tags/DockerTest.java     |    26 +
 .../org/apache/spark/tags/ExtendedHiveTest.java    |    27 +
 .../org/apache/spark/tags/ExtendedLevelDBTest.java |    30 +
 .../org/apache/spark/tags/ExtendedSQLTest.java     |    30 +
 .../org/apache/spark/tags/ExtendedYarnTest.java    |    27 +
 .../java/org/apache/spark/tags/SlowHiveTest.java   |    30 +
 .../java/org/apache/spark/tags/SlowSQLTest.java    |    30 +
 .../java/org/apache/spark/tags/WebBrowserTest.java |    27 +
 common/unsafe/pom.xml                              |   141 +
 .../spark/sql/catalyst/expressions/HiveHasher.java |    49 +
 .../catalyst/util/CollationAwareUTF8String.java    |  1607 +
 .../spark/sql/catalyst/util/CollationFactory.java  |  1504 +
 .../spark/sql/catalyst/util/CollationNames.java    |    25 +
 .../spark/sql/catalyst/util/CollationSupport.java  |   721 +
 .../spark/sql/catalyst/util/DateTimeConstants.java |    49 +
 .../catalyst/util/SpecialCodePointConstants.java   |    33 +
 .../java/org/apache/spark/unsafe/KVIterator.java   |    31 +
 .../java/org/apache/spark/unsafe/Platform.java     |   339 +
 .../org/apache/spark/unsafe/UTF8StringBuilder.java |   128 +
 .../apache/spark/unsafe/UnsafeAlignedOffset.java   |    64 +
 .../spark/unsafe/array/ByteArrayMethods.java       |   122 +
 .../org/apache/spark/unsafe/array/LongArray.java   |    94 +
 .../apache/spark/unsafe/bitset/BitSetMethods.java  |   130 +
 .../apache/spark/unsafe/hash/Murmur3_x86_32.java   |   148 +
 .../spark/unsafe/memory/HeapMemoryAllocator.java   |   113 +
 .../spark/unsafe/memory/MemoryAllocator.java       |    44 +
 .../apache/spark/unsafe/memory/MemoryBlock.java    |    82 +
 .../apache/spark/unsafe/memory/MemoryLocation.java |    54 +
 .../spark/unsafe/memory/UnsafeMemoryAllocator.java |    57 +
 .../org/apache/spark/unsafe/types/ByteArray.java   |   269 +
 .../spark/unsafe/types/CalendarInterval.java       |   152 +
 .../apache/spark/unsafe/types/GeographyVal.java    |    53 +
 .../org/apache/spark/unsafe/types/GeometryVal.java |    53 +
 .../org/apache/spark/unsafe/types/UTF8String.java  |  2244 ++
 .../org/apache/spark/unsafe/types/VariantVal.java  |   136 +
 .../org/apache/spark/unsafe/PlatformUtilSuite.java |   167 +
 .../apache/spark/unsafe/array/ByteArraySuite.java  |   125 +
 .../apache/spark/unsafe/array/LongArraySuite.java  |    42 +
 .../spark/unsafe/hash/Murmur3_x86_32Suite.java     |   139 +
 .../spark/unsafe/types/CalendarIntervalSuite.java  |   101 +
 .../spark/unsafe/types/CollationSupportSuite.java  |  3894 +++
 .../spark/unsafe/types/GeographyValSuite.java      |    57 +
 .../spark/unsafe/types/GeometryValSuite.java       |    57 +
 .../apache/spark/unsafe/types/UTF8StringSuite.java |  1389 +
 .../spark/unsafe/types/CollationFactorySuite.scala |   553 +
 .../types/UTF8StringPropertyCheckSuite.scala       |   253 +
 common/utils-java/pom.xml                          |    84 +
 .../main/java/org/apache/spark/QueryContext.java   |    57 +
 .../java/org/apache/spark/QueryContextType.java    |    31 +
 .../spark/api/java/function/CoGroupFunction.java   |    30 +
 .../api/java/function/DoubleFlatMapFunction.java   |    29 +
 .../spark/api/java/function/DoubleFunction.java    |    28 +
 .../spark/api/java/function/FilterFunction.java    |    30 +
 .../spark/api/java/function/FlatMapFunction.java   |    29 +
 .../spark/api/java/function/FlatMapFunction2.java  |    29 +
 .../api/java/function/FlatMapGroupsFunction.java   |    29 +
 .../spark/api/java/function/ForeachFunction.java   |    30 +
 .../java/function/ForeachPartitionFunction.java    |    29 +
 .../apache/spark/api/java/function/Function.java   |    30 +
 .../apache/spark/api/java/function/Function0.java  |    28 +
 .../apache/spark/api/java/function/Function2.java  |    28 +
 .../apache/spark/api/java/function/Function3.java  |    28 +
 .../apache/spark/api/java/function/Function4.java  |    28 +
 .../spark/api/java/function/MapFunction.java       |    28 +
 .../spark/api/java/function/MapGroupsFunction.java |    29 +
 .../api/java/function/MapPartitionsFunction.java   |    29 +
 .../api/java/function/PairFlatMapFunction.java     |    32 +
 .../spark/api/java/function/PairFunction.java      |    31 +
 .../spark/api/java/function/ReduceFunction.java    |    28 +
 .../spark/api/java/function/VoidFunction.java      |    28 +
 .../spark/api/java/function/VoidFunction2.java     |    28 +
 .../spark/api/java/function/package-info.java      |    23 +
 .../java/org/apache/spark/internal/LogKey.java     |    52 +
 .../java/org/apache/spark/internal/LogKeys.java    |   897 +
 .../main/java/org/apache/spark/internal/MDC.java   |    29 +
 .../org/apache/spark/internal/SparkLogger.java     |   249 +
 .../apache/spark/internal/SparkLoggerFactory.java  |    47 +
 .../java/org/apache/spark/memory/MemoryMode.java   |    26 +
 .../org/apache/spark/network/util/ByteUnit.java    |    67 +
 .../org/apache/spark/network/util/JavaUtils.java   |   770 +
 .../apache/spark/unsafe/array/ByteArrayUtils.java  |    27 +
 .../src/main/java/org/apache/spark/util/Pair.java  |    29 +
 .../org/apache/spark/util/UUIDv7Generator.java     |    67 +
 .../resources/org/apache/spark/SparkLayout.json    |    61 +
 .../org/apache/spark/log4j2-defaults.properties    |    55 +
 .../org/apache/spark/log4j2-json-layout.properties |    55 +
 .../java/org/apache/spark/util/CustomLogKeys.java  |    24 +
 .../apache/spark/util/PatternSparkLoggerSuite.java |    96 +
 .../apache/spark/util/SparkLoggerSuiteBase.java    |   246 +
 .../spark/util/StructuredSparkLoggerSuite.java     |   191 +
 .../apache/spark/util/UUIDv7GeneratorSuite.java    |    73 +
 .../src/test/resources/log4j2.properties           |    60 +
 common/utils/pom.xml                               |   178 +
 .../main/java/org/apache/spark/SparkThrowable.java |   102 +
 .../apache/spark/storage/StorageLevelMapper.java   |    47 +
 common/utils/src/main/resources/error/README.md    |   175 +
 .../src/main/resources/error/error-classes.json    |    90 +
 .../src/main/resources/error/error-conditions.json | 10583 +++++++
 .../src/main/resources/error/error-states.json     |  7563 +++++
 .../org/apache/spark/ErrorClassesJSONReader.scala  |   272 +
 .../scala/org/apache/spark/SparkBuildInfo.scala    |    65 +
 .../scala/org/apache/spark/SparkException.scala    |   828 +
 .../org/apache/spark/SparkThrowableHelper.scala    |   194 +
 .../scala/org/apache/spark/StringSubstitutor.scala |   126 +
 .../apache/spark/api/java/function/package.scala   |    25 +
 .../scala/org/apache/spark/internal/Logging.scala  |   694 +
 .../spark/internal/config/ConfigBuilder.scala      |   372 +
 .../apache/spark/internal/config/ConfigEntry.scala |   288 +
 .../spark/internal/config/ConfigProvider.scala     |    47 +
 .../spark/internal/config/ConfigReader.scala       |   123 +
 .../spark/scheduler/SparkListenerEvent.scala       |    28 +
 .../sql/streaming/StreamingQueryException.scala    |   110 +
 .../org/apache/spark/storage/StorageLevel.scala    |   231 +
 .../org/apache/spark/util/ArrayImplicits.scala     |    35 +
 .../org/apache/spark/util/ClosureCleaner.scala     |  1104 +
 .../org/apache/spark/util/CompletionIterator.scala |    50 +
 .../scala/org/apache/spark/util/JsonUtils.scala    |    44 +
 .../org/apache/spark/util/LexicalThreadLocal.scala |    66 +
 .../scala/org/apache/spark/util/LogUtils.scala     |    50 +
 .../scala/org/apache/spark/util/MavenUtils.scala   |   660 +
 .../org/apache/spark/util/SparkClassUtils.scala    |   169 +
 .../apache/spark/util/SparkCollectionUtils.scala   |    70 +
 .../org/apache/spark/util/SparkEnvUtils.scala      |    30 +
 .../org/apache/spark/util/SparkErrorUtils.scala    |   157 +
 .../apache/spark/util/SparkFatalException.scala    |    28 +
 .../org/apache/spark/util/SparkFileUtils.scala     |   243 +
 .../org/apache/spark/util/SparkSchemaUtils.scala   |    36 +
 .../org/apache/spark/util/SparkSerDeUtils.scala    |    54 +
 .../org/apache/spark/util/SparkStreamUtils.scala   |   118 +
 .../org/apache/spark/util/SparkStringUtils.scala   |   146 +
 .../org/apache/spark/util/SparkSystemUtils.scala   |    79 +
 .../org/apache/spark/util/SparkTestUtils.scala     |   104 +
 .../org/apache/spark/util/SparkThreadUtils.scala   |    64 +
 .../scala/org/apache/spark/util/VersionUtils.scala |    99 +
 common/utils/src/test/resources/log4j2.properties  |    60 +
 .../scala/org/apache/spark/util/IvyTestUtils.scala |   416 +
 .../scala/org/apache/spark/util/LogKeysSuite.scala |    90 +
 .../org/apache/spark/util/LogThrottlingSuite.scala |   234 +
 .../scala/org/apache/spark/util/MDCSuite.scala     |    87 +
 .../org/apache/spark/util/MavenUtilsSuite.scala    |   312 +
 .../scala/org/apache/spark/util/MaybeNull.scala    |    36 +
 .../apache/spark/util/PatternLoggingSuite.scala    |    69 +
 .../apache/spark/util/SparkErrorUtilsSuite.scala   |   120 +
 .../apache/spark/util/StructuredLoggingSuite.scala |   493 +
 .../org/apache/spark/util/VersionUtilsSuite.scala  |   115 +
 common/variant/README.md                           |     1 +
 common/variant/pom.xml                             |    75 +
 .../apache/spark/types/variant/ShreddingUtils.java |   173 +
 .../org/apache/spark/types/variant/Variant.java    |   364 +
 .../apache/spark/types/variant/VariantBuilder.java |   586 +
 .../apache/spark/types/variant/VariantSchema.java  |   162 +
 .../types/variant/VariantShreddingWriter.java      |   303 +
 .../types/variant/VariantSizeLimitException.java   |    25 +
 .../apache/spark/types/variant/VariantUtil.java    |   611 +
 conf/fairscheduler.xml.template                    |    31 +
 conf/log4j2-json-layout.properties.template        |    56 +
 conf/log4j2.properties.template                    |    62 +
 conf/metrics.properties.template                   |   212 +
 conf/spark-defaults.conf.template                  |    27 +
 conf/spark-env.sh.template                         |    80 +
 conf/workers.template                              |    19 +
 .../benchmarks/AvroReadBenchmark-jdk21-results.txt |   141 +
 .../avro/benchmarks/AvroReadBenchmark-results.txt  |   141 +
 .../AvroWriteBenchmark-jdk21-results.txt           |    56 +
 .../avro/benchmarks/AvroWriteBenchmark-results.txt |    56 +
 connector/avro/pom.xml                             |    86 +
 ...org.apache.spark.sql.sources.DataSourceRegister |    18 +
 .../apache/spark/sql/avro/AvroDataToCatalyst.scala |   150 +
 .../spark/sql/avro/AvroExpressionEvalUtils.scala   |    45 +
 .../apache/spark/sql/avro/CatalystDataToAvro.scala |    70 +
 .../org/apache/spark/sql/avro/SchemaOfAvro.scala   |    70 +
 .../scala/org/apache/spark/sql/avro/package.scala  |    47 +
 .../spark/sql/v2/avro/AvroDataSourceV2.scala       |    45 +
 .../sql/v2/avro/AvroPartitionReaderFactory.scala   |   122 +
 .../org/apache/spark/sql/v2/avro/AvroScan.scala    |    76 +
 .../apache/spark/sql/v2/avro/AvroScanBuilder.scala |    55 +
 .../org/apache/spark/sql/v2/avro/AvroTable.scala   |    55 +
 .../org/apache/spark/sql/v2/avro/AvroWrite.scala   |    40 +
 .../spark/sql/avro/JavaAvroFunctionsSuite.java     |    66 +
 .../test/resources/before_1582_date_v2_4_5.avro    |   Bin 0 -> 200 bytes
 .../test/resources/before_1582_date_v2_4_6.avro    |   Bin 0 -> 231 bytes
 .../test/resources/before_1582_date_v3_2_0.avro    |   Bin 0 -> 264 bytes
 .../before_1582_timestamp_micros_v2_4_5.avro       |   Bin 0 -> 218 bytes
 .../before_1582_timestamp_micros_v2_4_6.avro       |   Bin 0 -> 249 bytes
 .../before_1582_timestamp_micros_v3_2_0.avro       |   Bin 0 -> 282 bytes
 .../before_1582_timestamp_millis_v2_4_5.avro       |   Bin 0 -> 244 bytes
 .../before_1582_timestamp_millis_v2_4_6.avro       |   Bin 0 -> 275 bytes
 .../before_1582_timestamp_millis_v3_2_0.avro       |   Bin 0 -> 308 bytes
 .../avro/src/test/resources/empty_blocks.avro      |     5 +
 connector/avro/src/test/resources/empty_file.avro  |   Bin 0 -> 508 bytes
 connector/avro/src/test/resources/episodes.avro    |   Bin 0 -> 597 bytes
 .../avro/src/test/resources/log4j2.properties      |    31 +
 .../test-random-partitioned/part-r-00000.avro      |   Bin 0 -> 1768 bytes
 .../test-random-partitioned/part-r-00001.avro      |   Bin 0 -> 2313 bytes
 .../test-random-partitioned/part-r-00002.avro      |   Bin 0 -> 1621 bytes
 .../test-random-partitioned/part-r-00003.avro      |   Bin 0 -> 2117 bytes
 .../test-random-partitioned/part-r-00004.avro      |   Bin 0 -> 3282 bytes
 .../test-random-partitioned/part-r-00005.avro      |   Bin 0 -> 1550 bytes
 .../test-random-partitioned/part-r-00006.avro      |   Bin 0 -> 1729 bytes
 .../test-random-partitioned/part-r-00007.avro      |   Bin 0 -> 1897 bytes
 .../test-random-partitioned/part-r-00008.avro      |   Bin 0 -> 3420 bytes
 .../test-random-partitioned/part-r-00009.avro      |   Bin 0 -> 1796 bytes
 .../test-random-partitioned/part-r-00010.avro      |   Bin 0 -> 3872 bytes
 connector/avro/src/test/resources/test.avro        |   Bin 0 -> 1365 bytes
 connector/avro/src/test/resources/test.avsc        |    53 +
 connector/avro/src/test/resources/test.json        |    42 +
 connector/avro/src/test/resources/test_sub.avsc    |     9 +
 .../sql/avro/AvroCatalystDataConversionSuite.scala |   389 +
 .../org/apache/spark/sql/avro/AvroCodecSuite.scala |    78 +
 .../apache/spark/sql/avro/AvroFunctionsSuite.scala |   746 +
 .../spark/sql/avro/AvroLogicalTypeInitSuite.scala  |    76 +
 .../spark/sql/avro/AvroLogicalTypeSuite.scala      |   557 +
 .../apache/spark/sql/avro/AvroRowReaderSuite.scala |   102 +
 .../org/apache/spark/sql/avro/AvroScanSuite.scala  |    30 +
 .../spark/sql/avro/AvroSchemaHelperSuite.scala     |   159 +
 .../org/apache/spark/sql/avro/AvroSerdeSuite.scala |   256 +
 .../org/apache/spark/sql/avro/AvroSuite.scala      |  3510 +++
 .../sql/avro/DeprecatedAvroFunctionsSuite.scala    |   123 +
 .../execution/benchmark/AvroReadBenchmark.scala    |   308 +
 .../execution/benchmark/AvroWriteBenchmark.scala   |   107 +
 .../datasources/AvroReadSchemaSuite.scala          |    29 +
 connector/docker-integration-tests/README.md       |   132 +
 connector/docker-integration-tests/pom.xml         |   129 +
 .../src/test/resources/db2-krb-setup.sh            |    28 +
 .../src/test/resources/log4j2.properties           |    41 +
 .../test/resources/mariadb-docker-entrypoint.sh    |    26 +
 .../src/test/resources/mariadb-krb-setup.sh        |    20 +
 .../src/test/resources/postgres-krb-setup.sh       |    21 +
 .../spark/sql/jdbc/DB2DatabaseOnDocker.scala       |    35 +
 .../spark/sql/jdbc/DB2IntegrationSuite.scala       |   283 +
 .../spark/sql/jdbc/DB2KrbIntegrationSuite.scala    |    89 +
 .../spark/sql/jdbc/DockerIntegrationFunSuite.scala |    54 +
 .../sql/jdbc/DockerJDBCIntegrationSuite.scala      |   291 +
 .../sql/jdbc/DockerKrbJDBCIntegrationSuite.scala   |   201 +
 .../spark/sql/jdbc/MariaDBDatabaseOnDocker.scala   |    33 +
 .../sql/jdbc/MariaDBKrbIntegrationSuite.scala      |    65 +
 .../sql/jdbc/MsSQLServerDatabaseOnDocker.scala     |    33 +
 .../sql/jdbc/MsSqlServerIntegrationSuite.scala     |   496 +
 .../spark/sql/jdbc/MySQLDatabaseOnDocker.scala     |    31 +
 .../spark/sql/jdbc/MySQLIntegrationSuite.scala     |   388 +
 .../spark/sql/jdbc/OracleDatabaseOnDocker.scala    |    39 +
 .../spark/sql/jdbc/OracleIntegrationSuite.scala    |   617 +
 .../spark/sql/jdbc/PostgresDatabaseOnDocker.scala  |    36 +
 .../spark/sql/jdbc/PostgresIntegrationSuite.scala  |   623 +
 .../sql/jdbc/PostgresKrbIntegrationSuite.scala     |    61 +
 .../sql/jdbc/SharedJDBCIntegrationSuite.scala      |    89 +
 .../jdbc/querytest/CrossDbmsQueryTestSuite.scala   |   184 +
 .../jdbc/querytest/GeneratedSubquerySuite.scala    |   487 +
 .../jdbc/querytest/PostgresSQLQueryTestSuite.scala |    64 +
 .../spark/sql/jdbc/v2/DB2IntegrationSuite.scala    |   125 +
 .../spark/sql/jdbc/v2/DB2NamespaceSuite.scala      |    56 +
 .../sql/jdbc/v2/DockerJDBCIntegrationV2Suite.scala |    70 +
 .../sql/jdbc/v2/MsSqlServerIntegrationSuite.scala  |   294 +
 .../sql/jdbc/v2/MsSqlServerNamespaceSuite.scala    |    60 +
 .../spark/sql/jdbc/v2/MySQLIntegrationSuite.scala  |   346 +
 .../spark/sql/jdbc/v2/MySQLNamespaceSuite.scala    |    99 +
 .../spark/sql/jdbc/v2/OracleIntegrationSuite.scala |   344 +
 .../spark/sql/jdbc/v2/OracleNamespaceSuite.scala   |    79 +
 .../sql/jdbc/v2/PostgresIntegrationSuite.scala     |   416 +
 .../spark/sql/jdbc/v2/PostgresNamespaceSuite.scala |    49 +
 .../spark/sql/jdbc/v2/V2JDBCNamespaceTest.scala    |   133 +
 .../org/apache/spark/sql/jdbc/v2/V2JDBCTest.scala  |  1285 +
 .../MsSqlServerJoinPushdownIntegrationSuite.scala  |    51 +
 .../join/MySQLJoinPushdownIntegrationSuite.scala   |    52 +
 .../join/OracleJoinPushdownIntegrationSuite.scala  |   123 +
 .../PostgresJoinPushdownIntegrationSuite.scala     |    47 +
 .../scala/org/apache/spark/util/DockerUtils.scala  |    68 +
 connector/kafka-0-10-assembly/pom.xml              |   186 +
 connector/kafka-0-10-sql/pom.xml                   |   191 +
 ...org.apache.spark.sql.sources.DataSourceRegister |    18 +
 .../resources/error/kafka-error-conditions.json    |   124 +
 .../spark/sql/kafka010/ConsumerStrategy.scala      |   149 +
 .../org/apache/spark/sql/kafka010/JsonUtils.scala  |   116 +
 .../org/apache/spark/sql/kafka010/KafkaBatch.scala |    78 +
 .../sql/kafka010/KafkaBatchPartitionReader.scala   |   183 +
 .../spark/sql/kafka010/KafkaBatchWrite.scala       |    68 +
 .../spark/sql/kafka010/KafkaContinuousStream.scala |   254 +
 .../spark/sql/kafka010/KafkaDataWriter.scala       |    74 +
 .../spark/sql/kafka010/KafkaExceptions.scala       |   279 +
 .../spark/sql/kafka010/KafkaMicroBatchStream.scala |   553 +
 .../sql/kafka010/KafkaOffsetRangeCalculator.scala  |   173 +
 .../spark/sql/kafka010/KafkaOffsetRangeLimit.scala |    71 +
 .../spark/sql/kafka010/KafkaOffsetReader.scala     |   256 +
 .../sql/kafka010/KafkaOffsetReaderAdmin.scala      |   497 +
 .../sql/kafka010/KafkaOffsetReaderConsumer.scala   |   580 +
 .../sql/kafka010/KafkaRecordToRowConverter.scala   |    93 +
 .../apache/spark/sql/kafka010/KafkaRelation.scala  |    93 +
 .../org/apache/spark/sql/kafka010/KafkaSink.scala  |    43 +
 .../apache/spark/sql/kafka010/KafkaSource.scala    |   432 +
 .../kafka010/KafkaSourceInitialOffsetWriter.scala  |    63 +
 .../spark/sql/kafka010/KafkaSourceOffset.scala     |    80 +
 .../spark/sql/kafka010/KafkaSourceProvider.scala   |   800 +
 .../apache/spark/sql/kafka010/KafkaSourceRDD.scala |   154 +
 .../spark/sql/kafka010/KafkaStreamingWrite.scala   |    75 +
 .../org/apache/spark/sql/kafka010/KafkaWrite.scala |    41 +
 .../apache/spark/sql/kafka010/KafkaWriteTask.scala |   137 +
 .../apache/spark/sql/kafka010/KafkaWriter.scala    |   127 +
 .../sql/kafka010/consumer/FetchedDataPool.scala    |   186 +
 .../consumer/InternalKafkaConsumerPool.scala       |   229 +
 .../sql/kafka010/consumer/KafkaDataConsumer.scala  |   855 +
 .../apache/spark/sql/kafka010/package-info.java    |    21 +
 .../org/apache/spark/sql/kafka010/package.scala    |    92 +
 .../kafka010/producer/CachedKafkaProducer.scala    |    42 +
 .../producer/InternalKafkaProducerPool.scala       |   208 +
 .../kafka-source-initial-offset-future-version.bin |     2 +
 .../kafka-source-initial-offset-version-2.1.0.bin  |     1 +
 .../kafka-source-offset-version-2.1.0.txt          |     1 +
 .../src/test/resources/log4j2.properties           |    37 +
 .../commits/0                                      |     2 +
 .../metadata                                       |     1 +
 .../offsets/0                                      |     3 +
 .../sources/0/0                                    |   Bin 0 -> 93 bytes
 .../state/0/0/1.delta                              |   Bin 0 -> 182 bytes
 .../state/0/1/1.delta                              |   Bin 0 -> 160 bytes
 .../state/0/2/1.delta                              |   Bin 0 -> 256 bytes
 .../state/0/3/1.delta                              |   Bin 0 -> 244 bytes
 .../state/0/4/1.delta                              |   Bin 0 -> 240 bytes
 .../spark/sql/kafka010/ConsumerStrategySuite.scala |   148 +
 .../apache/spark/sql/kafka010/JsonUtilsSuite.scala |    45 +
 .../sql/kafka010/KafkaContinuousSourceSuite.scala  |   244 +
 .../spark/sql/kafka010/KafkaContinuousTest.scala   |    98 +
 .../sql/kafka010/KafkaDelegationTokenSuite.scala   |   118 +
 .../kafka010/KafkaDontFailOnDataLossSuite.scala    |   290 +
 .../sql/kafka010/KafkaMicroBatchSourceSuite.scala  |  2810 ++
 .../kafka010/KafkaOffsetRangeCalculatorSuite.scala |   336 +
 .../sql/kafka010/KafkaOffsetReaderSuite.scala      |   266 +
 .../kafka010/KafkaRealTimeIntegrationSuite.scala   |   293 +
 .../sql/kafka010/KafkaRealTimeModeSuite.scala      |   776 +
 .../spark/sql/kafka010/KafkaRelationSuite.scala    |   772 +
 .../apache/spark/sql/kafka010/KafkaSinkSuite.scala |   605 +
 .../sql/kafka010/KafkaSourceOffsetSuite.scala      |   106 +
 .../sql/kafka010/KafkaSourceProviderSuite.scala    |   136 +
 .../spark/sql/kafka010/KafkaSparkConfSuite.scala   |    30 +
 .../org/apache/spark/sql/kafka010/KafkaTest.scala  |    34 +
 .../apache/spark/sql/kafka010/KafkaTestUtils.scala |   668 +
 .../apache/spark/sql/kafka010/RecordBuilder.scala  |    74 +
 .../kafka010/consumer/FetchedDataPoolSuite.scala   |   305 +
 .../consumer/InternalKafkaConsumerPoolSuite.scala  |   286 +
 .../kafka010/consumer/KafkaDataConsumerSuite.scala |   394 +
 .../producer/InternalKafkaProducerPoolSuite.scala  |   192 +
 connector/kafka-0-10-token-provider/pom.xml        |   109 +
 ...he.spark.security.HadoopDelegationTokenProvider |    18 +
 .../kafka-token-provider-error-conditions.json     |     8 +
 .../apache/spark/kafka010/KafkaConfigUpdater.scala |    95 +
 .../kafka010/KafkaDelegationTokenProvider.scala    |    87 +
 .../apache/spark/kafka010/KafkaRedactionUtil.scala |    52 +
 .../kafka010/KafkaTokenProviderException.scala     |    45 +
 .../spark/kafka010/KafkaTokenSparkConf.scala       |   110 +
 .../org/apache/spark/kafka010/KafkaTokenUtil.scala |   292 +
 .../src/test/resources/log4j2.properties           |    31 +
 .../spark/kafka010/KafkaConfigUpdaterSuite.scala   |   176 +
 .../spark/kafka010/KafkaDelegationTokenTest.scala  |   139 +
 .../KafkaHadoopDelegationTokenManagerSuite.scala   |    32 +
 .../spark/kafka010/KafkaRedactionUtilSuite.scala   |   125 +
 .../spark/kafka010/KafkaTokenSparkConfSuite.scala  |   147 +
 .../spark/kafka010/KafkaTokenUtilSuite.scala       |   292 +
 connector/kafka-0-10/pom.xml                       |   158 +
 .../streaming/kafka010/ConsumerStrategy.scala      |   471 +
 .../kafka010/DirectKafkaInputDStream.scala         |   352 +
 .../streaming/kafka010/KafkaDataConsumer.scala     |   371 +
 .../apache/spark/streaming/kafka010/KafkaRDD.scala |   315 +
 .../streaming/kafka010/KafkaRDDPartition.scala     |    45 +
 .../spark/streaming/kafka010/KafkaUtils.scala      |   214 +
 .../streaming/kafka010/LocationStrategy.scala      |    71 +
 .../spark/streaming/kafka010/OffsetRange.scala     |   145 +
 .../streaming/kafka010/PerPartitionConfig.scala    |    47 +
 .../spark/streaming/kafka010/package-info.java     |    21 +
 .../apache/spark/streaming/kafka010/package.scala  |    76 +
 .../kafka010/JavaConsumerStrategySuite.java        |    94 +
 .../kafka010/JavaDirectKafkaStreamSuite.java       |   155 +
 .../streaming/kafka010/JavaKafkaRDDSuite.java      |   121 +
 .../kafka010/JavaLocationStrategySuite.java        |    58 +
 .../src/test/resources/log4j2.properties           |    31 +
 .../kafka010/DirectKafkaStreamSuite.scala          |   815 +
 .../kafka010/KafkaDataConsumerSuite.scala          |   162 +
 .../spark/streaming/kafka010/KafkaRDDSuite.scala   |   308 +
 .../spark/streaming/kafka010/KafkaTestUtils.scala  |   348 +
 .../streaming/kafka010/mocks/MockScheduler.scala   |    71 +
 .../spark/streaming/kafka010/mocks/MockTime.scala  |    58 +
 connector/kinesis-asl-assembly/pom.xml             |   202 +
 connector/kinesis-asl/pom.xml                      |   180 +
 .../streaming/JavaKinesisWordCountASL.java         |   207 +
 .../streaming/kinesis/KinesisInitialPositions.java |    93 +
 .../examples/streaming/kinesis_wordcount_asl.py    |    82 +
 .../src/main/resources/log4j2.properties           |    43 +
 .../examples/streaming/KinesisExampleUtils.scala   |    35 +
 .../examples/streaming/KinesisWordCountASL.scala   |   300 +
 .../streaming/kinesis/KinesisBackedBlockRDD.scala  |   314 +
 .../streaming/kinesis/KinesisCheckpointer.scala    |   146 +
 .../streaming/kinesis/KinesisInputDStream.scala    |   371 +
 .../kinesis/KinesisReadConfigurations.scala        |    78 +
 .../spark/streaming/kinesis/KinesisReceiver.scala  |   393 +
 .../streaming/kinesis/KinesisRecordProcessor.scala |   201 +
 .../kinesis/KinesisUtilsPythonHelper.scala         |   103 +
 .../streaming/kinesis/SparkAWSCredentials.scala    |   189 +
 .../JavaKinesisInputDStreamBuilderSuite.java       |    99 +
 .../src/test/resources/log4j2.properties           |    31 +
 .../kinesis/KPLBasedKinesisTestUtils.scala         |    79 +
 .../kinesis/KinesisBackedBlockRDDSuite.scala       |   260 +
 .../kinesis/KinesisCheckpointerSuite.scala         |   154 +
 .../spark/streaming/kinesis/KinesisFunSuite.scala  |    46 +
 .../kinesis/KinesisInputDStreamBuilderSuite.scala  |   190 +
 .../streaming/kinesis/KinesisReceiverSuite.scala   |   259 +
 .../streaming/kinesis/KinesisStreamSuite.scala     |   434 +
 .../spark/streaming/kinesis/KinesisTestUtils.scala |   312 +
 .../kinesis/SparkAWSCredentialsBuilderSuite.scala  |   100 +
 connector/profiler/README.md                       |   134 +
 connector/profiler/pom.xml                         |    54 +
 .../org/apache/spark/profiler/ProfilerPlugin.scala |    97 +
 .../apache/spark/profiler/SparkAsyncProfiler.scala |   218 +
 .../scala/org/apache/spark/profiler/package.scala  |    78 +
 connector/protobuf/README.md                       |    36 +
 connector/protobuf/pom.xml                         |   211 +
 .../sql/protobuf/CatalystDataToProtobuf.scala      |    89 +
 .../sql/protobuf/ProtobufDataToCatalyst.scala      |   176 +
 .../spark/sql/protobuf/ProtobufDeserializer.scala  |   526 +
 .../spark/sql/protobuf/ProtobufSerializer.scala    |   342 +
 .../org/apache/spark/sql/protobuf/package.scala    |    20 +
 .../spark/sql/protobuf/utils/ProtobufOptions.scala |   270 +
 .../spark/sql/protobuf/utils/ProtobufUtils.scala   |   304 +
 .../sql/protobuf/utils/SchemaConverters.scala      |   243 +
 .../protobuf/src/test/resources/log4j2.properties  |    39 +
 .../src/test/resources/protobuf/basicmessage.proto |    40 +
 .../test/resources/protobuf/catalyst_types.proto   |    82 +
 .../src/test/resources/protobuf/duration.proto     |    26 +
 .../test/resources/protobuf/functions_suite.proto  |   352 +
 .../src/test/resources/protobuf/nestedenum.proto   |    28 +
 .../test/resources/protobuf/proto2_messages.proto  |    57 +
 .../src/test/resources/protobuf/pyspark_test.proto |    32 +
 .../src/test/resources/protobuf/serde_suite.proto  |    65 +
 .../src/test/resources/protobuf/timestamp.proto    |    26 +
 .../ProtobufCatalystDataConversionSuite.scala      |   438 +
 .../sql/protobuf/ProtobufFunctionsSuite.scala      |  2282 ++
 .../spark/sql/protobuf/ProtobufSerdeSuite.scala    |   329 +
 .../spark/sql/protobuf/ProtobufTestBase.scala      |    76 +
 connector/spark-ganglia-lgpl/pom.xml               |    48 +
 .../codahale/metrics/ganglia/GangliaReporter.java  |   433 +
 .../apache/spark/metrics/sink/GangliaSink.scala    |    95 +
 .../benchmarks/ChecksumBenchmark-jdk21-results.txt |    13 +
 core/benchmarks/ChecksumBenchmark-results.txt      |    13 +
 .../CoalescedRDDBenchmark-jdk21-results.txt        |    40 +
 core/benchmarks/CoalescedRDDBenchmark-results.txt  |    40 +
 core/benchmarks/KryoBenchmark-jdk21-results.txt    |    28 +
 core/benchmarks/KryoBenchmark-results.txt          |    28 +
 .../KryoIteratorBenchmark-jdk21-results.txt        |    28 +
 core/benchmarks/KryoIteratorBenchmark-results.txt  |    28 +
 .../KryoSerializerBenchmark-jdk21-results.txt      |    12 +
 .../benchmarks/KryoSerializerBenchmark-results.txt |    12 +
 .../LZ4TPCDSDataBenchmark-jdk21-results.txt        |    17 +
 core/benchmarks/LZ4TPCDSDataBenchmark-results.txt  |    17 +
 core/benchmarks/LZFBenchmark-jdk21-results.txt     |    19 +
 core/benchmarks/LZFBenchmark-results.txt           |    19 +
 .../MapStatusesConvertBenchmark-jdk21-results.txt  |    13 +
 .../MapStatusesConvertBenchmark-results.txt        |    13 +
 .../MapStatusesSerDeserBenchmark-jdk21-results.txt |    66 +
 .../MapStatusesSerDeserBenchmark-results.txt       |    66 +
 .../PercentileHeapBenchmark-jdk21-results.txt      |    41 +
 .../benchmarks/PercentileHeapBenchmark-results.txt |    41 +
 .../PersistenceEngineBenchmark-jdk21-results.txt   |    18 +
 .../PersistenceEngineBenchmark-results.txt         |    18 +
 .../PropertiesCloneBenchmark-jdk21-results.txt     |    40 +
 .../PropertiesCloneBenchmark-results.txt           |    40 +
 core/benchmarks/RocksDBBenchmark-jdk21-results.txt |   102 +
 core/benchmarks/RocksDBBenchmark-results.txt       |   102 +
 core/benchmarks/SorterBenchmark-jdk21-results.txt  |    29 +
 core/benchmarks/SorterBenchmark-results.txt        |    29 +
 .../XORShiftRandomBenchmark-jdk21-results.txt      |    44 +
 .../benchmarks/XORShiftRandomBenchmark-results.txt |    44 +
 .../ZStandardBenchmark-jdk21-results.txt           |    82 +
 core/benchmarks/ZStandardBenchmark-results.txt     |    82 +
 .../ZStandardTPCDSDataBenchmark-jdk21-results.txt  |    49 +
 .../ZStandardTPCDSDataBenchmark-results.txt        |    49 +
 core/pom.xml                                       |   736 +
 .../java/org/apache/spark/JobExecutionStatus.java  |    31 +
 .../java/org/apache/spark/SparkExecutorInfo.java   |    37 +
 .../org/apache/spark/SparkFirehoseListener.java    |   224 +
 .../main/java/org/apache/spark/SparkJobInfo.java   |    32 +
 .../main/java/org/apache/spark/SparkStageInfo.java |    37 +
 .../apache/spark/api/java/JavaFutureAction.java    |    33 +
 .../java/org/apache/spark/api/java/Optional.java   |   183 +
 .../org/apache/spark/api/java/StorageLevels.java   |    56 +
 .../org/apache/spark/api/plugin/DriverPlugin.java  |   111 +
 .../apache/spark/api/plugin/ExecutorPlugin.java    |    99 +
 .../org/apache/spark/api/plugin/PluginContext.java |    89 +
 .../org/apache/spark/api/plugin/SparkPlugin.java   |    53 +
 .../api/resource/ResourceDiscoveryPlugin.java      |    63 +
 .../spark/io/NioBufferedFileInputStream.java       |   163 +
 .../org/apache/spark/io/ReadAheadInputStream.java  |   406 +
 .../org/apache/spark/memory/MemoryConsumer.java    |   160 +
 .../apache/spark/memory/SparkOutOfMemoryError.java |    62 +
 .../org/apache/spark/memory/TaskMemoryManager.java |   586 +
 .../apache/spark/memory/TooLargePageException.java |    24 +
 .../main/java/org/apache/spark/package-info.java   |    23 +
 .../spark/serializer/DummySerializerInstance.java  |    92 +
 .../apache/spark/shuffle/api/ShuffleDataIO.java    |    55 +
 .../spark/shuffle/api/ShuffleDriverComponents.java |    73 +
 .../shuffle/api/ShuffleExecutorComponents.java     |    76 +
 .../spark/shuffle/api/ShuffleMapOutputWriter.java  |    88 +
 .../spark/shuffle/api/ShufflePartitionWriter.java  |    98 +
 .../api/SingleSpillShuffleMapOutputWriter.java     |    39 +
 .../shuffle/api/WritableByteChannelWrapper.java    |    41 +
 .../api/metadata/MapOutputCommitMessage.java       |    63 +
 .../shuffle/api/metadata/MapOutputMetadata.java    |    30 +
 .../spark/shuffle/checksum/RowBasedChecksum.scala  |    80 +
 .../shuffle/checksum/ShuffleChecksumSupport.java   |    45 +
 .../shuffle/sort/BypassMergeSortShuffleWriter.java |   331 +
 .../spark/shuffle/sort/PackedRecordPointer.java    |   102 +
 .../spark/shuffle/sort/ShuffleExternalSorter.java  |   480 +
 .../spark/shuffle/sort/ShuffleInMemorySorter.java  |   201 +
 .../spark/shuffle/sort/ShuffleSortDataFormat.java  |    78 +
 .../org/apache/spark/shuffle/sort/SpillInfo.java   |    37 +
 .../spark/shuffle/sort/UnsafeShuffleWriter.java    |   568 +
 .../shuffle/sort/io/LocalDiskShuffleDataIO.java    |    46 +
 .../sort/io/LocalDiskShuffleDriverComponents.java  |    49 +
 .../io/LocalDiskShuffleExecutorComponents.java     |    89 +
 .../sort/io/LocalDiskShuffleMapOutputWriter.java   |   281 +
 .../io/LocalDiskSingleSpillMapOutputWriter.java    |    57 +
 .../spark/status/api/v1/ApplicationStatus.java     |    30 +
 .../apache/spark/status/api/v1/StageStatus.java    |    32 +
 .../apache/spark/status/api/v1/TaskSorting.java    |    48 +
 .../org/apache/spark/status/api/v1/TaskStatus.java |    32 +
 .../spark/storage/TimeTrackingOutputStream.java    |    76 +
 .../apache/spark/unsafe/map/BytesToBytesMap.java   |  1011 +
 .../spark/unsafe/map/HashMapGrowthStrategy.java    |    47 +
 .../org/apache/spark/util/BestEffortLazyVal.java   |    77 +
 .../spark/util/ChildFirstURLClassLoader.java       |    77 +
 .../main/java/org/apache/spark/util/EnumUtil.java  |    47 +
 .../util/ExposedBufferByteArrayOutputStream.java   |    26 +
 .../apache/spark/util/MutableURLClassLoader.java   |    40 +
 .../org/apache/spark/util/ParentClassLoader.java   |    42 +
 .../spark/util/TransientBestEffortLazyVal.java     |    84 +
 .../org/apache/spark/util/collection/TimSort.java  |   949 +
 .../collection/unsafe/sort/PrefixComparator.java   |    29 +
 .../collection/unsafe/sort/PrefixComparators.java  |   182 +
 .../util/collection/unsafe/sort/RadixSort.java     |   260 +
 .../collection/unsafe/sort/RecordComparator.java   |    39 +
 .../unsafe/sort/RecordPointerAndKeyPrefix.java     |    31 +
 .../unsafe/sort/UnsafeExternalSorter.java          |   896 +
 .../unsafe/sort/UnsafeInMemorySorter.java          |   390 +
 .../unsafe/sort/UnsafeSortDataFormat.java          |    92 +
 .../unsafe/sort/UnsafeSorterIterator.java          |    39 +
 .../unsafe/sort/UnsafeSorterSpillMerger.java       |   107 +
 .../unsafe/sort/UnsafeSorterSpillReader.java       |   169 +
 .../unsafe/sort/UnsafeSorterSpillWriter.java       |   165 +
 core/src/main/protobuf/buf.yaml                    |    23 +
 .../apache/spark/status/protobuf/store_types.proto |   820 +
 core/src/main/resources/META-INF/LICENSE           |   240 +
 core/src/main/resources/META-INF/NOTICE            |    29 +
 ....apache.spark.deploy.history.EventFilterBuilder |    18 +
 ...he.spark.security.HadoopDelegationTokenProvider |    19 +
 .../org.apache.spark.status.protobuf.ProtobufSerDe |    33 +
 .../apache/spark/ui/static/bootstrap.bundle.min.js |     7 +
 .../org/apache/spark/ui/static/bootstrap.min.css   |     7 +
 .../org/apache/spark/ui/static/d3-flamegraph.css   |    47 +
 .../apache/spark/ui/static/d3-flamegraph.min.js    |     2 +
 .../resources/org/apache/spark/ui/static/d3.min.js |     2 +
 .../org/apache/spark/ui/static/dagre-d3.min.js     |  4819 +++
 .../spark/ui/static/dataTables.bootstrap4.min.css  |     1 +
 .../spark/ui/static/dataTables.bootstrap4.min.js   |     4 +
 .../apache/spark/ui/static/dataTables.rowsGroup.js |   224 +
 .../org/apache/spark/ui/static/environmentpage.js  |    62 +
 .../spark/ui/static/executorspage-template.html    |   158 +
 .../org/apache/spark/ui/static/executorspage.js    |   820 +
 .../org/apache/spark/ui/static/flamegraph.js       |    51 +
 .../org/apache/spark/ui/static/graphlib-dot.min.js |   347 +
 .../apache/spark/ui/static/historypage-common.js   |    29 +
 .../spark/ui/static/historypage-template.html      |    80 +
 .../org/apache/spark/ui/static/historypage.js      |   271 +
 .../apache/spark/ui/static/initialize-tooltips.js  |    23 +
 .../apache/spark/ui/static/jquery.blockUI.min.js   |     6 +
 .../spark/ui/static/jquery.cookies.2.2.0.min.js    |    18 +
 .../spark/ui/static/jquery.dataTables.min.css      |     1 +
 .../spark/ui/static/jquery.dataTables.min.js       |     4 +
 .../org/apache/spark/ui/static/jquery.min.js       |     2 +
 .../org/apache/spark/ui/static/jquery.mustache.js  |   682 +
 .../org/apache/spark/ui/static/log-view.js         |   146 +
 .../org/apache/spark/ui/static/package.json        |     5 +
 .../org/apache/spark/ui/static/scroll-button.js    |    48 +
 .../org/apache/spark/ui/static/sorttable.js        |   507 +
 .../org/apache/spark/ui/static/spark-dag-viz.css   |   153 +
 .../org/apache/spark/ui/static/spark-dag-viz.js    |   579 +
 .../spark/ui/static/spark-logo-77x50px-hd.png      |   Bin 0 -> 3077 bytes
 .../org/apache/spark/ui/static/stagepage.js        |  1183 +
 .../spark/ui/static/stagespage-template.html       |   139 +
 .../org/apache/spark/ui/static/streaming-page.css  |    66 +
 .../org/apache/spark/ui/static/streaming-page.js   |   365 +
 .../spark/ui/static/structured-streaming-page.js   |   159 +
 .../resources/org/apache/spark/ui/static/table.js  |   119 +
 .../org/apache/spark/ui/static/timeline-view.css   |   278 +
 .../org/apache/spark/ui/static/timeline-view.js    |   339 +
 .../resources/org/apache/spark/ui/static/utils.js  |   287 +
 .../spark/ui/static/vis-timeline-graph2d.min.css   |     1 +
 .../spark/ui/static/vis-timeline-graph2d.min.js    |    46 +
 .../apache/spark/ui/static/webui-dataTables.css    |    61 +
 .../resources/org/apache/spark/ui/static/webui.css |   501 +
 .../resources/org/apache/spark/ui/static/webui.js  |   113 +
 .../main/scala/org/apache/spark/Aggregator.scala   |    63 +
 .../org/apache/spark/BarrierCoordinator.scala      |   263 +
 .../org/apache/spark/BarrierTaskContext.scala      |   314 +
 .../scala/org/apache/spark/BarrierTaskInfo.scala   |    31 +
 .../org/apache/spark/ContextAwareIterator.scala    |    44 +
 .../scala/org/apache/spark/ContextCleaner.scala    |   324 +
 .../main/scala/org/apache/spark/Dependency.scala   |   290 +
 .../apache/spark/ExecutorAllocationClient.scala    |   156 +
 .../apache/spark/ExecutorAllocationManager.scala   |  1030 +
 .../main/scala/org/apache/spark/FutureAction.scala |   306 +
 .../scala/org/apache/spark/HeartbeatReceiver.scala |   263 +
 .../main/scala/org/apache/spark/Heartbeater.scala  |    63 +
 .../org/apache/spark/InternalAccumulator.scala     |    94 +
 .../org/apache/spark/InterruptibleIterator.scala   |    41 +
 .../scala/org/apache/spark/JobArtifactSet.scala    |   122 +
 .../org/apache/spark/MapOutputStatistics.scala     |    27 +
 .../scala/org/apache/spark/MapOutputTracker.scala  |  1784 ++
 .../main/scala/org/apache/spark/Partition.scala    |    33 +
 .../org/apache/spark/PartitionEvaluator.scala      |    36 +
 .../apache/spark/PartitionEvaluatorFactory.scala   |    38 +
 .../main/scala/org/apache/spark/Partitioner.scala  |   388 +
 .../main/scala/org/apache/spark/SSLOptions.scala   |   358 +
 .../scala/org/apache/spark/SecurityManager.scala   |   457 +
 .../org/apache/spark/SerializableWritable.scala    |    48 +
 .../main/scala/org/apache/spark/SparkConf.scala    |   899 +
 .../main/scala/org/apache/spark/SparkContext.scala |  3608 +++
 .../src/main/scala/org/apache/spark/SparkEnv.scala |   559 +
 .../spark/SparkFileAlreadyExistsException.scala    |    40 +
 .../main/scala/org/apache/spark/SparkFiles.scala   |    43 +
 .../org/apache/spark/SparkStatusTracker.scala      |   134 +
 .../scala/org/apache/spark/StatusAPIImpl.scala     |    46 +
 .../main/scala/org/apache/spark/TaskContext.scala  |   340 +
 .../scala/org/apache/spark/TaskContextImpl.scala   |   346 +
 .../scala/org/apache/spark/TaskEndReason.scala     |   286 +
 .../org/apache/spark/TaskKilledException.scala     |    29 +
 .../spark/TaskNotSerializableException.scala       |    23 +
 .../TaskOutputFileAlreadyExistException.scala      |    23 +
 .../main/scala/org/apache/spark/TaskState.scala    |    31 +
 .../main/scala/org/apache/spark/TestUtils.scala    |   454 +
 .../org/apache/spark/api/java/JavaDoubleRDD.scala  |   289 +
 .../org/apache/spark/api/java/JavaHadoopRDD.scala  |    43 +
 .../apache/spark/api/java/JavaNewHadoopRDD.scala   |    43 +
 .../org/apache/spark/api/java/JavaPairRDD.scala    |  1084 +
 .../scala/org/apache/spark/api/java/JavaRDD.scala  |   264 +
 .../org/apache/spark/api/java/JavaRDDLike.scala    |   759 +
 .../apache/spark/api/java/JavaSparkContext.scala   |   865 +
 .../spark/api/java/JavaSparkStatusTracker.scala    |    72 +
 .../org/apache/spark/api/java/JavaUtils.scala      |   101 +
 .../org/apache/spark/api/java/package-info.java    |    21 +
 .../scala/org/apache/spark/api/java/package.scala  |    23 +
 .../org/apache/spark/api/python/Py4JServer.scala   |    70 +
 .../apache/spark/api/python/PythonErrorUtils.scala |    41 +
 .../spark/api/python/PythonGatewayServer.scala     |    77 +
 .../apache/spark/api/python/PythonHadoopUtil.scala |   189 +
 .../spark/api/python/PythonPartitioner.scala       |    55 +
 .../org/apache/spark/api/python/PythonRDD.scala    |  1021 +
 .../org/apache/spark/api/python/PythonRunner.scala |  1088 +
 .../org/apache/spark/api/python/PythonUtils.scala  |   198 +
 .../spark/api/python/PythonWorkerFactory.scala     |   548 +
 .../spark/api/python/PythonWorkerLogCapture.scala  |   198 +
 .../spark/api/python/PythonWorkerUtils.scala       |   254 +
 .../org/apache/spark/api/python/SerDeUtil.scala    |   223 +
 .../spark/api/python/StreamingPythonRunner.scala   |   183 +
 .../scala/org/apache/spark/api/r/BaseRRunner.scala |   360 +
 .../org/apache/spark/api/r/JVMObjectTracker.scala  |    75 +
 .../scala/org/apache/spark/api/r/RAuthHelper.scala |    39 +
 .../scala/org/apache/spark/api/r/RBackend.scala    |   195 +
 .../apache/spark/api/r/RBackendAuthHandler.scala   |    54 +
 .../org/apache/spark/api/r/RBackendHandler.scala   |   285 +
 .../main/scala/org/apache/spark/api/r/RRDD.scala   |   186 +
 .../scala/org/apache/spark/api/r/RRunner.scala     |   179 +
 .../main/scala/org/apache/spark/api/r/RUtils.scala |   117 +
 .../main/scala/org/apache/spark/api/r/SerDe.scala  |   489 +
 .../org/apache/spark/broadcast/Broadcast.scala     |   151 +
 .../apache/spark/broadcast/BroadcastFactory.scala  |    51 +
 .../apache/spark/broadcast/BroadcastManager.scala  |    84 +
 .../apache/spark/broadcast/TorrentBroadcast.scala  |   401 +
 .../spark/broadcast/TorrentBroadcastFactory.scala  |    51 +
 .../org/apache/spark/broadcast/package-info.java   |    21 +
 .../scala/org/apache/spark/broadcast/package.scala |    25 +
 .../spark/deploy/ApplicationDescription.scala      |    45 +
 .../scala/org/apache/spark/deploy/Client.scala     |   301 +
 .../org/apache/spark/deploy/ClientArguments.scala  |   138 +
 .../scala/org/apache/spark/deploy/Command.scala    |    29 +
 .../org/apache/spark/deploy/DeployMessage.scala    |   303 +
 .../apache/spark/deploy/DriverDescription.scala    |    31 +
 .../apache/spark/deploy/DriverTimeoutPlugin.scala  |    64 +
 .../apache/spark/deploy/ExecutorDescription.scala  |    37 +
 .../spark/deploy/ExecutorFailureTracker.scala      |   104 +
 .../org/apache/spark/deploy/ExecutorState.scala    |    31 +
 .../spark/deploy/ExternalShuffleService.scala      |   192 +
 .../deploy/ExternalShuffleServiceSource.scala      |    37 +
 .../apache/spark/deploy/FaultToleranceTest.scala   |   453 +
 .../org/apache/spark/deploy/JsonProtocol.scala     |   320 +
 .../apache/spark/deploy/LocalSparkCluster.scala    |   124 +
 .../org/apache/spark/deploy/PythonRunner.scala     |   204 +
 .../org/apache/spark/deploy/RPackageUtils.scala    |   261 +
 .../scala/org/apache/spark/deploy/RRunner.scala    |   120 +
 .../spark/deploy/RedirectConsolePlugin.scala       |   137 +
 .../org/apache/spark/deploy/SparkApplication.scala |    55 +
 .../org/apache/spark/deploy/SparkCuratorUtil.scala |    68 +
 .../org/apache/spark/deploy/SparkHadoopUtil.scala  |   594 +
 .../org/apache/spark/deploy/SparkPipelines.scala   |   111 +
 .../org/apache/spark/deploy/SparkSubmit.scala      |  1318 +
 .../apache/spark/deploy/SparkSubmitArguments.scala |   678 +
 .../spark/deploy/StandaloneResourceUtils.scala     |   167 +
 .../main/scala/org/apache/spark/deploy/Utils.scala |   104 +
 .../spark/deploy/client/StandaloneAppClient.scala  |   346 +
 .../client/StandaloneAppClientListener.scala       |    47 +
 .../spark/deploy/history/ApplicationCache.scala    |   417 +
 .../history/ApplicationHistoryProvider.scala       |   169 +
 .../deploy/history/BasicEventFilterBuilder.scala   |   177 +
 .../apache/spark/deploy/history/EventFilter.scala  |   109 +
 .../deploy/history/EventLogFileCompactor.scala     |   230 +
 .../spark/deploy/history/EventLogFileReaders.scala |   291 +
 .../spark/deploy/history/EventLogFileWriters.scala |   430 +
 .../spark/deploy/history/FsHistoryProvider.scala   |  1702 ++
 .../deploy/history/HistoryAppStatusStore.scala     |    81 +
 .../apache/spark/deploy/history/HistoryPage.scala  |   122 +
 .../spark/deploy/history/HistoryServer.scala       |   370 +
 .../deploy/history/HistoryServerArguments.scala    |   109 +
 .../deploy/history/HistoryServerDiskManager.scala  |   347 +
 .../history/HistoryServerMemoryManager.scala       |    88 +
 .../apache/spark/deploy/history/HybridStore.scala  |   191 +
 .../org/apache/spark/deploy/history/LogPage.scala  |    79 +
 .../spark/deploy/master/ApplicationInfo.scala      |   207 +
 .../spark/deploy/master/ApplicationSource.scala    |    41 +
 .../spark/deploy/master/ApplicationState.scala     |    25 +
 .../apache/spark/deploy/master/DriverInfo.scala    |    58 +
 .../apache/spark/deploy/master/DriverState.scala   |    33 +
 .../apache/spark/deploy/master/ExecutorDesc.scala  |    57 +
 .../master/ExecutorResourceDescription.scala       |    32 +
 .../master/FileSystemPersistenceEngine.scala       |   102 +
 .../spark/deploy/master/LeaderElectionAgent.scala  |    43 +
 .../org/apache/spark/deploy/master/Master.scala    |  1436 +
 .../spark/deploy/master/MasterArguments.scala      |   103 +
 .../spark/deploy/master/MasterMessages.scala       |    42 +
 .../apache/spark/deploy/master/MasterSource.scala  |    47 +
 .../spark/deploy/master/PersistenceEngine.scala    |   102 +
 .../spark/deploy/master/RecoveryModeFactory.scala  |    99 +
 .../apache/spark/deploy/master/RecoveryState.scala |    24 +
 .../deploy/master/RocksDBPersistenceEngine.scala   |   108 +
 .../apache/spark/deploy/master/WorkerInfo.scala    |   187 +
 .../apache/spark/deploy/master/WorkerState.scala   |    24 +
 .../master/ZooKeeperLeaderElectionAgent.scala      |    90 +
 .../deploy/master/ZooKeeperPersistenceEngine.scala |    80 +
 .../spark/deploy/master/ui/ApplicationPage.scala   |   169 +
 .../spark/deploy/master/ui/EnvironmentPage.scala   |   160 +
 .../apache/spark/deploy/master/ui/LogPage.scala    |    78 +
 .../apache/spark/deploy/master/ui/MasterPage.scala |   390 +
 .../spark/deploy/master/ui/MasterWebUI.scala       |   134 +
 .../spark/deploy/rest/RestSubmissionClient.scala   |   584 +
 .../spark/deploy/rest/RestSubmissionServer.scala   |   438 +
 .../spark/deploy/rest/StandaloneRestServer.scala   |   294 +
 .../deploy/rest/SubmitRestProtocolException.scala  |    36 +
 .../deploy/rest/SubmitRestProtocolMessage.scala    |   146 +
 .../deploy/rest/SubmitRestProtocolRequest.scala    |    75 +
 .../deploy/rest/SubmitRestProtocolResponse.scala   |   115 +
 .../security/HBaseDelegationTokenProvider.scala    |   130 +
 .../security/HadoopDelegationTokenManager.scala    |   320 +
 .../security/HadoopFSDelegationTokenProvider.scala |   206 +
 .../org/apache/spark/deploy/security/README.md     |   267 +
 .../apache/spark/deploy/worker/CommandUtils.scala  |   129 +
 .../apache/spark/deploy/worker/DriverRunner.scala  |   279 +
 .../apache/spark/deploy/worker/DriverWrapper.scala |    97 +
 .../spark/deploy/worker/ExecutorRunner.scala       |   214 +
 .../org/apache/spark/deploy/worker/Worker.scala    |  1061 +
 .../spark/deploy/worker/WorkerArguments.scala      |   174 +
 .../apache/spark/deploy/worker/WorkerSource.scala  |    51 +
 .../apache/spark/deploy/worker/WorkerWatcher.scala |    93 +
 .../apache/spark/deploy/worker/ui/LogPage.scala    |   183 +
 .../apache/spark/deploy/worker/ui/WorkerPage.scala |   226 +
 .../spark/deploy/worker/ui/WorkerWebUI.scala       |    57 +
 .../org/apache/spark/errors/SparkCoreErrors.scala  |   511 +
 .../executor/CoarseGrainedExecutorBackend.scala    |   611 +
 .../spark/executor/CommitDeniedException.scala     |    33 +
 .../scala/org/apache/spark/executor/Executor.scala |  1620 +
 .../apache/spark/executor/ExecutorBackend.scala    |    30 +
 .../spark/executor/ExecutorClassLoader.scala       |   270 +
 .../apache/spark/executor/ExecutorExitCode.scala   |    83 +
 .../spark/executor/ExecutorLogUrlHandler.scala     |    96 +
 .../apache/spark/executor/ExecutorMetrics.scala    |   109 +
 .../spark/executor/ExecutorMetricsPoller.scala     |   201 +
 .../spark/executor/ExecutorMetricsSource.scala     |    64 +
 .../org/apache/spark/executor/ExecutorSource.scala |   152 +
 .../org/apache/spark/executor/InputMetrics.scala   |    61 +
 .../org/apache/spark/executor/OutputMetrics.scala  |    57 +
 .../spark/executor/ProcfsMetricsGetter.scala       |   170 +
 .../apache/spark/executor/ShuffleReadMetrics.scala |   300 +
 .../spark/executor/ShuffleWriteMetrics.scala       |    60 +
 .../org/apache/spark/executor/TaskMetrics.scala    |   399 +
 .../org/apache/spark/executor/package-info.java    |    21 +
 .../scala/org/apache/spark/executor/package.scala  |    24 +
 .../spark/input/FixedLengthBinaryInputFormat.scala |    88 +
 .../input/FixedLengthBinaryRecordReader.scala      |   126 +
 .../apache/spark/input/PortableDataStream.scala    |   217 +
 .../spark/input/WholeTextFileInputFormat.scala     |    71 +
 .../spark/input/WholeTextFileRecordReader.scala    |   117 +
 .../org/apache/spark/internal/config/Deploy.scala  |   155 +
 .../org/apache/spark/internal/config/History.scala |   320 +
 .../org/apache/spark/internal/config/Kryo.scala    |    66 +
 .../org/apache/spark/internal/config/Network.scala |    95 +
 .../org/apache/spark/internal/config/Python.scala  |   153 +
 .../scala/org/apache/spark/internal/config/R.scala |    45 +
 .../internal/config/SparkConfigProvider.scala      |    35 +
 .../org/apache/spark/internal/config/Status.scala  |    80 +
 .../apache/spark/internal/config/Streaming.scala   |    75 +
 .../org/apache/spark/internal/config/Tests.scala   |    95 +
 .../org/apache/spark/internal/config/UI.scala      |   280 +
 .../org/apache/spark/internal/config/Worker.scala  |   120 +
 .../org/apache/spark/internal/config/package.scala |  2922 ++
 .../spark/internal/io/FileCommitProtocol.scala     |   249 +
 .../internal/io/HadoopMapRedCommitProtocol.scala   |    41 +
 .../io/HadoopMapReduceCommitProtocol.scala         |   302 +
 .../spark/internal/io/HadoopWriteConfigUtil.scala  |    79 +
 .../spark/internal/io/SparkHadoopWriter.scala      |   410 +
 .../spark/internal/io/SparkHadoopWriterUtils.scala |   132 +
 .../spark/internal/plugin/PluginContainer.scala    |   220 +
 .../spark/internal/plugin/PluginContextImpl.scala  |    88 +
 .../spark/internal/plugin/PluginEndpoint.scala     |    67 +
 .../org/apache/spark/io/CompressionCodec.scala     |   273 +
 .../org/apache/spark/io/HadoopCodecStreams.scala   |    94 +
 .../spark/io/MutableCheckedOutputStream.scala      |    49 +
 .../scala/org/apache/spark/io/package-info.java    |    21 +
 .../main/scala/org/apache/spark/io/package.scala   |    23 +
 .../apache/spark/launcher/LauncherBackend.scala    |   129 +
 .../launcher/SparkSubmitArgumentsParser.scala      |    25 +
 .../spark/launcher/WorkerCommandBuilder.scala      |    47 +
 .../spark/mapred/SparkHadoopMapRedUtil.scala       |   102 +
 .../apache/spark/memory/ExecutionMemoryPool.scala  |   184 +
 .../org/apache/spark/memory/MemoryManager.scala    |   282 +
 .../scala/org/apache/spark/memory/MemoryPool.scala |    71 +
 .../apache/spark/memory/StorageMemoryPool.scala    |   135 +
 .../apache/spark/memory/UnifiedMemoryManager.scala |   491 +
 .../spark/memory/UnmanagedMemoryConsumer.scala     |    72 +
 .../scala/org/apache/spark/memory/package.scala    |    73 +
 .../apache/spark/metrics/ExecutorMetricType.scala  |   229 +
 .../org/apache/spark/metrics/MetricsConfig.scala   |   152 +
 .../org/apache/spark/metrics/MetricsSystem.scala   |   286 +
 .../apache/spark/metrics/sink/ConsoleSink.scala    |    64 +
 .../org/apache/spark/metrics/sink/CsvSink.scala    |    73 +
 .../apache/spark/metrics/sink/GraphiteSink.scala   |   103 +
 .../org/apache/spark/metrics/sink/JmxSink.scala    |    41 +
 .../apache/spark/metrics/sink/MetricsServlet.scala |    64 +
 .../spark/metrics/sink/PrometheusServlet.scala     |   134 +
 .../scala/org/apache/spark/metrics/sink/Sink.scala |    24 +
 .../org/apache/spark/metrics/sink/Slf4jSink.scala  |    64 +
 .../apache/spark/metrics/sink/StatsdReporter.scala |   163 +
 .../org/apache/spark/metrics/sink/StatsdSink.scala |    83 +
 .../org/apache/spark/metrics/sink/package.scala    |    23 +
 .../spark/metrics/source/AccumulatorSource.scala   |    82 +
 .../apache/spark/metrics/source/JVMCPUSource.scala |    49 +
 .../apache/spark/metrics/source/JvmSource.scala    |    33 +
 .../org/apache/spark/metrics/source/Source.scala   |    25 +
 .../spark/metrics/source/StaticSources.scala       |   110 +
 .../org/apache/spark/metrics/source/package.scala  |    23 +
 .../apache/spark/network/BlockDataManager.scala    |    83 +
 .../spark/network/BlockTransferService.scala       |   124 +
 .../spark/network/netty/NettyBlockRpcServer.scala  |   195 +
 .../network/netty/NettyBlockTransferService.scala  |   231 +
 .../spark/network/netty/SparkTransportConf.scala   |    72 +
 core/src/main/scala/org/apache/spark/package.scala |    54 +
 .../spark/partial/ApproximateActionListener.scala  |    88 +
 .../spark/partial/ApproximateEvaluator.scala       |    27 +
 .../org/apache/spark/partial/BoundedDouble.scala   |    42 +
 .../org/apache/spark/partial/CountEvaluator.scala  |    65 +
 .../spark/partial/GroupedCountEvaluator.scala      |    54 +
 .../org/apache/spark/partial/MeanEvaluator.scala   |    64 +
 .../org/apache/spark/partial/PartialResult.scala   |   137 +
 .../org/apache/spark/partial/SumEvaluator.scala    |    90 +
 .../scala/org/apache/spark/partial/package.scala   |    26 +
 .../scala/org/apache/spark/paths/SparkPath.scala   |    55 +
 .../org/apache/spark/rdd/AsyncRDDActions.scala     |   147 +
 .../scala/org/apache/spark/rdd/BinaryFileRDD.scala |    59 +
 .../main/scala/org/apache/spark/rdd/BlockRDD.scala |    91 +
 .../scala/org/apache/spark/rdd/CartesianRDD.scala  |    98 +
 .../scala/org/apache/spark/rdd/CheckpointRDD.scala |    39 +
 .../scala/org/apache/spark/rdd/CoGroupedRDD.scala  |   194 +
 .../scala/org/apache/spark/rdd/CoalescedRDD.scala  |   399 +
 .../org/apache/spark/rdd/DoubleRDDFunctions.scala  |   253 +
 .../main/scala/org/apache/spark/rdd/EmptyRDD.scala |    35 +
 .../scala/org/apache/spark/rdd/HadoopRDD.scala     |   501 +
 .../apache/spark/rdd/InputFileBlockHolder.scala    |   104 +
 .../main/scala/org/apache/spark/rdd/JdbcRDD.scala  |   229 +
 .../org/apache/spark/rdd/LocalCheckpointRDD.scala  |    64 +
 .../apache/spark/rdd/LocalRDDCheckpointData.scala  |    79 +
 .../org/apache/spark/rdd/MapPartitionsRDD.scala    |    69 +
 .../spark/rdd/MapPartitionsWithEvaluatorRDD.scala  |    41 +
 .../scala/org/apache/spark/rdd/NewHadoopRDD.scala  |   408 +
 .../org/apache/spark/rdd/OrderedRDDFunctions.scala |   111 +
 .../org/apache/spark/rdd/PairRDDFunctions.scala    |  1114 +
 .../apache/spark/rdd/ParallelCollectionRDD.scala   |   157 +
 .../org/apache/spark/rdd/PartitionPruningRDD.scala |    82 +
 .../spark/rdd/PartitionerAwareUnionRDD.scala       |   148 +
 .../apache/spark/rdd/PartitionwiseSampledRDD.scala |    78 +
 .../main/scala/org/apache/spark/rdd/PipedRDD.scala |   250 +
 core/src/main/scala/org/apache/spark/rdd/RDD.scala |  2207 ++
 .../scala/org/apache/spark/rdd/RDDBarrier.scala    |    92 +
 .../org/apache/spark/rdd/RDDCheckpointData.scala   |   112 +
 .../org/apache/spark/rdd/RDDOperationScope.scala   |   158 +
 .../apache/spark/rdd/ReliableCheckpointRDD.scala   |   336 +
 .../spark/rdd/ReliableRDDCheckpointData.scala      |    90 +
 .../spark/rdd/SequenceFileRDDFunctions.scala       |    79 +
 .../scala/org/apache/spark/rdd/ShuffledRDD.scala   |   117 +
 .../scala/org/apache/spark/rdd/SubtractedRDD.scala |   136 +
 .../main/scala/org/apache/spark/rdd/UnionRDD.scala |   118 +
 .../org/apache/spark/rdd/WholeTextFileRDD.scala    |    62 +
 .../org/apache/spark/rdd/ZippedPartitionsRDD.scala |   153 +
 .../rdd/ZippedPartitionsWithEvaluatorRDD.scala     |    38 +
 .../org/apache/spark/rdd/ZippedWithIndexRDD.scala  |    70 +
 .../org/apache/spark/rdd/coalesce-public.scala     |    52 +
 .../scala/org/apache/spark/rdd/package-info.java   |    21 +
 .../main/scala/org/apache/spark/rdd/package.scala  |    27 +
 .../spark/rdd/util/PeriodicRDDCheckpointer.scala   |   102 +
 .../spark/resource/ExecutorResourceRequest.scala   |    77 +
 .../spark/resource/ExecutorResourceRequests.scala  |   150 +
 .../apache/spark/resource/ResourceAllocator.scala  |   149 +
 .../resource/ResourceDiscoveryScriptPlugin.scala   |    68 +
 .../spark/resource/ResourceInformation.scala       |   105 +
 .../apache/spark/resource/ResourceProfile.scala    |   570 +
 .../spark/resource/ResourceProfileBuilder.scala    |   103 +
 .../spark/resource/ResourceProfileManager.scala    |   178 +
 .../org/apache/spark/resource/ResourceUtils.scala  |   504 +
 .../spark/resource/TaskResourceRequest.scala       |    58 +
 .../spark/resource/TaskResourceRequests.scala      |    88 +
 .../scala/org/apache/spark/rpc/RpcAddress.scala    |    57 +
 .../org/apache/spark/rpc/RpcCallContext.scala      |    41 +
 .../scala/org/apache/spark/rpc/RpcEndpoint.scala   |   177 +
 .../org/apache/spark/rpc/RpcEndpointAddress.scala  |    74 +
 .../spark/rpc/RpcEndpointNotFoundException.scala   |    22 +
 .../org/apache/spark/rpc/RpcEndpointRef.scala      |   113 +
 .../main/scala/org/apache/spark/rpc/RpcEnv.scala   |   232 +
 .../apache/spark/rpc/RpcEnvStoppedException.scala  |    20 +
 .../scala/org/apache/spark/rpc/RpcTimeout.scala    |   134 +
 .../org/apache/spark/rpc/netty/Dispatcher.scala    |   223 +
 .../scala/org/apache/spark/rpc/netty/Inbox.scala   |   239 +
 .../org/apache/spark/rpc/netty/MessageLoop.scala   |   199 +
 .../spark/rpc/netty/NettyRpcCallContext.scala      |    67 +
 .../org/apache/spark/rpc/netty/NettyRpcEnv.scala   |   765 +
 .../spark/rpc/netty/NettyStreamManager.scala       |   102 +
 .../scala/org/apache/spark/rpc/netty/Outbox.scala  |   283 +
 .../spark/rpc/netty/RpcEndpointVerifier.scala      |    40 +
 .../apache/spark/scheduler/AccumulableInfo.scala   |    49 +
 .../org/apache/spark/scheduler/ActiveJob.scala     |    66 +
 .../apache/spark/scheduler/AsyncEventQueue.scala   |   225 +
 .../scheduler/BarrierJobAllocationFailed.scala     |    65 +
 .../org/apache/spark/scheduler/DAGScheduler.scala  |  3626 +++
 .../apache/spark/scheduler/DAGSchedulerEvent.scala |   136 +
 .../spark/scheduler/DAGSchedulerSource.scala       |    52 +
 .../spark/scheduler/EventLoggingListener.scala     |   324 +
 .../spark/scheduler/ExecutorDecommissionInfo.scala |    40 +
 .../scheduler/ExecutorFailuresInTaskSet.scala      |    54 +
 .../spark/scheduler/ExecutorLossReason.scala       |    89 +
 .../spark/scheduler/ExecutorResourceInfo.scala     |    38 +
 .../spark/scheduler/ExecutorResourcesAmounts.scala |   217 +
 .../spark/scheduler/ExternalClusterManager.scala   |    62 +
 .../org/apache/spark/scheduler/HealthTracker.scala |   520 +
 .../apache/spark/scheduler/InputFormatInfo.scala   |   183 +
 .../org/apache/spark/scheduler/JobListener.scala   |    28 +
 .../org/apache/spark/scheduler/JobResult.scala     |    33 +
 .../org/apache/spark/scheduler/JobWaiter.scala     |    79 +
 .../apache/spark/scheduler/LiveListenerBus.scala   |   307 +
 .../org/apache/spark/scheduler/MapStatus.scala     |   339 +
 .../org/apache/spark/scheduler/MergeStatus.scala   |   116 +
 .../scheduler/MiscellaneousProcessDetails.scala    |    31 +
 .../spark/scheduler/OutputCommitCoordinator.scala  |   239 +
 .../scala/org/apache/spark/scheduler/Pool.scala    |   128 +
 .../apache/spark/scheduler/ReplayListenerBus.scala |   153 +
 .../org/apache/spark/scheduler/ResultStage.scala   |    68 +
 .../org/apache/spark/scheduler/ResultTask.scala    |   100 +
 .../org/apache/spark/scheduler/Schedulable.scala   |    50 +
 .../spark/scheduler/SchedulableBuilder.scala       |   233 +
 .../apache/spark/scheduler/SchedulerBackend.scala  |   113 +
 .../spark/scheduler/SchedulingAlgorithm.scala      |    75 +
 .../apache/spark/scheduler/SchedulingMode.scala    |    29 +
 .../apache/spark/scheduler/ShuffleMapStage.scala   |   113 +
 .../apache/spark/scheduler/ShuffleMapTask.scala    |   117 +
 .../org/apache/spark/scheduler/SparkListener.scala |   588 +
 .../apache/spark/scheduler/SparkListenerBus.scala  |   104 +
 .../org/apache/spark/scheduler/SplitInfo.scala     |    84 +
 .../scala/org/apache/spark/scheduler/Stage.scala   |   158 +
 .../org/apache/spark/scheduler/StageInfo.scala     |   125 +
 .../spark/scheduler/StatsReportListener.scala      |   200 +
 .../scala/org/apache/spark/scheduler/Task.scala    |   250 +
 .../apache/spark/scheduler/TaskDescription.scala   |   235 +
 .../org/apache/spark/scheduler/TaskInfo.scala      |   159 +
 .../org/apache/spark/scheduler/TaskLocality.scala  |    32 +
 .../org/apache/spark/scheduler/TaskLocation.scala  |    86 +
 .../org/apache/spark/scheduler/TaskResult.scala    |   113 +
 .../apache/spark/scheduler/TaskResultGetter.scala  |   180 +
 .../org/apache/spark/scheduler/TaskScheduler.scala |   124 +
 .../apache/spark/scheduler/TaskSchedulerImpl.scala |  1314 +
 .../scala/org/apache/spark/scheduler/TaskSet.scala |    48 +
 .../spark/scheduler/TaskSetExcludeList.scala       |   184 +
 .../apache/spark/scheduler/TaskSetManager.scala    |  1493 +
 .../org/apache/spark/scheduler/WorkerOffer.scala   |    34 +
 .../cluster/CoarseGrainedClusterMessage.scala      |   171 +
 .../cluster/CoarseGrainedSchedulerBackend.scala    |  1101 +
 .../spark/scheduler/cluster/ExecutorData.scala     |    49 +
 .../spark/scheduler/cluster/ExecutorInfo.scala     |    87 +
 .../scheduler/cluster/SchedulerBackendUtils.scala  |    47 +
 .../cluster/StandaloneSchedulerBackend.scala       |   366 +
 .../spark/scheduler/dynalloc/ExecutorMonitor.scala |   642 +
 .../scheduler/local/LocalSchedulerBackend.scala    |   189 +
 .../org/apache/spark/scheduler/package-info.java   |    21 +
 .../scala/org/apache/spark/scheduler/package.scala |    24 +
 .../apache/spark/security/CryptoStreamUtils.scala  |   169 +
 .../security/GroupMappingServiceProvider.scala     |    38 +
 .../security/HadoopDelegationTokenProvider.scala   |    56 +
 .../spark/security/SecurityConfigurationLock.scala |    24 +
 .../security/ShellBasedGroupsMappingProvider.scala |    46 +
 .../apache/spark/security/SocketAuthHelper.scala   |   128 +
 .../apache/spark/security/SocketAuthServer.scala   |   177 +
 .../spark/serializer/GenericAvroSerializer.scala   |   163 +
 .../apache/spark/serializer/JavaSerializer.scala   |   185 +
 .../apache/spark/serializer/KryoSerializer.scala   |   755 +
 .../spark/serializer/SerializationDebugger.scala   |   435 +
 .../org/apache/spark/serializer/Serializer.scala   |   200 +
 .../apache/spark/serializer/SerializerHelper.scala |    56 +
 .../spark/serializer/SerializerManager.scala       |   224 +
 .../org/apache/spark/serializer/package-info.java  |    21 +
 .../org/apache/spark/serializer/package.scala      |    25 +
 .../apache/spark/shuffle/BaseShuffleHandle.scala   |    28 +
 .../spark/shuffle/BlockStoreShuffleReader.scala    |   160 +
 .../spark/shuffle/FetchFailedException.scala       |    72 +
 .../spark/shuffle/IndexShuffleBlockResolver.scala  |   677 +
 .../apache/spark/shuffle/MigratableResolver.scala  |    54 +
 .../apache/spark/shuffle/ShuffleBlockInfo.scala    |    30 +
 .../apache/spark/shuffle/ShuffleBlockPusher.scala  |   522 +
 .../spark/shuffle/ShuffleBlockResolver.scala       |    67 +
 .../spark/shuffle/ShuffleChecksumUtils.scala       |    93 +
 .../apache/spark/shuffle/ShuffleDataIOUtils.scala  |    42 +
 .../org/apache/spark/shuffle/ShuffleHandle.scala   |    28 +
 .../org/apache/spark/shuffle/ShuffleManager.scala  |   120 +
 .../shuffle/ShufflePartitionPairsWriter.scala      |   147 +
 .../org/apache/spark/shuffle/ShuffleReader.scala   |    33 +
 .../spark/shuffle/ShuffleWriteProcessor.scala      |   102 +
 .../org/apache/spark/shuffle/ShuffleWriter.scala   |    37 +
 .../scala/org/apache/spark/shuffle/metrics.scala   |    62 +
 .../spark/shuffle/sort/SortShuffleManager.scala    |   278 +
 .../spark/shuffle/sort/SortShuffleWriter.scala     |   127 +
 .../spark/status/AppHistoryServerPlugin.scala      |    43 +
 .../apache/spark/status/AppStatusListener.scala    |  1448 +
 .../org/apache/spark/status/AppStatusSource.scala  |    94 +
 .../org/apache/spark/status/AppStatusStore.scala   |   882 +
 .../org/apache/spark/status/AppStatusUtils.scala   |    87 +
 .../apache/spark/status/ElementTrackingStore.scala |   209 +
 .../scala/org/apache/spark/status/KVUtils.scala    |   243 +
 .../scala/org/apache/spark/status/LiveEntity.scala |  1056 +
 .../spark/status/api/v1/ApiRootResource.scala      |   184 +
 .../status/api/v1/ApplicationListResource.scala    |    68 +
 .../spark/status/api/v1/JacksonMessageWriter.scala |    90 +
 .../status/api/v1/OneApplicationResource.scala     |   221 +
 .../spark/status/api/v1/PrometheusResource.scala   |   121 +
 .../spark/status/api/v1/SimpleDateParam.scala      |    49 +
 .../spark/status/api/v1/StagesResource.scala       |   272 +
 .../scala/org/apache/spark/status/api/v1/api.scala |   585 +
 .../protobuf/AccumulableInfoSerializer.scala       |    50 +
 .../status/protobuf/AppSummarySerializer.scala     |    38 +
 ...plicationEnvironmentInfoWrapperSerializer.scala |   156 +
 .../ApplicationInfoWrapperSerializer.scala         |   116 +
 .../status/protobuf/CachedQuantileSerializer.scala |   118 +
 .../protobuf/ExecutorMetricsSerializer.scala       |    38 +
 .../protobuf/ExecutorStageSummarySerializer.scala  |    72 +
 .../ExecutorStageSummaryWrapperSerializer.scala    |    46 +
 .../ExecutorSummaryWrapperSerializer.scala         |   181 +
 .../status/protobuf/JobDataWrapperSerializer.scala |   112 +
 .../protobuf/JobExecutionStatusSerializer.scala    |    43 +
 .../protobuf/KVStoreProtobufSerializer.scala       |    56 +
 .../spark/status/protobuf/PoolDataSerializer.scala |    41 +
 .../protobuf/ProcessSummaryWrapperSerializer.scala |    72 +
 .../spark/status/protobuf/ProtobufSerDe.scala      |    45 +
 .../RDDOperationGraphWrapperSerializer.scala       |   146 +
 .../protobuf/RDDStorageInfoWrapperSerializer.scala |   129 +
 .../ResourceProfileWrapperSerializer.scala         |    39 +
 .../SpeculationStageSummaryWrapperSerializer.scala |    64 +
 .../protobuf/StageDataWrapperSerializer.scala      |   716 +
 .../status/protobuf/StageStatusSerializer.scala    |    45 +
 .../protobuf/StreamBlockDataSerializer.scala       |    54 +
 .../protobuf/TaskDataWrapperSerializer.scala       |   142 +
 .../org/apache/spark/status/protobuf/Utils.scala   |    46 +
 .../scala/org/apache/spark/status/storeTypes.scala |   611 +
 .../org/apache/spark/storage/BlockException.scala  |    22 +
 .../scala/org/apache/spark/storage/BlockId.scala   |   337 +
 .../apache/spark/storage/BlockInfoManager.scala    |   687 +
 .../org/apache/spark/storage/BlockManager.scala    |  2342 ++
 .../spark/storage/BlockManagerDecommissioner.scala |   467 +
 .../org/apache/spark/storage/BlockManagerId.scala  |   152 +
 .../spark/storage/BlockManagerManagedBuffer.scala  |    72 +
 .../apache/spark/storage/BlockManagerMaster.scala  |   325 +
 .../spark/storage/BlockManagerMasterEndpoint.scala |  1161 +
 .../BlockManagerMasterHeartbeatEndpoint.scala      |    58 +
 .../spark/storage/BlockManagerMessages.scala       |   165 +
 .../apache/spark/storage/BlockManagerSource.scala  |    64 +
 .../storage/BlockManagerStorageEndpoint.scala      |   127 +
 .../spark/storage/BlockNotFoundException.scala     |    20 +
 .../spark/storage/BlockReplicationPolicy.scala     |   224 +
 ...avedOnDecommissionedBlockManagerException.scala |    21 +
 .../apache/spark/storage/BlockUpdatedInfo.scala    |    45 +
 .../apache/spark/storage/DiskBlockManager.scala    |   388 +
 .../spark/storage/DiskBlockObjectWriter.scala      |   378 +
 .../scala/org/apache/spark/storage/DiskStore.scala |   380 +
 .../org/apache/spark/storage/FallbackStorage.scala |   217 +
 .../org/apache/spark/storage/FileSegment.scala     |    32 +
 .../apache/spark/storage/LogBlockIdGenerator.scala |    60 +
 .../org/apache/spark/storage/LogBlockWriter.scala  |   182 +
 .../scala/org/apache/spark/storage/LogLine.scala   |    53 +
 .../spark/storage/PushBasedFetchHelper.scala       |   360 +
 .../scala/org/apache/spark/storage/RDDInfo.scala   |    73 +
 .../apache/spark/storage/RollingLogWriter.scala    |   115 +
 .../storage/ShuffleBlockFetcherIterator.scala      |  1681 ++
 .../org/apache/spark/storage/StorageUtils.scala    |   237 +
 .../org/apache/spark/storage/TopologyMapper.scala  |    87 +
 .../apache/spark/storage/memory/MemoryStore.scala  |   959 +
 .../org/apache/spark/ui/ConsoleProgressBar.scala   |   130 +
 .../scala/org/apache/spark/ui/DriverLogPage.scala  |   155 +
 .../scala/org/apache/spark/ui/GraphUIData.scala    |   182 +
 .../org/apache/spark/ui/HttpSecurityFilter.scala   |   131 +
 .../main/scala/org/apache/spark/ui/JWSFilter.scala |    80 +
 .../scala/org/apache/spark/ui/JettyUtils.scala     |   648 +
 .../scala/org/apache/spark/ui/PagedTable.scala     |   399 +
 .../main/scala/org/apache/spark/ui/SparkUI.scala   |   256 +
 .../main/scala/org/apache/spark/ui/ToolTips.scala  |    97 +
 .../main/scala/org/apache/spark/ui/UIUtils.scala   |   756 +
 .../org/apache/spark/ui/UIWorkloadGenerator.scala  |   122 +
 .../src/main/scala/org/apache/spark/ui/WebUI.scala |   256 +
 .../org/apache/spark/ui/env/EnvironmentPage.scala  |   195 +
 .../spark/ui/exec/ExecutorHeapHistogramPage.scala  |    89 +
 .../spark/ui/exec/ExecutorThreadDumpPage.scala     |   184 +
 .../org/apache/spark/ui/exec/ExecutorsTab.scala    |    78 +
 .../spark/ui/flamegraph/FlamegraphNode.scala       |    50 +
 .../org/apache/spark/ui/jobs/AllJobsPage.scala     |   618 +
 .../org/apache/spark/ui/jobs/AllStagesPage.scala   |   160 +
 .../org/apache/spark/ui/jobs/JobDataUtil.scala     |    38 +
 .../scala/org/apache/spark/ui/jobs/JobPage.scala   |   530 +
 .../scala/org/apache/spark/ui/jobs/JobsTab.scala   |    65 +
 .../scala/org/apache/spark/ui/jobs/PoolPage.scala  |    68 +
 .../scala/org/apache/spark/ui/jobs/PoolTable.scala |    75 +
 .../org/apache/spark/ui/jobs/StageDataUtil.scala   |    39 +
 .../scala/org/apache/spark/ui/jobs/StagePage.scala |   559 +
 .../org/apache/spark/ui/jobs/StageTable.scala      |   369 +
 .../scala/org/apache/spark/ui/jobs/StagesTab.scala |    70 +
 .../apache/spark/ui/jobs/TaskThreadDumpPage.scala  |   109 +
 .../apache/spark/ui/scope/RDDOperationGraph.scala  |   298 +
 .../org/apache/spark/ui/storage/RDDPage.scala      |   271 +
 .../org/apache/spark/ui/storage/StoragePage.scala  |   249 +
 .../org/apache/spark/ui/storage/StorageTab.scala   |    29 +
 .../org/apache/spark/ui/storage/ToolTips.scala     |    42 +
 .../org/apache/spark/util/AccumulatorV2.scala      |   526 +
 .../util/BlockingThreadPoolExecutorService.scala   |   124 +
 .../apache/spark/util/BoundedPriorityQueue.scala   |    71 +
 .../apache/spark/util/ByteBufferInputStream.scala  |    75 +
 .../apache/spark/util/ByteBufferOutputStream.scala |    60 +
 .../scala/org/apache/spark/util/CausedBy.scala     |    36 +
 .../main/scala/org/apache/spark/util/Clock.scala   |   101 +
 .../org/apache/spark/util/CollectionsUtils.scala   |    47 +
 .../org/apache/spark/util/CommandLineUtils.scala   |    50 +
 .../org/apache/spark/util/DependencyUtils.scala    |   261 +
 .../spark/util/DirectByteBufferOutputStream.scala  |   103 +
 .../scala/org/apache/spark/util/Distribution.scala |    91 +
 .../scala/org/apache/spark/util/EventLoop.scala    |   143 +
 .../org/apache/spark/util/HadoopFSUtils.scala      |   380 +
 .../scala/org/apache/spark/util/IdGenerator.scala  |    31 +
 .../scala/org/apache/spark/util/IntParam.scala     |    31 +
 .../scala/org/apache/spark/util/JsonProtocol.scala |  1705 ++
 .../main/scala/org/apache/spark/util/KeyLock.scala |    69 +
 .../main/scala/org/apache/spark/util/LazyTry.scala |    74 +
 .../scala/org/apache/spark/util/ListenerBus.scala  |   166 +
 .../scala/org/apache/spark/util/ManualClock.scala  |    71 +
 .../scala/org/apache/spark/util/MemoryParam.scala  |    32 +
 .../scala/org/apache/spark/util/MetricUtils.scala  |   106 +
 .../scala/org/apache/spark/util/MutablePair.scala  |    49 +
 .../scala/org/apache/spark/util/NextIterator.scala |    90 +
 .../apache/spark/util/NonFateSharingCache.scala    |   110 +
 .../apache/spark/util/PeriodicCheckpointer.scala   |   193 +
 .../scala/org/apache/spark/util/RpcUtils.scala     |    67 +
 .../org/apache/spark/util/SecurityUtils.scala      |    69 +
 .../org/apache/spark/util/SerializableBuffer.scala |    54 +
 .../spark/util/SerializableConfiguration.scala     |    53 +
 .../apache/spark/util/SerializableJobConf.scala    |    35 +
 .../apache/spark/util/ShutdownHookManager.scala    |   233 +
 .../scala/org/apache/spark/util/SignalUtils.scala  |   141 +
 .../org/apache/spark/util/SizeEstimator.scala      |   398 +
 .../apache/spark/util/SparkClosureCleaner.scala    |    49 +
 .../org/apache/spark/util/SparkExitCode.scala      |    56 +
 .../spark/util/SparkUncaughtExceptionHandler.scala |   119 +
 .../scala/org/apache/spark/util/StatCounter.scala  |   162 +
 .../org/apache/spark/util/StubClassLoader.scala    |    96 +
 .../scala/org/apache/spark/util/ThreadUtils.scala  |   441 +
 .../apache/spark/util/UninterruptibleThread.scala  |   171 +
 .../spark/util/UninterruptibleThreadRunner.scala   |    55 +
 .../main/scala/org/apache/spark/util/Utils.scala   |  3356 +++
 .../spark/util/collection/AppendOnlyMap.scala      |   301 +
 .../org/apache/spark/util/collection/BitSet.scala  |   280 +
 .../spark/util/collection/CompactBuffer.scala      |   161 +
 .../util/collection/ExternalAppendOnlyMap.scala    |   633 +
 .../spark/util/collection/ExternalSorter.scala     |   870 +
 .../spark/util/collection/ImmutableBitSet.scala    |    58 +
 .../apache/spark/util/collection/OpenHashMap.scala |   178 +
 .../apache/spark/util/collection/OpenHashSet.scala |   322 +
 .../apache/spark/util/collection/PairsWriter.scala |    28 +
 .../util/collection/PartitionedAppendOnlyMap.scala |    40 +
 .../util/collection/PartitionedPairBuffer.scala    |   101 +
 .../spark/util/collection/PercentileHeap.scala     |    81 +
 .../spark/util/collection/PrimitiveVector.scala    |    91 +
 .../apache/spark/util/collection/SizeTracker.scala |   105 +
 .../collection/SizeTrackingAppendOnlyMap.scala     |    41 +
 .../spark/util/collection/SizeTrackingVector.scala |    39 +
 .../spark/util/collection/SortDataFormat.scala     |   112 +
 .../org/apache/spark/util/collection/Sorter.scala  |    39 +
 .../apache/spark/util/collection/Spillable.scala   |   161 +
 .../org/apache/spark/util/collection/Utils.scala   |    96 +
 .../WritablePartitionedPairCollection.scala        |    90 +
 .../apache/spark/util/io/ChunkedByteBuffer.scala   |   356 +
 .../util/io/ChunkedByteBufferFileRegion.scala      |    82 +
 .../util/io/ChunkedByteBufferOutputStream.scala    |   124 +
 .../apache/spark/util/logging/DriverLogger.scala   |   240 +
 .../apache/spark/util/logging/FileAppender.scala   |   202 +
 .../spark/util/logging/RollingFileAppender.scala   |   193 +
 .../apache/spark/util/logging/RollingPolicy.scala  |   142 +
 .../scala/org/apache/spark/util/package-info.java  |    21 +
 .../main/scala/org/apache/spark/util/package.scala |    23 +
 .../apache/spark/util/random/Pseudorandom.scala    |    30 +
 .../apache/spark/util/random/RandomSampler.scala   |   340 +
 .../apache/spark/util/random/SamplingUtils.scala   |   169 +
 .../util/random/StratifiedSamplingUtils.scala      |   332 +
 .../apache/spark/util/random/XORShiftRandom.scala  |    67 +
 .../org/apache/spark/util/random/package-info.java |    21 +
 .../org/apache/spark/util/random/package.scala     |    23 +
 .../org/apache/spark/util/taskListeners.scala      |    70 +
 .../java/org/apache/spark/JavaJdbcRDDSuite.java    |    98 +
 .../org/apache/spark/api/java/OptionalSuite.java   |    97 +
 .../spark/io/GenericFileInputStreamSuite.java      |   159 +
 .../spark/io/NioBufferedInputStreamSuite.java      |    37 +
 .../apache/spark/io/ReadAheadInputStreamSuite.java |    45 +
 .../apache/spark/launcher/SparkLauncherSuite.java  |   328 +
 .../spark/memory/TaskMemoryManagerSuite.java       |   292 +
 .../apache/spark/memory/TestMemoryConsumer.java    |    58 +
 .../memory/TestPartialSpillingMemoryConsumer.java  |    48 +
 .../spark/resource/JavaResourceProfileSuite.java   |    66 +
 .../spark/serializer/ContainsProxyClass.java       |    50 +
 .../spark/serializer/TestJavaSerializerImpl.java   |    94 +
 .../shuffle/sort/PackedRecordPointerSuite.java     |   110 +
 .../sort/ShuffleInMemoryRadixSorterSuite.java      |    23 +
 .../shuffle/sort/ShuffleInMemorySorterSuite.java   |   146 +
 .../shuffle/sort/UnsafeShuffleWriterSuite.java     |   716 +
 .../unsafe/map/AbstractBytesToBytesMapSuite.java   |   733 +
 .../unsafe/map/BytesToBytesMapOffHeapSuite.java    |    26 +
 .../unsafe/map/BytesToBytesMapOnHeapSuite.java     |    26 +
 .../spark/util/SerializableConfigurationSuite.java |    55 +
 .../apache/spark/util/collection/TestTimSort.java  |   134 +
 .../sort/UnsafeExternalSorterRadixSortSuite.java   |    23 +
 .../unsafe/sort/UnsafeExternalSorterSuite.java     |   617 +
 .../sort/UnsafeInMemorySorterRadixSortSuite.java   |    23 +
 .../unsafe/sort/UnsafeInMemorySorterSuite.java     |   194 +
 .../test/org/apache/spark/Java8RDDAPISuite.java    |   351 +
 .../java/test/org/apache/spark/JavaAPISuite.java   |  1534 +
 .../org/apache/spark/JavaSparkContextSuite.java    |    65 +
 .../apache/spark/JavaTaskContextCompileCheck.java  |    80 +
 .../app_environment_expectation.json               |    13 +
 .../application_list_json_expectation.json         |   282 +
 .../complete_stage_list_json_expectation.json      |   247 +
 .../completed_app_list_json_expectation.json       |   282 +
 .../excludeOnFailure_for_stage_expectation.json    |   967 +
 ...xcludeOnFailure_node_for_stage_expectation.json |  1230 +
 .../executor_list_json_expectation.json            |    54 +
 ...ist_with_executor_metrics_json_expectation.json |   276 +
 .../executor_memory_usage_expectation.json         |   283 +
 ...executor_node_excludeOnFailure_expectation.json |   283 +
 ...e_excludeOnFailure_unexcluding_expectation.json |   253 +
 .../executor_resource_information_expectation.json |   139 +
 .../failed_stage_list_json_expectation.json        |    84 +
 ...from_multi_attempt_app_json_1__expectation.json |    19 +
 ...from_multi_attempt_app_json_2__expectation.json |    19 +
 .../job_list_json_expectation.json                 |    55 +
 .../limit_app_list_json_expectation.json           |    46 +
 .../maxDate2_app_list_json_expectation.json        |    16 +
 .../maxDate_app_list_json_expectation.json         |    31 +
 .../maxEndDate_app_list_json_expectation.json      |   102 +
 ...e_and_maxEndDate_app_list_json_expectation.json |    57 +
 .../minDate_app_list_json_expectation.json         |   252 +
 ...e_and_maxEndDate_app_list_json_expectation.json |    57 +
 .../minEndDate_app_list_json_expectation.json      |   209 +
 .../miscellaneous_process_expectation.json         |    11 +
 .../multiple_resource_profiles_expectation.json    |   113 +
 .../one_app_json_expectation.json                  |    16 +
 .../one_app_multi_attempt_json_expectation.json    |    29 +
 .../one_job_json_expectation.json                  |    19 +
 .../one_rdd_storage_json_expectation.json          |    10 +
 ..._json_details_with_failed_task_expectation.json |   129 +
 .../one_stage_attempt_json_expectation.json        |   634 +
 .../one_stage_json_expectation.json                |   634 +
 .../one_stage_json_with_details_expectation.json   |   634 +
 ...ne_stage_json_with_partitionId_expectation.json |   760 +
 .../rdd_list_storage_json_expectation.json         |     1 +
 .../running_app_list_json_expectation.json         |     1 +
 .../stage_list_json_expectation.json               |   330 +
 ...age_list_with_accumulable_json_expectation.json |    87 +
 .../stage_list_with_peak_metrics_expectation.json  |   249 +
 .../stage_task_list_expectation.json               |  1241 +
 ...from_multi_attempt_app_json_1__expectation.json |   537 +
 ...from_multi_attempt_app_json_2__expectation.json |   537 +
 ...e_task_list_w__offset___length_expectation.json |  3101 ++
 .../stage_task_list_w__sortBy_expectation.json     |  1241 +
 ...__sortBy_short_names___runtime_expectation.json |  1241 +
 ...w__sortBy_short_names__runtime_expectation.json |  1241 +
 ...st_w__status___offset___length_expectation.json |   125 +
 ...___sortBy_short_names__runtime_expectation.json |  1241 +
 .../stage_task_list_w__status_expectation.json     |   661 +
 ...age_task_list_with_partitionId_expectation.json |   497 +
 ...sk_summary_w__custom_quantiles_expectation.json |    51 +
 ...ge_task_summary_w_shuffle_read_expectation.json |    51 +
 ...e_task_summary_w_shuffle_write_expectation.json |    51 +
 .../stage_with_accumulable_json_expectation.json   |   678 +
 .../stage_with_peak_metrics_expectation.json       |  1229 +
 ...stage_with_speculation_summary_expectation.json |   599 +
 .../stage_with_summaries_expectation.json          |  1322 +
 ...succeeded_failed_job_list_json_expectation.json |    55 +
 .../succeeded_job_list_json_expectation.json       |    37 +
 ...g.apache.spark.scheduler.ExternalClusterManager |    21 +
 ...he.spark.security.HadoopDelegationTokenProvider |    18 +
 core/src/test/resources/ProcfsMetrics/22763/stat   |     1 +
 core/src/test/resources/ProcfsMetrics/26109/stat   |     1 +
 core/src/test/resources/ProcfsMetrics/487713/stat  |     1 +
 core/src/test/resources/TestHelloV2_2.13.jar       |   Bin 0 -> 4118 bytes
 core/src/test/resources/TestHelloV3_2.13.jar       |   Bin 0 -> 3827 bytes
 core/src/test/resources/TestUDTF.jar               |   Bin 0 -> 1328 bytes
 core/src/test/resources/certchain.pem              |    17 +
 .../resources/fairscheduler-with-invalid-data.xml  |    80 +
 .../resources/fairscheduler-with-valid-data.xml    |    35 +
 core/src/test/resources/fairscheduler.xml          |    32 +
 core/src/test/resources/hive-site.xml              |    34 +
 core/src/test/resources/key.pem                    |    29 +
 core/src/test/resources/keystore                   |   Bin 0 -> 2247 bytes
 core/src/test/resources/log4j2.properties          |    39 +
 .../application_1656321732247_0006_1               |    10 +
 .../application_1656321732247_0006_2               |     9 +
 .../application_1656321732247_0006_1               |     9 +
 .../application_1656321732247_0006_2               |    10 +
 .../resources/spark-events/app-20161115172038-0000 |    75 +
 .../resources/spark-events/app-20161116163331-0000 |    68 +
 .../resources/spark-events/app-20180109111548-0000 |    59 +
 .../resources/spark-events/app-20200706201101-0003 |   124 +
 .../spark-events/application_1516285256255_0012    |    71 +
 .../spark-events/application_1553914137147_0018    |    44 +
 .../spark-events/application_1555004656427_0144    |    10 +
 .../spark-events/application_1578436911597_0052    |    27 +
 .../spark-events/application_1628109047826_1317105 |    52 +
 .../.appstatus_local-1766844910796.crc             |   Bin 0 -> 8 bytes
 .../appstatus_local-1766844910796                  |     0
 .../events_1_local-1766844910796                   |    15 +
 .../resources/spark-events/local-1422981759269     |    88 +
 .../resources/spark-events/local-1422981780767     |    82 +
 .../resources/spark-events/local-1425081759269     |    88 +
 .../resources/spark-events/local-1426533911241     |    24 +
 .../resources/spark-events/local-1426633911242     |    24 +
 .../resources/spark-events/local-1430917381534     |   231 +
 .../resources/spark-events/local-1430917381535_1   |     5 +
 .../resources/spark-events/local-1430917381535_2   |     5 +
 .../resources/spark-events/local-1642039451826     |   115 +
 core/src/test/resources/spark.keystore             |   Bin 0 -> 1383 bytes
 .../test/resources/test_metrics_config.properties  |    23 +
 .../test/resources/test_metrics_system.properties  |    22 +
 core/src/test/resources/truststore                 |   Bin 0 -> 957 bytes
 core/src/test/resources/unencrypted-certchain.pem  |    21 +
 core/src/test/resources/unencrypted-key.pem        |    28 +
 core/src/test/resources/untrusted-keystore         |   Bin 0 -> 2246 bytes
 .../scala/org/apache/spark/AccumulatorSuite.scala  |   239 +
 .../spark/BarrierStageOnSubmittedSuite.scala       |   298 +
 .../scala/org/apache/spark/CheckpointSuite.scala   |   687 +
 .../org/apache/spark/ContextCleanerSuite.scala     |   551 +
 .../scala/org/apache/spark/DebugFilesystem.scala   |   133 +
 .../scala/org/apache/spark/DistributedSuite.scala  |   388 +
 .../test/scala/org/apache/spark/DriverSuite.scala  |    59 +
 .../spark/ExecutorAllocationManagerSuite.scala     |  2034 ++
 .../apache/spark/ExternalShuffleServiceSuite.scala |   342 +
 .../test/scala/org/apache/spark/FailureSuite.scala |   282 +
 .../apache/spark/FeatureSpecSparkTestSuite.scala   |    51 +
 .../test/scala/org/apache/spark/FileSuite.scala    |   762 +
 .../org/apache/spark/FlatSpecSparkTestSuite.scala  |    47 +
 .../org/apache/spark/FreeSpecSparkTestSuite.scala  |    51 +
 .../org/apache/spark/FunSpecSparkTestSuite.scala   |    49 +
 .../scala/org/apache/spark/FutureActionSuite.scala |    55 +
 .../org/apache/spark/HeartbeatReceiverSuite.scala  |   367 +
 .../org/apache/spark/ImplicitOrderingSuite.scala   |    83 +
 .../apache/spark/InternalAccumulatorSuite.scala    |   226 +
 .../org/apache/spark/JobArtifactSetSuite.scala     |   124 +
 .../org/apache/spark/JobCancellationSuite.scala    |   918 +
 .../scala/org/apache/spark/JsonTestUtils.scala     |    34 +
 .../scala/org/apache/spark/LocalRootDirsTest.scala |    61 +
 .../scala/org/apache/spark/LocalSparkContext.scala |    71 +
 .../org/apache/spark/MapOutputTrackerSuite.scala   |  1169 +
 .../apache/spark/MapStatusesConvertBenchmark.scala |    92 +
 .../spark/MapStatusesSerDeserBenchmark.scala       |   130 +
 .../scala/org/apache/spark/PartitioningSuite.scala |   327 +
 .../scala/org/apache/spark/ProcessTestUtils.scala  |    37 +
 .../org/apache/spark/PropSpecSparkTestSuite.scala  |    47 +
 .../org/apache/spark/RefSpecSparkTestSuite.scala   |    51 +
 .../scala/org/apache/spark/SSLOptionsSuite.scala   |   357 +
 .../org/apache/spark/SecurityManagerSuite.scala    |   517 +
 .../org/apache/spark/SerializerTestUtils.scala     |    44 +
 .../org/apache/spark/SharedSparkContext.scala      |    72 +
 .../scala/org/apache/spark/ShuffleNettySuite.scala |    61 +
 .../spark/ShuffleOldFetchProtocolSuite.scala       |    27 +
 .../test/scala/org/apache/spark/ShuffleSuite.scala |   577 +
 core/src/test/scala/org/apache/spark/Smuggle.scala |    83 +
 .../scala/org/apache/spark/SortShuffleSuite.scala  |    76 +
 .../scala/org/apache/spark/SparkConfSuite.scala    |   547 +
 .../org/apache/spark/SparkContextInfoSuite.scala   |   107 +
 .../spark/SparkContextSchedulerCreationSuite.scala |   145 +
 .../scala/org/apache/spark/SparkContextSuite.scala |  1485 +
 .../scala/org/apache/spark/SparkFunSuite.scala     |   100 +
 .../scala/org/apache/spark/SparkTestSuite.scala    |   486 +
 .../org/apache/spark/SparkThrowableSuite.scala     |   803 +
 .../spark/SslExternalShuffleServiceSuite.scala     |    53 +
 .../org/apache/spark/SslShuffleNettySuite.scala    |    28 +
 .../org/apache/spark/StatusTrackerSuite.scala      |   158 +
 .../org/apache/spark/TempLocalSparkContext.scala   |   100 +
 .../TestShuffleDataIOWithMockedComponents.scala    |    53 +
 .../test/scala/org/apache/spark/ThreadAudit.scala  |   131 +
 .../scala/org/apache/spark/ThreadingSuite.scala    |   245 +
 .../scala/org/apache/spark/UnpersistSuite.scala    |    51 +
 .../org/apache/spark/WordSpecSparkTestSuite.scala  |    51 +
 .../org/apache/spark/api/java/JavaUtilsSuite.scala |    51 +
 .../spark/api/python/PythonBroadcastSuite.scala    |    59 +
 .../spark/api/python/PythonHadoopUtilSuite.scala   |    79 +
 .../apache/spark/api/python/PythonRDDSuite.scala   |   155 +
 .../api/python/PythonWorkerFactorySuite.scala      |   112 +
 .../apache/spark/api/python/SerDeUtilSuite.scala   |    36 +
 .../python/WriteInputFormatTestDataGenerator.scala |   189 +
 .../org/apache/spark/api/r/BaseRRunnerSuite.scala  |    36 +
 .../apache/spark/api/r/JVMObjectTrackerSuite.scala |    73 +
 .../org/apache/spark/api/r/RBackendSuite.scala     |    31 +
 .../org/apache/spark/benchmark/Benchmark.scala     |   238 +
 .../org/apache/spark/benchmark/BenchmarkBase.scala |    90 +
 .../org/apache/spark/benchmark/Benchmarks.scala    |   142 +
 .../apache/spark/broadcast/BroadcastSuite.scala    |   345 +
 .../org/apache/spark/deploy/ClientSuite.scala      |    51 +
 .../spark/deploy/DecommissionWorkerSuite.scala     |   470 +
 .../org/apache/spark/deploy/DeployTestUtils.scala  |   148 +
 .../spark/deploy/ExecutorFailureTrackerSuite.scala |   103 +
 .../deploy/ExternalShuffleServiceDbSuite.scala     |   155 +
 .../ExternalShuffleServiceMetricsSuite.scala       |    73 +
 .../apache/spark/deploy/JsonProtocolSuite.scala    |   252 +
 .../spark/deploy/LogUrlsStandaloneSuite.scala      |    80 +
 .../apache/spark/deploy/PythonRunnerSuite.scala    |    95 +
 .../apache/spark/deploy/RPackageUtilsSuite.scala   |   172 +
 .../apache/spark/deploy/SparkHadoopUtilSuite.scala |   203 +
 .../apache/spark/deploy/SparkPipelinesSuite.scala  |   212 +
 .../org/apache/spark/deploy/SparkSubmitSuite.scala |  2119 ++
 .../apache/spark/deploy/SparkSubmitTestUtils.scala |   107 +
 .../deploy/StandaloneDynamicAllocationSuite.scala  |   660 +
 .../spark/deploy/client/AppClientSuite.scala       |   323 +
 .../apache/spark/deploy/client/TestExecutor.scala  |    29 +
 .../deploy/history/ApplicationCacheSuite.scala     |   472 +
 .../history/BasicEventFilterBuilderSuite.scala     |   230 +
 .../deploy/history/BasicEventFilterSuite.scala     |   214 +
 .../history/ChromeUIHistoryServerSuite.scala       |    64 +
 .../history/EventLogFileCompactorSuite.scala       |   333 +
 .../deploy/history/EventLogFileReadersSuite.scala  |   378 +
 .../deploy/history/EventLogFileWritersSuite.scala  |   393 +
 .../spark/deploy/history/EventLogTestHelper.scala  |   113 +
 .../deploy/history/FsHistoryProviderSuite.scala    |  1940 ++
 .../history/HistoryServerArgumentsSuite.scala      |    66 +
 .../history/HistoryServerDiskManagerSuite.scala    |   239 +
 .../history/HistoryServerMemoryManagerSuite.scala  |    55 +
 .../deploy/history/HistoryServerPageSuite.scala    |   119 +
 .../spark/deploy/history/HistoryServerSuite.scala  |   862 +
 .../spark/deploy/history/HybridStoreSuite.scala    |   251 +
 .../history/RealBrowserUIHistoryServerSuite.scala  |   162 +
 .../deploy/master/CustomRecoveryModeFactory.scala  |   105 +
 .../deploy/master/MasterDecommissionSuite.scala    |    62 +
 .../apache/spark/deploy/master/MasterSuite.scala   |   252 +
 .../spark/deploy/master/MasterSuiteBase.scala      |   598 +
 .../spark/deploy/master/MasterWorkerUISuite.scala  |   161 +
 .../deploy/master/PersistenceEngineBenchmark.scala |   185 +
 .../deploy/master/PersistenceEngineSuite.scala     |   202 +
 .../apache/spark/deploy/master/RecoverySuite.scala |   588 +
 .../deploy/master/ResourceProfilesSuite.scala      |    66 +
 .../spark/deploy/master/WorkerSelectionSuite.scala |    81 +
 .../deploy/master/ui/ApplicationPageSuite.scala    |    86 +
 .../spark/deploy/master/ui/MasterWebUISuite.scala  |   140 +
 .../master/ui/ReadOnlyMasterWebUISuite.scala       |   122 +
 .../deploy/rest/StandaloneRestSubmitSuite.scala    |   895 +
 .../deploy/rest/SubmitRestProtocolSuite.scala      |   327 +
 .../HadoopDelegationTokenManagerSuite.scala        |   122 +
 .../HadoopFSDelegationTokenProviderSuite.scala     |    45 +
 .../spark/deploy/worker/CommandUtilsSuite.scala    |    97 +
 .../spark/deploy/worker/DriverRunnerTest.scala     |   211 +
 .../spark/deploy/worker/ExecutorRunnerTest.scala   |    42 +
 .../spark/deploy/worker/WorkerArgumentsTest.scala  |    59 +
 .../apache/spark/deploy/worker/WorkerSuite.scala   |   446 +
 .../spark/deploy/worker/WorkerWatcherSuite.scala   |    46 +
 .../spark/deploy/worker/ui/LogPageSuite.scala      |    88 +
 .../spark/executor/ClassLoaderIsolationSuite.scala |   253 +
 .../CoarseGrainedExecutorBackendSuite.scala        |   658 +
 .../spark/executor/ExecutorClassLoaderSuite.scala  |   355 +
 .../executor/ExecutorMetricsPollerSuite.scala      |    48 +
 .../ExecutorSideSessionManagementSuite.scala       |   233 +
 .../org/apache/spark/executor/ExecutorSuite.scala  |   928 +
 .../spark/executor/ProcfsMetricsGetterSuite.scala  |   114 +
 .../apache/spark/executor/TaskMetricsSuite.scala   |   247 +
 .../input/WholeTextFileInputFormatSuite.scala      |    86 +
 .../input/WholeTextFileRecordReaderSuite.scala     |   151 +
 .../org/apache/spark/internal/LoggingSuite.scala   |    73 +
 .../spark/internal/config/ConfigEntrySuite.scala   |   494 +
 .../spark/internal/config/ConfigReaderSuite.scala  |    62 +
 .../io/FileCommitProtocolInstantiationSuite.scala  |   148 +
 .../internal/io/SparkHadoopWriterUtilsSuite.scala  |   102 +
 .../internal/plugin/PluginContainerSuite.scala     |   531 +
 .../io/ChunkedByteBufferFileRegionSuite.scala      |   150 +
 .../apache/spark/io/ChunkedByteBufferSuite.scala   |   152 +
 .../apache/spark/io/CompressionCodecSuite.scala    |   177 +
 .../apache/spark/io/LZ4TPCDSDataBenchmark.scala    |    78 +
 .../scala/org/apache/spark/io/LZFBenchmark.scala   |    93 +
 .../org/apache/spark/io/TPCDSDataBenchmark.scala   |    64 +
 .../org/apache/spark/io/ZStandardBenchmark.scala   |   157 +
 .../spark/io/ZStandardTPCDSDataBenchmark.scala     |   116 +
 .../spark/launcher/LauncherBackendSuite.scala      |    85 +
 .../apache/spark/memory/MemoryManagerSuite.scala   |   360 +
 .../apache/spark/memory/MemoryTestingUtils.scala   |    41 +
 .../apache/spark/memory/TestMemoryManager.scala    |   134 +
 .../spark/memory/TestMemoryManagerSuite.scala      |    55 +
 .../spark/memory/UnifiedMemoryManagerSuite.scala   |   630 +
 .../spark/metrics/InputOutputMetricsSuite.scala    |   419 +
 .../apache/spark/metrics/MetricsConfigSuite.scala  |   167 +
 .../apache/spark/metrics/MetricsSystemSuite.scala  |   293 +
 .../spark/metrics/sink/GraphiteSinkSuite.scala     |   122 +
 .../metrics/sink/PrometheusServletSuite.scala      |    73 +
 .../spark/metrics/sink/StatsdSinkSuite.scala       |   205 +
 .../metrics/source/AccumulatorSourceSuite.scala    |    90 +
 .../spark/metrics/source/SourceConfigSuite.scala   |    95 +
 .../spark/network/BlockTransferServiceSuite.scala  |   106 +
 .../network/netty/NettyBlockRpcServerSuite.scala   |    59 +
 .../netty/NettyBlockTransferSecuritySuite.scala    |   194 +
 .../netty/NettyBlockTransferServiceSuite.scala     |   152 +
 .../network/netty/SparkTransportConfSuite.scala    |    80 +
 .../apache/spark/partial/CountEvaluatorSuite.scala |    43 +
 .../apache/spark/partial/MeanEvaluatorSuite.scala  |    57 +
 .../apache/spark/partial/SumEvaluatorSuite.scala   |    77 +
 .../apache/spark/rdd/AsyncRDDActionsSuite.scala    |   239 +
 .../apache/spark/rdd/CoalescedRDDBenchmark.scala   |    87 +
 .../org/apache/spark/rdd/DoubleRDDSuite.scala      |   313 +
 .../org/apache/spark/rdd/HadoopRDDSuite.scala      |    30 +
 .../scala/org/apache/spark/rdd/JdbcRDDSuite.scala  |   110 +
 .../apache/spark/rdd/LocalCheckpointSuite.scala    |   334 +
 .../apache/spark/rdd/PairRDDFunctionsSuite.scala   |  1031 +
 .../spark/rdd/ParallelCollectionSplitSuite.scala   |   257 +
 .../spark/rdd/PartitionPruningRDDSuite.scala       |    76 +
 .../spark/rdd/PartitionwiseSampledRDDSuite.scala   |    61 +
 .../scala/org/apache/spark/rdd/PipedRDDSuite.scala |   281 +
 .../org/apache/spark/rdd/RDDBarrierSuite.scala     |    52 +
 .../org/apache/spark/rdd/RDDCleanerSuite.scala     |    60 +
 .../apache/spark/rdd/RDDOperationScopeSuite.scala  |   142 +
 .../test/scala/org/apache/spark/rdd/RDDSuite.scala |  1410 +
 .../scala/org/apache/spark/rdd/RDDSuiteUtils.scala |    31 +
 .../scala/org/apache/spark/rdd/SortingSuite.scala  |   152 +
 .../apache/spark/rdd/ZippedPartitionsSuite.scala   |    41 +
 .../resource/ResourceDiscoveryPluginSuite.scala    |   189 +
 .../spark/resource/ResourceInformationSuite.scala  |    64 +
 .../resource/ResourceProfileManagerSuite.scala     |   223 +
 .../spark/resource/ResourceProfileSuite.scala      |   432 +
 .../apache/spark/resource/ResourceUtilsSuite.scala |   586 +
 .../apache/spark/resource/TestResourceIDs.scala    |    35 +
 .../org/apache/spark/rpc/RpcAddressSuite.scala     |    83 +
 .../scala/org/apache/spark/rpc/RpcEnvSuite.scala   |  1005 +
 .../org/apache/spark/rpc/TestRpcEndpoint.scala     |   124 +
 .../org/apache/spark/rpc/netty/InboxSuite.scala    |   152 +
 .../spark/rpc/netty/NettyRpcAddressSuite.scala     |    35 +
 .../apache/spark/rpc/netty/NettyRpcEnvSuite.scala  |   194 +
 .../spark/rpc/netty/NettyRpcHandlerSuite.scala     |    68 +
 .../apache/spark/scheduler/AQEShuffledRDD.scala    |   118 +
 .../spark/scheduler/AdaptiveSchedulingSuite.scala  |    76 +
 .../spark/scheduler/BarrierTaskContextSuite.scala  |   393 +
 .../CoarseGrainedSchedulerBackendSuite.scala       |   680 +
 .../apache/spark/scheduler/DAGSchedulerSuite.scala |  6182 ++++
 .../scheduler/EventLoggingListenerSuite.scala      |   704 +
 .../scheduler/ExecutorResourceInfoSuite.scala      |   190 +
 .../spark/scheduler/ExecutorResourceUtils.scala    |    28 +
 .../scheduler/ExecutorResourcesAmountsSuite.scala  |   537 +
 .../scheduler/ExternalClusterManagerSuite.scala    |   108 +
 .../org/apache/spark/scheduler/FakeTask.scala      |   142 +
 .../scheduler/HealthTrackerIntegrationSuite.scala  |   157 +
 .../spark/scheduler/HealthTrackerSuite.scala       |   675 +
 .../apache/spark/scheduler/JobWaiterSuite.scala    |    41 +
 .../apache/spark/scheduler/MapStatusSuite.scala    |   292 +
 .../spark/scheduler/NotSerializableFakeTask.scala  |    42 +
 .../OutputCommitCoordinatorIntegrationSuite.scala  |    66 +
 .../scheduler/OutputCommitCoordinatorSuite.scala   |   334 +
 .../org/apache/spark/scheduler/PoolSuite.scala     |   378 +
 .../spark/scheduler/ReplayListenerSuite.scala      |   274 +
 .../scheduler/SchedulerIntegrationSuite.scala      |   759 +
 .../spark/scheduler/SparkListenerSuite.scala       |   846 +
 .../scheduler/SparkListenerWithClusterSuite.scala  |    65 +
 .../apache/spark/scheduler/TaskContextSuite.scala  |   762 +
 .../spark/scheduler/TaskDescriptionSuite.scala     |   108 +
 .../spark/scheduler/TaskResultGetterSuite.scala    |   322 +
 .../spark/scheduler/TaskSchedulerImplSuite.scala   |  2716 ++
 .../spark/scheduler/TaskSetExcludelistSuite.scala  |   309 +
 .../spark/scheduler/TaskSetManagerSuite.scala      |  2772 ++
 .../WorkerDecommissionExtendedSuite.scala          |    73 +
 .../spark/scheduler/WorkerDecommissionSuite.scala  |    86 +
 .../scheduler/dynalloc/ExecutorMonitorSuite.scala  |   555 +
 .../spark/security/CryptoStreamUtilsSuite.scala    |   176 +
 .../apache/spark/security/EncryptionFunSuite.scala |    43 +
 .../spark/security/SocketAuthHelperSuite.scala     |   103 +
 .../serializer/GenericAvroSerializerSuite.scala    |   129 +
 .../spark/serializer/JavaSerializerSuite.scala     |    73 +
 .../apache/spark/serializer/KryoBenchmark.scala    |   134 +
 .../spark/serializer/KryoIteratorBenchmark.scala   |   120 +
 .../spark/serializer/KryoSerializerBenchmark.scala |    96 +
 .../KryoSerializerDistributedSuite.scala           |    68 +
 .../KryoSerializerResizableOutputSuite.scala       |    51 +
 .../spark/serializer/KryoSerializerSuite.scala     |   678 +
 .../ProactiveClosureSerializationSuite.scala       |    81 +
 .../serializer/SerializationDebuggerSuite.scala    |   280 +
 .../serializer/SerializerPropertiesSuite.scala     |   120 +
 .../apache/spark/serializer/TestSerializer.scala   |    63 +
 .../serializer/UnsafeKryoSerializerSuite.scala     |    35 +
 .../shuffle/BlockStoreShuffleReaderSuite.scala     |   154 +
 .../apache/spark/shuffle/ChecksumBenchmark.scala   |    54 +
 .../shuffle/HostLocalShuffleReadingSuite.scala     |   146 +
 .../spark/shuffle/ShuffleBlockPusherSuite.scala    |   493 +
 .../spark/shuffle/ShuffleChecksumTestHelper.scala  |    54 +
 .../spark/shuffle/ShuffleDependencySuite.scala     |    67 +
 .../shuffle/ShuffleDriverComponentsSuite.scala     |    92 +
 .../checksum/OutputStreamRowBasedChecksum.scala    |    64 +
 .../sort/BypassMergeSortShuffleWriterSuite.scala   |   349 +
 .../sort/IndexShuffleBlockResolverSuite.scala      |   287 +
 .../shuffle/sort/ShuffleExternalSorterSuite.scala  |   117 +
 .../shuffle/sort/SortShuffleManagerSuite.scala     |   134 +
 .../shuffle/sort/SortShuffleWriterSuite.scala      |   235 +
 .../io/LocalDiskShuffleMapOutputWriterSuite.scala  |   150 +
 .../spark/status/AppStatusListenerSuite.scala      |  1996 ++
 .../apache/spark/status/AppStatusStoreSuite.scala  |   348 +
 .../apache/spark/status/AppStatusUtilsSuite.scala  |   108 +
 .../spark/status/AutoCleanupLiveUIDirSuite.scala   |    56 +
 .../spark/status/ElementTrackingStoreSuite.scala   |   137 +
 .../spark/status/ListenerEventsTestHelper.scala    |   208 +
 .../org/apache/spark/status/LiveEntitySuite.scala  |   211 +
 .../spark/status/api/v1/ExecutorSummarySuite.scala |    53 +
 .../spark/status/api/v1/SimpleDateParamSuite.scala |    37 +
 .../protobuf/KVStoreProtobufSerializerSuite.scala  |  1700 ++
 .../org/apache/spark/storage/BlockIdSuite.scala    |   253 +
 .../spark/storage/BlockInfoManagerSuite.scala      |   426 +
 .../BlockManagerDecommissionIntegrationSuite.scala |   434 +
 .../BlockManagerDecommissionUnitSuite.scala        |   426 +
 .../spark/storage/BlockManagerInfoSuite.scala      |   157 +
 .../spark/storage/BlockManagerMasterSuite.scala    |    95 +
 .../storage/BlockManagerReplicationSuite.scala     |   612 +
 .../apache/spark/storage/BlockManagerSuite.scala   |  2782 ++
 .../storage/BlockReplicationPolicySuite.scala      |   138 +
 .../spark/storage/DiskBlockManagerSuite.scala      |   189 +
 .../spark/storage/DiskBlockObjectWriterSuite.scala |   278 +
 .../org/apache/spark/storage/DiskStoreSuite.scala  |   205 +
 .../spark/storage/FallbackStorageSuite.scala       |   417 +
 .../spark/storage/FlatmapIteratorSuite.scala       |    68 +
 .../org/apache/spark/storage/LocalDirsSuite.scala  |    88 +
 .../apache/spark/storage/LogBlockWriterSuite.scala |   137 +
 .../apache/spark/storage/MemoryStoreSuite.scala    |   643 +
 .../storage/PartiallySerializedBlockSuite.scala    |   218 +
 .../storage/PartiallyUnrolledIteratorSuite.scala   |    61 +
 .../storage/ShuffleBlockFetcherIteratorSuite.scala |  2079 ++
 .../storage/SslBlockManagerReplicationSuite.scala  |    40 +
 .../org/apache/spark/storage/StorageSuite.scala    |   170 +
 .../apache/spark/storage/TopologyMapperSuite.scala |    70 +
 .../apache/spark/ui/ChromeUISeleniumSuite.scala    |    50 +
 .../org/apache/spark/ui/DriverLogPageSuite.scala   |    65 +
 .../apache/spark/ui/HttpSecurityFilterSuite.scala  |   185 +
 .../scala/org/apache/spark/ui/JWSFilterSuite.scala |   112 +
 .../org/apache/spark/ui/PagedTableSuite.scala      |   125 +
 .../spark/ui/RealBrowserUISeleniumSuite.scala      |   210 +
 .../org/apache/spark/ui/UISeleniumSuite.scala      |   923 +
 .../test/scala/org/apache/spark/ui/UISuite.scala   |   556 +
 .../scala/org/apache/spark/ui/UIUtilsSuite.scala   |   228 +
 .../apache/spark/ui/env/EnvironmentPageSuite.scala |    52 +
 .../spark/ui/scope/RDDOperationGraphSuite.scala    |    47 +
 .../apache/spark/ui/storage/StoragePageSuite.scala |   248 +
 .../org/apache/spark/util/AccumulatorV2Suite.scala |   132 +
 .../apache/spark/util/ArrayImplicitsSuite.scala    |    50 +
 .../apache/spark/util/BestEffortLazyValSuite.scala |   120 +
 .../spark/util/BoundedPriorityQueueSuite.scala     |    51 +
 .../org/apache/spark/util/CausedBySuite.scala      |    56 +
 .../apache/spark/util/ClosureCleanerSuite.scala    |   396 +
 .../apache/spark/util/ClosureCleanerSuite2.scala   |   294 +
 .../spark/util/CompletionIteratorSuite.scala       |    69 +
 .../apache/spark/util/DependencyUtilsSuite.scala   |    66 +
 .../util/DirectByteBufferOutputStreamSuite.scala   |    41 +
 .../org/apache/spark/util/DistributionSuite.scala  |    40 +
 .../org/apache/spark/util/EventLoopSuite.scala     |   281 +
 .../org/apache/spark/util/FileAppenderSuite.scala  |   419 +
 .../org/apache/spark/util/HadoopFSUtilsSuite.scala |    65 +
 .../org/apache/spark/util/JsonProtocolSuite.scala  |  3205 ++
 .../scala/org/apache/spark/util/KeyLockSuite.scala |   118 +
 .../scala/org/apache/spark/util/LazyTrySuite.scala |   151 +
 .../spark/util/MutableURLClassLoaderSuite.scala    |   153 +
 .../org/apache/spark/util/NextIteratorSuite.scala  |    89 +
 .../spark/util/NonFateSharingCacheSuite.scala      |   161 +
 .../spark/util/PeriodicRDDCheckpointerSuite.scala  |   175 +
 .../spark/util/PropertiesCloneBenchmark.scala      |    74 +
 .../apache/spark/util/ResetSystemProperties.scala  |    61 +
 .../org/apache/spark/util/SizeEstimatorSuite.scala |   246 +
 .../org/apache/spark/util/SparkConfWithEnv.scala   |    32 +
 .../util/SparkUncaughtExceptionHandlerSuite.scala  |   161 +
 .../scala/org/apache/spark/util/SslTestUtils.scala |    36 +
 .../org/apache/spark/util/ThreadUtilsSuite.scala   |   232 +
 .../util/TransientBestEffortLazyValSuite.scala     |   116 +
 .../util/UninterruptibleThreadRunnerSuite.scala    |    64 +
 .../spark/util/UninterruptibleThreadSuite.scala    |   209 +
 .../scala/org/apache/spark/util/UtilsSuite.scala   |  1883 ++
 .../spark/util/collection/AppendOnlyMapSuite.scala |   195 +
 .../apache/spark/util/collection/BitSetSuite.scala |   217 +
 .../spark/util/collection/CompactBufferSuite.scala |   105 +
 .../collection/ExternalAppendOnlyMapSuite.scala    |   568 +
 .../util/collection/ExternalSorterSpillSuite.scala |   146 +
 .../util/collection/ExternalSorterSuite.scala      |   648 +
 .../spark/util/collection/FixedHashObject.scala    |    29 +
 .../util/collection/ImmutableBitSetSuite.scala     |   140 +
 .../spark/util/collection/OpenHashMapSuite.scala   |   282 +
 .../spark/util/collection/OpenHashSetSuite.scala   |   311 +
 .../util/collection/PercentileHeapBenchmark.scala  |    74 +
 .../util/collection/PercentileHeapSuite.scala      |    60 +
 .../util/collection/PrimitiveVectorSuite.scala     |   116 +
 .../spark/util/collection/SizeTrackerSuite.scala   |   209 +
 .../spark/util/collection/SorterBenchmark.scala    |   232 +
 .../apache/spark/util/collection/SorterSuite.scala |   264 +
 .../unsafe/sort/PrefixComparatorsSuite.scala       |   168 +
 .../collection/unsafe/sort/RadixSortSuite.scala    |   199 +
 .../io/ChunkedByteBufferOutputStreamSuite.scala    |   133 +
 .../spark/util/kvstore/RocksDBBenchmark.scala      |   547 +
 .../spark/util/logging/DriverLoggerSuite.scala     |   118 +
 .../spark/util/random/RandomSamplerSuite.scala     |   742 +
 .../spark/util/random/SamplingUtilsSuite.scala     |    81 +
 .../util/random/XORShiftRandomBenchmark.scala      |   129 +
 .../spark/util/random/XORShiftRandomSuite.scala    |    77 +
 .../scala/org/apache/sparktest/ImplicitSuite.scala |   116 +
 data/artifact-tests/crc/README.md                  |     5 +
 data/artifact-tests/crc/junitLargeJar.txt          |    12 +
 data/artifact-tests/crc/smallJar.txt               |     1 +
 data/artifact-tests/junitLargeJar.jar              |   Bin 0 -> 384581 bytes
 data/artifact-tests/smallJar.jar                   |   Bin 0 -> 787 bytes
 data/graphx/followers.txt                          |     8 +
 data/graphx/users.txt                              |     7 +
 data/mllib/als/sample_movielens_ratings.txt        |  1501 +
 data/mllib/als/test.data                           |    16 +
 data/mllib/gmm_data.txt                            |  2000 ++
 data/mllib/images/license.txt                      |    13 +
 .../images/origin/kittens/29.5.a_b_EGDP022204.jpg  |   Bin 0 -> 27295 bytes
 data/mllib/images/origin/kittens/54893.jpg         |   Bin 0 -> 35914 bytes
 data/mllib/images/origin/kittens/DP153539.jpg      |   Bin 0 -> 26354 bytes
 data/mllib/images/origin/kittens/DP802813.jpg      |   Bin 0 -> 30432 bytes
 data/mllib/images/origin/kittens/not-image.txt     |     1 +
 data/mllib/images/origin/license.txt               |    13 +
 data/mllib/images/origin/multi-channel/BGRA.png    |   Bin 0 -> 683 bytes
 .../images/origin/multi-channel/BGRA_alpha_60.png  |   Bin 0 -> 747 bytes
 .../images/origin/multi-channel/chr30.4.184.jpg    |   Bin 0 -> 59472 bytes
 .../images/origin/multi-channel/grayscale.jpg      |   Bin 0 -> 36728 bytes
 data/mllib/kmeans_data.txt                         |     6 +
 data/mllib/pagerank_data.txt                       |     6 +
 data/mllib/pic_data.txt                            |    19 +
 data/mllib/ridge-data/lpsa.data                    |    67 +
 data/mllib/sample_binary_classification_data.txt   |   100 +
 data/mllib/sample_fpgrowth.txt                     |     6 +
 .../sample_isotonic_regression_libsvm_data.txt     |   100 +
 data/mllib/sample_kmeans_data.txt                  |     6 +
 data/mllib/sample_lda_data.txt                     |    12 +
 data/mllib/sample_lda_libsvm_data.txt              |    12 +
 data/mllib/sample_libsvm_data.txt                  |   100 +
 data/mllib/sample_linear_regression_data.txt       |   501 +
 data/mllib/sample_movielens_data.txt               |  1501 +
 .../sample_multiclass_classification_data.txt      |   150 +
 data/mllib/sample_svm_data.txt                     |   322 +
 data/mllib/streaming_kmeans_data_test.txt          |     2 +
 data/streaming/AFINN-111.txt                       |  2477 ++
 dev/.rat-excludes                                  |   145 +
 dev/.scalafmt.conf                                 |    30 +
 dev/README.md                                      |     5 +
 dev/change-scala-version.sh                        |   100 +
 dev/check-license                                  |    86 +
 dev/check-protos.py                                |    87 +
 dev/check_pyspark_custom_errors.py                 |   189 +
 dev/checkstyle-suppressions.xml                    |    73 +
 dev/checkstyle.xml                                 |   299 +
 dev/connect-gen-protos.sh                          |    27 +
 dev/connect-jvm-client-mima-check                  |    70 +
 dev/create-release/announce.tmpl                   |    16 +
 dev/create-release/do-release-docker.sh            |   180 +
 dev/create-release/do-release.sh                   |    90 +
 dev/create-release/generate-contributors.py        |   189 +
 dev/create-release/generate-llms-txt.py            |   198 +
 dev/create-release/release-build.sh                |  1099 +
 dev/create-release/release-tag.sh                  |   144 +
 dev/create-release/release-util.sh                 |   258 +
 dev/create-release/releaseutils.py                 |   137 +
 dev/create-release/spark-rm/Dockerfile             |    79 +
 dev/create-release/spark-rm/Dockerfile.base        |   110 +
 dev/create-release/vote.tmpl                       |    66 +
 dev/deps/spark-deps-hadoop-3-hive-2.3              |   297 +
 dev/eslint.js                                      |    46 +
 dev/free_disk_space                                |    59 +
 dev/free_disk_space_container                      |    40 +
 dev/gen-protos.sh                                  |   127 +
 dev/infra/Dockerfile                               |   159 +
 dev/is-changed.py                                  |   120 +
 dev/java-file-header                               |    16 +
 dev/lint-java                                      |    30 +
 dev/lint-js                                        |    57 +
 dev/lint-python                                    |   395 +
 dev/lint-r                                         |    44 +
 dev/lint-r.R                                       |    36 +
 dev/lint-scala                                     |    48 +
 dev/make-distribution.sh                           |   337 +
 dev/merge_spark_pr.py                              |   745 +
 dev/mima                                           |    53 +
 dev/package-lock.json                              |  2169 ++
 dev/package.json                                   |    11 +
 dev/pip-sanity-check.py                            |    31 +
 dev/protobuf-breaking-changes-check.sh             |    46 +
 dev/py-cleanup                                     |    31 +
 dev/reformat-python                                |    32 +
 dev/requirements.txt                               |    89 +
 dev/run-pip-tests                                  |   143 +
 dev/run-tests                                      |    29 +
 dev/run-tests.py                                   |   659 +
 dev/sbt-checkstyle                                 |    35 +
 dev/scalafmt                                       |    21 +
 dev/scalastyle                                     |    37 +
 dev/spark-test-image-util/docs/build-docs          |    73 +
 dev/spark-test-image-util/docs/run-in-container    |    37 +
 dev/spark-test-image/docs/Dockerfile               |    96 +
 dev/spark-test-image/lint/Dockerfile               |   106 +
 dev/spark-test-image/pypy-310/Dockerfile           |    59 +
 dev/spark-test-image/pypy-311/Dockerfile           |    59 +
 dev/spark-test-image/python-310/Dockerfile         |    70 +
 dev/spark-test-image/python-311/Dockerfile         |    68 +
 .../python-312-classic-only/Dockerfile             |    67 +
 .../python-312-pandas-3/Dockerfile                 |    71 +
 dev/spark-test-image/python-312/Dockerfile         |    68 +
 dev/spark-test-image/python-313/Dockerfile         |    68 +
 dev/spark-test-image/python-314-nogil/Dockerfile   |    68 +
 dev/spark-test-image/python-314/Dockerfile         |    68 +
 dev/spark-test-image/python-minimum/Dockerfile     |    62 +
 dev/spark-test-image/python-ps-minimum/Dockerfile  |    63 +
 dev/spark-test-image/sparkr/Dockerfile             |    78 +
 dev/sparktestsupport/__init__.py                   |    39 +
 dev/sparktestsupport/modules.py                    |  1687 ++
 dev/sparktestsupport/shellutils.py                 |    89 +
 dev/sparktestsupport/toposort.py                   |    84 +
 dev/sparktestsupport/utils.py                      |   179 +
 dev/streaming-gen-protos.sh                        |    27 +
 dev/structured_logging_style.py                    |    94 +
 dev/test-classes.txt                               |     7 +
 dev/test-dependencies.sh                           |   150 +
 dev/test-jars.txt                                  |    17 +
 dev/tox.ini                                        |    62 +
 docs/.bundle/config                                |     4 +
 docs/.gitignore                                    |     1 +
 docs/404.md                                        |    28 +
 docs/Gemfile                                       |    31 +
 docs/Gemfile.lock                                  |    89 +
 docs/README.md                                     |   141 +
 docs/_config.yml                                   |    49 +
 docs/_data/menu-migration.yaml                     |    12 +
 docs/_data/menu-ml.yaml                            |    20 +
 docs/_data/menu-mllib.yaml                         |    75 +
 docs/_data/menu-sql.yaml                           |   127 +
 docs/_data/menu-streaming.yaml                     |    57 +
 docs/_includes/nav-left-wrapper-migration.html     |     6 +
 docs/_includes/nav-left-wrapper-ml.html            |     8 +
 docs/_includes/nav-left-wrapper-sql.html           |     6 +
 docs/_includes/nav-left-wrapper-streaming.html     |    22 +
 docs/_includes/nav-left.html                       |    18 +
 docs/_layouts/global.html                          |   244 +
 docs/_layouts/redirect.html                        |    29 +
 docs/_plugins/build-error-docs.py                  |   152 +
 docs/_plugins/build_api_docs.rb                    |   237 +
 docs/_plugins/conditonal_includes.rb               |    71 +
 docs/_plugins/include_example.rb                   |   126 +
 docs/_plugins/production_tag.rb                    |    26 +
 docs/app-dev-spark-connect.md                      |   270 +
 docs/building-spark.md                             |   322 +
 docs/cloud-integration.md                          |   398 +
 docs/cluster-overview.md                           |   147 +
 docs/configuration.md                              |  4195 +++
 docs/control-flow/case-stmt.md                     |   102 +
 docs/control-flow/close-stmt.md                    |   147 +
 docs/control-flow/compound-stmt.md                 |   213 +
 docs/control-flow/fetch-stmt.md                    |   194 +
 docs/control-flow/for-stmt.md                      |    92 +
 docs/control-flow/if-stmt.md                       |    70 +
 docs/control-flow/iterate-stmt.md                  |    70 +
 docs/control-flow/leave-stmt.md                    |    71 +
 docs/control-flow/loop-stmt.md                     |    83 +
 docs/control-flow/open-stmt.md                     |   162 +
 docs/control-flow/repeat-stmt.md                   |    84 +
 docs/control-flow/while-stmt.md                    |    83 +
 docs/core-migration-guide.md                       |   136 +
 docs/css/api-docs.css                              |    35 +
 docs/css/api-javadocs.css                          |    52 +
 docs/css/bootstrap.min.css                         |     7 +
 docs/css/bootstrap.min.css.map                     |     1 +
 docs/css/custom.css                                |  1028 +
 docs/css/docsearch.css                             |    36 +
 docs/css/docsearch.min.css                         |     2 +
 docs/css/docsearch.min.css.map                     |     1 +
 docs/css/pygments-default.css                      |    80 +
 docs/declarative-pipelines-programming-guide.md    |   520 +
 docs/graphx-programming-guide.md                   |   991 +
 docs/hadoop-provided.md                            |    63 +
 docs/hardware-provisioning.md                      |    83 +
 docs/img/AllJobsPageDetail1.png                    |   Bin 0 -> 20567 bytes
 docs/img/AllJobsPageDetail2.png                    |   Bin 0 -> 70557 bytes
 docs/img/AllJobsPageDetail3.png                    |   Bin 0 -> 94804 bytes
 docs/img/AllStagesPageDetail1.png                  |   Bin 0 -> 18689 bytes
 docs/img/AllStagesPageDetail2.png                  |   Bin 0 -> 21445 bytes
 docs/img/AllStagesPageDetail3.png                  |   Bin 0 -> 124154 bytes
 docs/img/AllStagesPageDetail4.png                  |   Bin 0 -> 18098 bytes
 docs/img/AllStagesPageDetail5.png                  |   Bin 0 -> 33047 bytes
 docs/img/AllStagesPageDetail6.png                  |   Bin 0 -> 163423 bytes
 docs/img/AllStagesPageDetail7.png                  |   Bin 0 -> 26097 bytes
 docs/img/AllStagesPageDetail8.png                  |   Bin 0 -> 10501 bytes
 docs/img/AllStagesPageDetail9.png                  |   Bin 0 -> 61336 bytes
 docs/img/JDBCServer1.png                           |   Bin 0 -> 14763 bytes
 docs/img/JDBCServer2.png                           |   Bin 0 -> 45084 bytes
 docs/img/JDBCServer3.png                           |   Bin 0 -> 108360 bytes
 docs/img/JobPageDetail1.png                        |   Bin 0 -> 73262 bytes
 docs/img/JobPageDetail2.png                        |   Bin 0 -> 24088 bytes
 docs/img/JobPageDetail3.png                        |   Bin 0 -> 48152 bytes
 docs/img/async-progress.png                        |   Bin 0 -> 29770 bytes
 docs/img/cluster-overview.png                      |   Bin 0 -> 22912 bytes
 docs/img/cluster-overview.pptx                     |   Bin 0 -> 28133 bytes
 docs/img/declarative-pipelines-dataflow-graph.png  |   Bin 0 -> 132100 bytes
 docs/img/edge_cut_vs_vertex_cut.png                |   Bin 0 -> 51015 bytes
 docs/img/extending-spark-connect-labelled.png      |   Bin 0 -> 131995 bytes
 docs/img/extending-spark-connect.png               |   Bin 0 -> 118823 bytes
 docs/img/glyphicons-halflings-white.png            |   Bin 0 -> 8777 bytes
 docs/img/glyphicons-halflings.png                  |   Bin 0 -> 12799 bytes
 docs/img/graphx_figures.pptx                       |   Bin 0 -> 1123363 bytes
 docs/img/graphx_logo.png                           |   Bin 0 -> 22875 bytes
 docs/img/java-sm.png                               |   Bin 0 -> 670 bytes
 docs/img/k8s-cluster-mode.png                      |   Bin 0 -> 55538 bytes
 docs/img/ml-Pipeline.png                           |   Bin 0 -> 38536 bytes
 docs/img/ml-PipelineModel.png                      |   Bin 0 -> 39228 bytes
 docs/img/ml-Pipelines.pptx                         |   Bin 0 -> 56777 bytes
 docs/img/property_graph.png                        |   Bin 0 -> 135699 bytes
 docs/img/pycharm-with-pyspark1.png                 |   Bin 0 -> 160166 bytes
 docs/img/pycharm-with-pyspark2.png                 |   Bin 0 -> 84813 bytes
 docs/img/pycharm-with-pyspark3.png                 |   Bin 0 -> 15981 bytes
 docs/img/pyspark-components.pptx                   |   Bin 0 -> 353804 bytes
 docs/img/pyspark-machine_learning.png              |   Bin 0 -> 8709 bytes
 docs/img/pyspark-pandas_api_on_spark.png           |   Bin 0 -> 8732 bytes
 .../pyspark-pandas_on_spark-transform_apply.pptx   |   Bin 0 -> 139692 bytes
 .../pyspark-pandas_on_spark-transform_apply1.png   |   Bin 0 -> 190159 bytes
 .../pyspark-pandas_on_spark-transform_apply2.png   |   Bin 0 -> 182356 bytes
 .../pyspark-pandas_on_spark-transform_apply3.png   |   Bin 0 -> 166645 bytes
 .../pyspark-pandas_on_spark-transform_apply4.png   |   Bin 0 -> 64346 bytes
 docs/img/pyspark-python_spark_connect_client.png   |   Bin 0 -> 10181 bytes
 docs/img/pyspark-remote-debug1.png                 |   Bin 0 -> 91214 bytes
 docs/img/pyspark-remote-debug2.png                 |   Bin 0 -> 10058 bytes
 docs/img/pyspark-spark_core_and_rdds.png           |   Bin 0 -> 11267 bytes
 docs/img/pyspark-spark_sql_and_dataframes.png      |   Bin 0 -> 9729 bytes
 docs/img/pyspark-structured_streaming.png          |   Bin 0 -> 9111 bytes
 docs/img/pyspark-udf-profile.png                   |   Bin 0 -> 281384 bytes
 docs/img/python-sm.png                             |   Bin 0 -> 1455 bytes
 docs/img/scala-sm.png                              |   Bin 0 -> 2241 bytes
 docs/img/spark-connect-api.png                     |   Bin 0 -> 808246 bytes
 docs/img/spark-connect-communication.png           |   Bin 0 -> 566977 bytes
 docs/img/spark-connect-gotchas-antipattern.png     |   Bin 0 -> 171837 bytes
 docs/img/spark-webui-accumulators.png              |   Bin 0 -> 160167 bytes
 docs/img/streaming-arch.png                        |   Bin 0 -> 51972 bytes
 docs/img/streaming-dstream-ops.png                 |   Bin 0 -> 33495 bytes
 docs/img/streaming-dstream-window.png              |   Bin 0 -> 26622 bytes
 docs/img/streaming-dstream.png                     |   Bin 0 -> 17843 bytes
 docs/img/streaming-figures.pptx                    |   Bin 0 -> 887551 bytes
 docs/img/streaming-flow.png                        |   Bin 0 -> 20425 bytes
 docs/img/streaming-kinesis-arch.png                |   Bin 0 -> 86336 bytes
 docs/img/structured-streaming-example-model.png    |   Bin 0 -> 79409 bytes
 docs/img/structured-streaming-late-data.png        |   Bin 0 -> 91513 bytes
 docs/img/structured-streaming-model.png            |   Bin 0 -> 37321 bytes
 .../img/structured-streaming-stream-as-a-table.png |   Bin 0 -> 47791 bytes
 .../img/structured-streaming-time-window-types.jpg |   Bin 0 -> 56637 bytes
 .../structured-streaming-watermark-append-mode.png |   Bin 0 -> 249196 bytes
 .../structured-streaming-watermark-update-mode.png |   Bin 0 -> 299141 bytes
 docs/img/structured-streaming-window.png           |   Bin 0 -> 88102 bytes
 docs/img/structured-streaming.pptx                 |   Bin 0 -> 1130887 bytes
 docs/img/triplet.png                               |   Bin 0 -> 19255 bytes
 docs/img/type-precedence-list.png                  |   Bin 0 -> 104106 bytes
 docs/img/vertex_routing_edge_tables.png            |   Bin 0 -> 323162 bytes
 docs/img/webui-env-class.png                       |   Bin 0 -> 100687 bytes
 docs/img/webui-env-hadoop.png                      |   Bin 0 -> 107531 bytes
 docs/img/webui-env-sys.png                         |   Bin 0 -> 71739 bytes
 docs/img/webui-env-tab.png                         |   Bin 0 -> 250669 bytes
 docs/img/webui-exe-err.png                         |   Bin 0 -> 718376 bytes
 docs/img/webui-exe-tab.png                         |   Bin 0 -> 554068 bytes
 docs/img/webui-exe-thread.png                      |   Bin 0 -> 201409 bytes
 docs/img/webui-sql-dag.png                         |   Bin 0 -> 74288 bytes
 docs/img/webui-sql-plan.png                        |   Bin 0 -> 458454 bytes
 docs/img/webui-sql-tab.png                         |   Bin 0 -> 372594 bytes
 docs/img/webui-storage-detail.png                  |   Bin 0 -> 190970 bytes
 docs/img/webui-storage-tab.png                     |   Bin 0 -> 78441 bytes
 docs/img/webui-structured-streaming-detail.png     |   Bin 0 -> 179135 bytes
 docs/img/webui-structured-streaming-detail2.png    |   Bin 0 -> 361848 bytes
 docs/index.md                                      |   167 +
 docs/job-scheduling.md                             |   308 +
 docs/js/api-docs.js                                |    66 +
 docs/js/api-javadocs.js                            |    62 +
 docs/js/main.js                                    |   126 +
 docs/js/vendor/anchor.min.js                       |     6 +
 docs/js/vendor/bootstrap.bundle.min.js             |     7 +
 docs/js/vendor/bootstrap.bundle.min.js.map         |     1 +
 docs/js/vendor/docsearch.min.js                    |     2 +
 docs/js/vendor/jquery.min.js                       |     2 +
 .../js/vendor/modernizr-2.6.1-respond-1.1.0.min.js |    11 +
 docs/migration-guide.md                            |    30 +
 docs/ml-advanced.md                                |   111 +
 docs/ml-ann.md                                     |    23 +
 docs/ml-classification-regression.md               |  1464 +
 docs/ml-clustering.md                              |   329 +
 docs/ml-collaborative-filtering.md                 |   204 +
 docs/ml-datasource.md                              |   239 +
 docs/ml-decision-tree.md                           |    23 +
 docs/ml-ensembles.md                               |    23 +
 docs/ml-features.md                                |  2239 ++
 docs/ml-frequent-pattern-mining.md                 |   163 +
 docs/ml-guide.md                                   |   105 +
 docs/ml-linalg-guide.md                            |    95 +
 docs/ml-linear-methods.md                          |    23 +
 docs/ml-migration-guide.md                         |   512 +
 docs/ml-pipeline.md                                |   312 +
 docs/ml-statistics.md                              |   141 +
 docs/ml-survival-regression.md                     |    23 +
 docs/ml-tuning.md                                  |   157 +
 docs/mllib-classification-regression.md            |    56 +
 docs/mllib-clustering.md                           |   480 +
 docs/mllib-collaborative-filtering.md              |   138 +
 docs/mllib-data-types.md                           |   783 +
 docs/mllib-decision-tree.md                        |   258 +
 docs/mllib-dimensionality-reduction.md             |   161 +
 docs/mllib-ensembles.md                            |   303 +
 docs/mllib-evaluation-metrics.md                   |   583 +
 docs/mllib-feature-extraction.md                   |   362 +
 docs/mllib-frequent-pattern-mining.md              |   198 +
 docs/mllib-guide.md                                |    61 +
 docs/mllib-isotonic-regression.md                  |   106 +
 docs/mllib-linear-methods.md                       |   458 +
 docs/mllib-naive-bayes.md                          |    91 +
 docs/mllib-optimization.md                         |   260 +
 docs/mllib-pmml-model-export.md                    |    74 +
 docs/mllib-statistics.md                           |   411 +
 docs/monitoring.md                                 |  1550 +
 docs/programming-guide.md                          |    22 +
 docs/pyspark-migration-guide.md                    |    22 +
 docs/quick-start.md                                |   488 +
 docs/rdd-programming-guide.md                      |  1624 +
 docs/running-on-kubernetes.md                      |  2062 ++
 docs/running-on-yarn.md                            |  1068 +
 docs/security.md                                   |  1135 +
 docs/spark-connect-gotchas.md                      |   432 +
 docs/spark-connect-overview.md                     |   415 +
 docs/spark-standalone.md                           |   882 +
 docs/sparkr-migration-guide.md                     |    84 +
 docs/sparkr.md                                     |   758 +
 docs/sql-data-sources-avro.md                      |   675 +
 docs/sql-data-sources-binaryFile.md                |    72 +
 docs/sql-data-sources-csv.md                       |   288 +
 docs/sql-data-sources-generic-options.md           |   161 +
 docs/sql-data-sources-hive-tables.md               |   212 +
 docs/sql-data-sources-jdbc.md                      |  2220 ++
 docs/sql-data-sources-json.md                      |   283 +
 docs/sql-data-sources-load-save-functions.md       |   384 +
 docs/sql-data-sources-orc.md                       |   248 +
 docs/sql-data-sources-parquet.md                   |   681 +
 docs/sql-data-sources-protobuf.md                  |   502 +
 docs/sql-data-sources-text.md                      |    71 +
 docs/sql-data-sources-troubleshooting.md           |    24 +
 docs/sql-data-sources-xml.md                       |   246 +
 docs/sql-data-sources.md                           |    72 +
 docs/sql-distributed-sql-engine-spark-sql-cli.md   |   191 +
 docs/sql-distributed-sql-engine.md                 |    97 +
 docs/sql-error-conditions.md                       |    37 +
 docs/sql-getting-started.md                        |   363 +
 docs/sql-migration-guide.md                        |  1232 +
 docs/sql-performance-tuning.md                     |   549 +
 docs/sql-pipe-syntax.md                            |   656 +
 docs/sql-programming-guide.md                      |    68 +
 docs/sql-pyspark-pandas-with-arrow.md              |    22 +
 docs/sql-ref-ansi-compliance.md                    |   820 +
 docs/sql-ref-datatypes.md                          |   388 +
 docs/sql-ref-datetime-pattern.md                   |   138 +
 docs/sql-ref-function-invocation.md                |   112 +
 docs/sql-ref-functions-builtin.md                  |   128 +
 docs/sql-ref-functions-udf-aggregate.md            |   127 +
 docs/sql-ref-functions-udf-hive.md                 |   120 +
 docs/sql-ref-functions-udf-scalar.md               |    55 +
 docs/sql-ref-functions.md                          |    63 +
 docs/sql-ref-identifier-clause.md                  |   194 +
 docs/sql-ref-identifier.md                         |    81 +
 docs/sql-ref-literals.md                           |   643 +
 docs/sql-ref-name-resolution.md                    |   423 +
 docs/sql-ref-null-semantics.md                     |   634 +
 docs/sql-ref-number-pattern.md                     |   186 +
 docs/sql-ref-operators.md                          |   124 +
 docs/sql-ref-parameter-markers.md                  |   228 +
 docs/sql-ref-scripting.md                          |   101 +
 docs/sql-ref-sketch-aggregates.md                  |   988 +
 docs/sql-ref-syntax-aux-analyze-table.md           |   185 +
 docs/sql-ref-syntax-aux-cache-cache-table.md       |    86 +
 docs/sql-ref-syntax-aux-cache-clear-cache.md       |    44 +
 docs/sql-ref-syntax-aux-cache-refresh-function.md  |    60 +
 docs/sql-ref-syntax-aux-cache-refresh-table.md     |    60 +
 docs/sql-ref-syntax-aux-cache-refresh.md           |    60 +
 docs/sql-ref-syntax-aux-cache-uncache-table.md     |    55 +
 docs/sql-ref-syntax-aux-cache.md                   |    27 +
 docs/sql-ref-syntax-aux-conf-mgmt-reset.md         |    56 +
 docs/sql-ref-syntax-aux-conf-mgmt-set-timezone.md  |    67 +
 docs/sql-ref-syntax-aux-conf-mgmt-set.md           |    74 +
 docs/sql-ref-syntax-aux-conf-mgmt.md               |    24 +
 docs/sql-ref-syntax-aux-describe-database.md       |    95 +
 docs/sql-ref-syntax-aux-describe-function.md       |   111 +
 docs/sql-ref-syntax-aux-describe-query.md          |   110 +
 docs/sql-ref-syntax-aux-describe-table.md          |   283 +
 docs/sql-ref-syntax-aux-describe.md                |    25 +
 docs/sql-ref-syntax-aux-exec-imm.md                |   113 +
 ...sql-ref-syntax-aux-resource-mgmt-add-archive.md |    54 +
 docs/sql-ref-syntax-aux-resource-mgmt-add-file.md  |    55 +
 docs/sql-ref-syntax-aux-resource-mgmt-add-jar.md   |    68 +
 ...ql-ref-syntax-aux-resource-mgmt-list-archive.md |    53 +
 docs/sql-ref-syntax-aux-resource-mgmt-list-file.md |    53 +
 docs/sql-ref-syntax-aux-resource-mgmt-list-jar.md  |    54 +
 docs/sql-ref-syntax-aux-resource-mgmt.md           |    27 +
 docs/sql-ref-syntax-aux-set-var.md                 |    98 +
 docs/sql-ref-syntax-aux-show-columns.md            |    97 +
 docs/sql-ref-syntax-aux-show-create-table.md       |    86 +
 docs/sql-ref-syntax-aux-show-databases.md          |    88 +
 docs/sql-ref-syntax-aux-show-functions.md          |   134 +
 docs/sql-ref-syntax-aux-show-partitions.md         |   110 +
 docs/sql-ref-syntax-aux-show-table.md              |   180 +
 docs/sql-ref-syntax-aux-show-tables.md             |   106 +
 docs/sql-ref-syntax-aux-show-tblproperties.md      |   113 +
 docs/sql-ref-syntax-aux-show-views.md              |   117 +
 docs/sql-ref-syntax-aux-show.md                    |    30 +
 docs/sql-ref-syntax-ddl-alter-database.md          |   138 +
 docs/sql-ref-syntax-ddl-alter-table.md             |   702 +
 docs/sql-ref-syntax-ddl-alter-view.md              |   250 +
 docs/sql-ref-syntax-ddl-create-database.md         |    87 +
 docs/sql-ref-syntax-ddl-create-function.md         |   167 +
 docs/sql-ref-syntax-ddl-create-sql-function.md     |   303 +
 docs/sql-ref-syntax-ddl-create-table-datasource.md |   172 +
 docs/sql-ref-syntax-ddl-create-table-hiveformat.md |   195 +
 docs/sql-ref-syntax-ddl-create-table-like.md       |    88 +
 docs/sql-ref-syntax-ddl-create-table.md            |    35 +
 docs/sql-ref-syntax-ddl-create-view.md             |   105 +
 docs/sql-ref-syntax-ddl-declare-variable.md        |    93 +
 docs/sql-ref-syntax-ddl-drop-database.md           |    68 +
 docs/sql-ref-syntax-ddl-drop-function.md           |   102 +
 docs/sql-ref-syntax-ddl-drop-table.md              |    80 +
 docs/sql-ref-syntax-ddl-drop-variable.md           |    67 +
 docs/sql-ref-syntax-ddl-drop-view.md               |    70 +
 docs/sql-ref-syntax-ddl-repair-table.md            |    78 +
 docs/sql-ref-syntax-ddl-truncate-table.md          |    90 +
 docs/sql-ref-syntax-ddl-usedb.md                   |    57 +
 ...ql-ref-syntax-dml-insert-overwrite-directory.md |   113 +
 docs/sql-ref-syntax-dml-insert-table.md            |   503 +
 docs/sql-ref-syntax-dml-load.md                    |   115 +
 docs/sql-ref-syntax-hive-format.md                 |    73 +
 docs/sql-ref-syntax-qry-explain.md                 |   127 +
 docs/sql-ref-syntax-qry-select-aggregate.md        |   141 +
 docs/sql-ref-syntax-qry-select-case.md             |   111 +
 docs/sql-ref-syntax-qry-select-clusterby.md        |   106 +
 docs/sql-ref-syntax-qry-select-cte.md              |   120 +
 docs/sql-ref-syntax-qry-select-distribute-by.md    |   101 +
 docs/sql-ref-syntax-qry-select-file.md             |    76 +
 docs/sql-ref-syntax-qry-select-groupby.md          |   321 +
 docs/sql-ref-syntax-qry-select-having.md           |   132 +
 docs/sql-ref-syntax-qry-select-hints.md            |   165 +
 docs/sql-ref-syntax-qry-select-inline-table.md     |    77 +
 docs/sql-ref-syntax-qry-select-join.md             |   239 +
 docs/sql-ref-syntax-qry-select-lateral-subquery.md |    87 +
 docs/sql-ref-syntax-qry-select-lateral-view.md     |   127 +
 docs/sql-ref-syntax-qry-select-like.md             |   177 +
 docs/sql-ref-syntax-qry-select-limit.md            |   111 +
 docs/sql-ref-syntax-qry-select-offset.md           |   101 +
 docs/sql-ref-syntax-qry-select-orderby.md          |   150 +
 docs/sql-ref-syntax-qry-select-pivot.md            |   103 +
 docs/sql-ref-syntax-qry-select-sampling.md         |    94 +
 docs/sql-ref-syntax-qry-select-setops.md           |   196 +
 docs/sql-ref-syntax-qry-select-sortby.md           |   183 +
 docs/sql-ref-syntax-qry-select-subqueries.md       |    22 +
 docs/sql-ref-syntax-qry-select-transform.md        |   268 +
 docs/sql-ref-syntax-qry-select-tvf.md              |   176 +
 docs/sql-ref-syntax-qry-select-unpivot.md          |   144 +
 docs/sql-ref-syntax-qry-select-where.md            |   132 +
 docs/sql-ref-syntax-qry-select-window.md           |   224 +
 docs/sql-ref-syntax-qry-select.md                  |   209 +
 docs/sql-ref-syntax-qry-star.md                    |    97 +
 docs/sql-ref-syntax.md                             |   136 +
 docs/sql-ref.md                                    |    50 +
 docs/ss-migration-guide.md                         |    23 +
 docs/storage-openstack-swift.md                    |   153 +
 docs/streaming-custom-receivers.md                 |   273 +
 docs/streaming-kafka-0-10-integration.md           |   339 +
 docs/streaming-kafka-integration.md                |    26 +
 docs/streaming-kinesis-integration.md              |   306 +
 docs/streaming-programming-guide.md                |  2565 ++
 docs/streaming/additional-information.md           |    58 +
 docs/streaming/apis-on-dataframes-and-datasets.md  |  3596 +++
 docs/streaming/getting-started.md                  |   509 +
 docs/streaming/index.md                            |    28 +
 docs/streaming/performance-tips.md                 |   174 +
 docs/streaming/ss-migration-guide.md               |    61 +
 .../structured-streaming-kafka-integration.md      |  1187 +
 .../structured-streaming-state-data-source.md      |   418 +
 .../structured-streaming-transform-with-state.md   |   399 +
 docs/structured-streaming-kafka-integration.md     |    22 +
 docs/structured-streaming-programming-guide.md     |    22 +
 docs/submitting-applications.md                    |   229 +
 docs/tuning.md                                     |   336 +
 docs/web-ui.md                                     |   493 +
 examples/pom.xml                                   |   154 +
 .../java/org/apache/spark/examples/JavaHdfsLR.java |   159 +
 .../org/apache/spark/examples/JavaLogQuery.java    |   122 +
 .../org/apache/spark/examples/JavaPageRank.java    |   122 +
 .../org/apache/spark/examples/JavaSparkPi.java     |    60 +
 .../spark/examples/JavaStatusTrackerDemo.java      |    74 +
 .../java/org/apache/spark/examples/JavaTC.java     |    98 +
 .../org/apache/spark/examples/JavaWordCount.java   |    59 +
 .../ml/JavaAFTSurvivalRegressionExample.java       |    82 +
 .../apache/spark/examples/ml/JavaALSExample.java   |   136 +
 .../spark/examples/ml/JavaBinarizerExample.java    |    68 +
 .../examples/ml/JavaBisectingKMeansExample.java    |    73 +
 .../ml/JavaBucketedRandomProjectionLSHExample.java |   110 +
 .../spark/examples/ml/JavaBucketizerExample.java   |   116 +
 .../examples/ml/JavaChiSqSelectorExample.java      |    74 +
 .../examples/ml/JavaChiSquareTestExample.java      |    75 +
 .../spark/examples/ml/JavaCorrelationExample.java  |    72 +
 .../examples/ml/JavaCountVectorizerExample.java    |    69 +
 .../apache/spark/examples/ml/JavaDCTExample.java   |    68 +
 .../ml/JavaDecisionTreeClassificationExample.java  |   104 +
 .../ml/JavaDecisionTreeRegressionExample.java      |    89 +
 .../org/apache/spark/examples/ml/JavaDocument.java |    43 +
 .../examples/ml/JavaElementwiseProductExample.java |    73 +
 .../ml/JavaEstimatorTransformerParamExample.java   |   117 +
 .../spark/examples/ml/JavaFMClassifierExample.java |   105 +
 .../spark/examples/ml/JavaFMRegressorExample.java  |    90 +
 .../spark/examples/ml/JavaFPGrowthExample.java     |    77 +
 .../examples/ml/JavaFeatureHasherExample.java      |    69 +
 .../examples/ml/JavaGaussianMixtureExample.java    |    64 +
 .../ml/JavaGeneralizedLinearRegressionExample.java |    83 +
 .../JavaGradientBoostedTreeClassifierExample.java  |   105 +
 .../JavaGradientBoostedTreeRegressorExample.java   |    90 +
 .../spark/examples/ml/JavaImputerExample.java      |    71 +
 .../examples/ml/JavaIndexToStringExample.java      |    87 +
 .../spark/examples/ml/JavaInteractionExample.java  |    87 +
 .../examples/ml/JavaIsotonicRegressionExample.java |    62 +
 .../spark/examples/ml/JavaKMeansExample.java       |    74 +
 .../apache/spark/examples/ml/JavaLDAExample.java   |    69 +
 .../spark/examples/ml/JavaLabeledDocument.java     |    38 +
 .../JavaLinearRegressionWithElasticNetExample.java |    65 +
 .../spark/examples/ml/JavaLinearSVCExample.java    |    54 +
 .../ml/JavaLogisticRegressionSummaryExample.java   |    77 +
 ...avaLogisticRegressionWithElasticNetExample.java |    69 +
 .../spark/examples/ml/JavaMaxAbsScalerExample.java |    73 +
 .../spark/examples/ml/JavaMinHashLSHExample.java   |   111 +
 .../spark/examples/ml/JavaMinMaxScalerExample.java |    74 +
 ...avaModelSelectionViaCrossValidationExample.java |   122 +
 ...delSelectionViaTrainValidationSplitExample.java |    88 +
 ...assLogisticRegressionWithElasticNetExample.java |   117 +
 .../JavaMultilayerPerceptronClassifierExample.java |    76 +
 .../apache/spark/examples/ml/JavaNGramExample.java |    66 +
 .../spark/examples/ml/JavaNaiveBayesExample.java   |    70 +
 .../spark/examples/ml/JavaNormalizerExample.java   |    74 +
 .../examples/ml/JavaOneHotEncoderExample.java      |    74 +
 .../spark/examples/ml/JavaOneVsRestExample.java    |    84 +
 .../apache/spark/examples/ml/JavaPCAExample.java   |    70 +
 .../spark/examples/ml/JavaPipelineExample.java     |    88 +
 .../ml/JavaPolynomialExpansionExample.java         |    66 +
 .../ml/JavaPowerIterationClusteringExample.java    |    71 +
 .../spark/examples/ml/JavaPrefixSpanExample.java   |    67 +
 .../ml/JavaQuantileDiscretizerExample.java         |    73 +
 .../spark/examples/ml/JavaRFormulaExample.java     |    68 +
 .../ml/JavaRandomForestClassifierExample.java      |   101 +
 .../ml/JavaRandomForestRegressorExample.java       |    90 +
 .../spark/examples/ml/JavaRobustScalerExample.java |    57 +
 .../examples/ml/JavaSQLTransformerExample.java     |    59 +
 .../examples/ml/JavaStandardScalerExample.java     |    55 +
 .../examples/ml/JavaStopWordsRemoverExample.java   |    64 +
 .../examples/ml/JavaStringIndexerExample.java      |    68 +
 .../spark/examples/ml/JavaSummarizerExample.java   |    71 +
 .../examples/ml/JavaTargetEncoderExample.java      |    90 +
 .../apache/spark/examples/ml/JavaTfIdfExample.java |    78 +
 .../spark/examples/ml/JavaTokenizerExample.java    |    87 +
 .../ml/JavaUnivariateFeatureSelectorExample.java   |    84 +
 .../ml/JavaVarianceThresholdSelectorExample.java   |    79 +
 .../examples/ml/JavaVectorAssemblerExample.java    |    66 +
 .../examples/ml/JavaVectorIndexerExample.java      |    61 +
 .../examples/ml/JavaVectorSizeHintExample.java     |    79 +
 .../spark/examples/ml/JavaVectorSlicerExample.java |    73 +
 .../spark/examples/ml/JavaWord2VecExample.java     |    72 +
 .../org/apache/spark/examples/mllib/JavaALS.java   |    90 +
 .../mllib/JavaAssociationRulesExample.java         |    58 +
 .../JavaBinaryClassificationMetricsExample.java    |   102 +
 .../examples/mllib/JavaBisectingKMeansExample.java |    66 +
 .../examples/mllib/JavaChiSqSelectorExample.java   |    65 +
 .../examples/mllib/JavaCorrelationsExample.java    |    71 +
 .../JavaDecisionTreeClassificationExample.java     |    79 +
 .../mllib/JavaDecisionTreeRegressionExample.java   |    79 +
 .../mllib/JavaElementwiseProductExample.java       |    59 +
 .../examples/mllib/JavaGaussianMixtureExample.java |    68 +
 .../JavaGradientBoostingClassificationExample.java |    81 +
 .../JavaGradientBoostingRegressionExample.java     |    80 +
 .../mllib/JavaHypothesisTestingExample.java        |    84 +
 ...othesisTestingKolmogorovSmirnovTestExample.java |    49 +
 .../mllib/JavaIsotonicRegressionExample.java       |    74 +
 .../spark/examples/mllib/JavaKMeansExample.java    |    75 +
 .../mllib/JavaKernelDensityEstimationExample.java  |    53 +
 .../spark/examples/mllib/JavaLBFGSExample.java     |   101 +
 .../JavaLatentDirichletAllocationExample.java      |    82 +
 .../JavaLogisticRegressionWithLBFGSExample.java    |    73 +
 ...JavaMultiLabelClassificationMetricsExample.java |    80 +
 ...JavaMulticlassClassificationMetricsExample.java |    90 +
 .../examples/mllib/JavaNaiveBayesExample.java      |    55 +
 .../spark/examples/mllib/JavaPCAExample.java       |    71 +
 .../mllib/JavaPowerIterationClusteringExample.java |    60 +
 .../examples/mllib/JavaPrefixSpanExample.java      |    57 +
 .../JavaRandomForestClassificationExample.java     |    80 +
 .../mllib/JavaRandomForestRegressionExample.java   |    80 +
 .../examples/mllib/JavaRankingMetricsExample.java  |   142 +
 .../examples/mllib/JavaRecommendationExample.java  |    75 +
 .../spark/examples/mllib/JavaSVDExample.java       |    74 +
 .../examples/mllib/JavaSVMWithSGDExample.java      |    75 +
 .../spark/examples/mllib/JavaSimpleFPGrowth.java   |    63 +
 .../mllib/JavaStratifiedSamplingExample.java       |    70 +
 .../examples/mllib/JavaStreamingTestExample.java   |   104 +
 .../mllib/JavaSummaryStatisticsExample.java        |    56 +
 .../examples/sql/JavaSQLDataSourceExample.java     |   551 +
 .../apache/spark/examples/sql/JavaSparkSQLCli.java |    41 +
 .../spark/examples/sql/JavaSparkSQLExample.java    |   344 +
 .../spark/examples/sql/JavaUserDefinedScalar.java  |    90 +
 .../sql/JavaUserDefinedTypedAggregation.java       |   166 +
 .../sql/JavaUserDefinedUntypedAggregation.java     |   142 +
 .../examples/sql/hive/JavaSparkHiveExample.java    |   129 +
 .../JavaStructuredComplexSessionization.java       |   461 +
 .../streaming/JavaStructuredKafkaWordCount.java    |    92 +
 .../JavaStructuredKerberizedKafkaWordCount.java    |   129 +
 .../streaming/JavaStructuredNetworkWordCount.java  |    78 +
 .../JavaStructuredNetworkWordCountWindowed.java    |   112 +
 .../streaming/JavaStructuredSessionization.java    |    86 +
 .../examples/streaming/JavaCustomReceiver.java     |   134 +
 .../streaming/JavaDirectKafkaWordCount.java        |    98 +
 .../JavaDirectKerberizedKafkaWordCount.java        |   140 +
 .../examples/streaming/JavaNetworkWordCount.java   |    73 +
 .../spark/examples/streaming/JavaQueueStream.java  |    71 +
 .../spark/examples/streaming/JavaRecord.java       |    31 +
 .../streaming/JavaRecoverableNetworkWordCount.java |   190 +
 .../streaming/JavaSqlNetworkWordCount.java         |   108 +
 .../streaming/JavaStatefulNetworkWordCount.java    |    94 +
 examples/src/main/python/__init__.py               |    16 +
 examples/src/main/python/als.py                    |   106 +
 examples/src/main/python/avro_inputformat.py       |    91 +
 examples/src/main/python/kmeans.py                 |    85 +
 examples/src/main/python/logistic_regression.py    |    89 +
 examples/src/main/python/ml/__init__,py            |    16 +
 .../src/main/python/ml/aft_survival_regression.py  |    55 +
 examples/src/main/python/ml/als_example.py         |    70 +
 examples/src/main/python/ml/binarizer_example.py   |    44 +
 .../main/python/ml/bisecting_k_means_example.py    |    59 +
 .../ml/bucketed_random_projection_lsh_example.py   |    77 +
 examples/src/main/python/ml/bucketizer_example.py  |    44 +
 .../src/main/python/ml/chi_square_test_example.py  |    55 +
 .../src/main/python/ml/chisq_selector_example.py   |    45 +
 examples/src/main/python/ml/correlation_example.py |    57 +
 .../src/main/python/ml/count_vectorizer_example.py |    45 +
 examples/src/main/python/ml/cross_validator.py     |    97 +
 examples/src/main/python/ml/dataframe_example.py   |    77 +
 examples/src/main/python/ml/dct_example.py         |    43 +
 .../ml/decision_tree_classification_example.py     |    76 +
 .../python/ml/decision_tree_regression_example.py  |    73 +
 .../main/python/ml/elementwise_product_example.py  |    40 +
 .../ml/estimator_transformer_param_example.py      |    93 +
 .../src/main/python/ml/feature_hasher_example.py   |    44 +
 .../src/main/python/ml/fm_classifier_example.py    |    75 +
 .../src/main/python/ml/fm_regressor_example.py     |    72 +
 examples/src/main/python/ml/fpgrowth_example.py    |    55 +
 .../src/main/python/ml/gaussian_mixture_example.py |    45 +
 .../ml/generalized_linear_regression_example.py    |    63 +
 .../ml/gradient_boosted_tree_classifier_example.py |    75 +
 .../ml/gradient_boosted_tree_regressor_example.py  |    72 +
 examples/src/main/python/ml/imputer_example.py     |    49 +
 .../src/main/python/ml/index_to_string_example.py  |    52 +
 examples/src/main/python/ml/interaction_example.py |    54 +
 .../main/python/ml/isotonic_regression_example.py  |    49 +
 examples/src/main/python/ml/kmeans_example.py      |    62 +
 examples/src/main/python/ml/lda_example.py         |    57 +
 .../ml/linear_regression_with_elastic_net.py       |    52 +
 examples/src/main/python/ml/linearsvc.py           |    44 +
 .../ml/logistic_regression_summary_example.py      |    65 +
 .../ml/logistic_regression_with_elastic_net.py     |    53 +
 .../src/main/python/ml/max_abs_scaler_example.py   |    48 +
 .../src/main/python/ml/min_hash_lsh_example.py     |    77 +
 .../src/main/python/ml/min_max_scaler_example.py   |    48 +
 ...ticlass_logistic_regression_with_elastic_net.py |    84 +
 .../ml/multilayer_perceptron_classification.py     |    56 +
 examples/src/main/python/ml/n_gram_example.py      |    42 +
 examples/src/main/python/ml/naive_bayes_example.py |    57 +
 examples/src/main/python/ml/normalizer_example.py  |    49 +
 examples/src/main/python/ml/one_vs_rest_example.py |    64 +
 .../src/main/python/ml/onehot_encoder_example.py   |    47 +
 examples/src/main/python/ml/pca_example.py         |    43 +
 examples/src/main/python/ml/pipeline_example.py    |    73 +
 .../main/python/ml/polynomial_expansion_example.py |    43 +
 .../ml/power_iteration_clustering_example.py       |    49 +
 examples/src/main/python/ml/prefixspan_example.py  |    48 +
 .../main/python/ml/quantile_discretizer_example.py |    46 +
 .../python/ml/random_forest_classifier_example.py  |    80 +
 .../python/ml/random_forest_regressor_example.py   |    72 +
 examples/src/main/python/ml/rformula_example.py    |    45 +
 .../src/main/python/ml/robust_scaler_example.py    |    43 +
 examples/src/main/python/ml/sql_transformer.py     |    39 +
 .../src/main/python/ml/standard_scaler_example.py  |    42 +
 .../main/python/ml/stopwords_remover_example.py    |    39 +
 .../src/main/python/ml/string_indexer_example.py   |    39 +
 examples/src/main/python/ml/summarizer_example.py  |    57 +
 .../src/main/python/ml/target_encoder_example.py   |    65 +
 examples/src/main/python/ml/tf_idf_example.py      |    50 +
 examples/src/main/python/ml/tokenizer_example.py   |    54 +
 .../src/main/python/ml/train_validation_split.py   |    73 +
 .../ml/univariate_feature_selector_example.py      |    55 +
 .../ml/variance_threshold_selector_example.py      |    53 +
 .../src/main/python/ml/vector_assembler_example.py |    44 +
 .../src/main/python/ml/vector_indexer_example.py   |    44 +
 .../src/main/python/ml/vector_size_hint_example.py |    55 +
 .../src/main/python/ml/vector_slicer_example.py    |    43 +
 examples/src/main/python/ml/word2vec_example.py    |    47 +
 examples/src/main/python/mllib/__init__.py         |    16 +
 .../mllib/binary_classification_metrics_example.py |    55 +
 .../main/python/mllib/bisecting_k_means_example.py |    43 +
 examples/src/main/python/mllib/correlations.py     |    59 +
 .../src/main/python/mllib/correlations_example.py  |    46 +
 .../mllib/decision_tree_classification_example.py  |    54 +
 .../mllib/decision_tree_regression_example.py      |    54 +
 .../python/mllib/elementwise_product_example.py    |    49 +
 examples/src/main/python/mllib/fpgrowth_example.py |    33 +
 .../main/python/mllib/gaussian_mixture_example.py  |    49 +
 .../main/python/mllib/gaussian_mixture_model.py    |    69 +
 .../gradient_boosting_classification_example.py    |    54 +
 .../mllib/gradient_boosting_regression_example.py  |    53 +
 .../python/mllib/hypothesis_testing_example.py     |    63 +
 ...esis_testing_kolmogorov_smirnov_test_example.py |    38 +
 .../python/mllib/isotonic_regression_example.py    |    59 +
 examples/src/main/python/mllib/k_means_example.py  |    52 +
 .../mllib/kernel_density_estimation_example.py     |    42 +
 examples/src/main/python/mllib/kmeans.py           |    45 +
 .../mllib/latent_dirichlet_allocation_example.py   |    52 +
 .../mllib/linear_regression_with_sgd_example.py    |    52 +
 .../src/main/python/mllib/logistic_regression.py   |    50 +
 .../logistic_regression_with_lbfgs_example.py      |    52 +
 .../python/mllib/multi_class_metrics_example.py    |    69 +
 .../python/mllib/multi_label_metrics_example.py    |    61 +
 .../src/main/python/mllib/naive_bayes_example.py   |    63 +
 .../src/main/python/mllib/normalizer_example.py    |    50 +
 .../src/main/python/mllib/pca_rowmatrix_example.py |    46 +
 .../mllib/power_iteration_clustering_example.py    |    42 +
 .../mllib/random_forest_classification_example.py  |    55 +
 .../mllib/random_forest_regression_example.py      |    55 +
 .../src/main/python/mllib/random_rdd_generation.py |    54 +
 .../main/python/mllib/ranking_metrics_example.py   |    55 +
 .../main/python/mllib/recommendation_example.py    |    50 +
 .../python/mllib/regression_metrics_example.py     |    59 +
 examples/src/main/python/mllib/sampled_rdds.py     |    85 +
 .../main/python/mllib/standard_scaler_example.py   |    51 +
 .../python/mllib/stratified_sampling_example.py    |    36 +
 .../main/python/mllib/streaming_k_means_example.py |    64 +
 .../mllib/streaming_linear_regression_example.py   |    60 +
 .../python/mllib/summary_statistics_example.py     |    40 +
 examples/src/main/python/mllib/svd_example.py      |    48 +
 .../src/main/python/mllib/svm_with_sgd_example.py  |    47 +
 examples/src/main/python/mllib/tf_idf_example.py   |    55 +
 examples/src/main/python/mllib/word2vec.py         |    49 +
 examples/src/main/python/mllib/word2vec_example.py |    38 +
 examples/src/main/python/pagerank.py               |    89 +
 examples/src/main/python/parquet_inputformat.py    |    69 +
 examples/src/main/python/pi.py                     |    45 +
 examples/src/main/python/sort.py                   |    45 +
 examples/src/main/python/sql/__init__.py           |    16 +
 examples/src/main/python/sql/arrow.py              |   345 +
 examples/src/main/python/sql/basic.py              |   214 +
 examples/src/main/python/sql/datasource.py         |   491 +
 examples/src/main/python/sql/hive.py               |    93 +
 examples/src/main/python/sql/streaming/__init__,py |    16 +
 .../sql/streaming/structured_kafka_wordcount.py    |    88 +
 .../sql/streaming/structured_network_wordcount.py  |    75 +
 .../structured_network_wordcount_session_window.py |   142 +
 .../structured_network_wordcount_windowed.py       |   100 +
 .../sql/streaming/structured_sessionization.py     |    87 +
 examples/src/main/python/sql/udtf.py               |   320 +
 examples/src/main/python/status_api_demo.py        |    71 +
 examples/src/main/python/streaming/__init__.py     |    16 +
 .../src/main/python/streaming/hdfs_wordcount.py    |    48 +
 .../src/main/python/streaming/network_wordcount.py |    47 +
 .../python/streaming/network_wordjoinsentiments.py |    86 +
 examples/src/main/python/streaming/queue_stream.py |    49 +
 .../streaming/recoverable_network_wordcount.py     |   110 +
 .../main/python/streaming/sql_network_wordcount.py |    84 +
 .../python/streaming/stateful_network_wordcount.py |    60 +
 examples/src/main/python/transitive_closure.py     |    73 +
 examples/src/main/python/wordcount.py              |    42 +
 examples/src/main/r/RSparkSQLExample.R             |   275 +
 examples/src/main/r/data-manipulation.R            |   103 +
 examples/src/main/r/dataframe.R                    |    59 +
 examples/src/main/r/ml/als.R                       |    47 +
 examples/src/main/r/ml/bisectingKmeans.R           |    45 +
 examples/src/main/r/ml/decisionTree.R              |    65 +
 examples/src/main/r/ml/fmClassifier.R              |    45 +
 examples/src/main/r/ml/fmRegressor.R               |    45 +
 examples/src/main/r/ml/fpm.R                       |    50 +
 examples/src/main/r/ml/gaussianMixture.R           |    44 +
 examples/src/main/r/ml/gbt.R                       |    65 +
 examples/src/main/r/ml/glm.R                       |    71 +
 examples/src/main/r/ml/isoreg.R                    |    44 +
 examples/src/main/r/ml/kmeans.R                    |    48 +
 examples/src/main/r/ml/kstest.R                    |    41 +
 examples/src/main/r/ml/lda.R                       |    48 +
 examples/src/main/r/ml/lm_with_elastic_net.R       |    45 +
 examples/src/main/r/ml/logit.R                     |    65 +
 examples/src/main/r/ml/ml.R                        |    65 +
 examples/src/main/r/ml/mlp.R                       |    50 +
 examples/src/main/r/ml/naiveBayes.R                |    43 +
 examples/src/main/r/ml/powerIterationClustering.R  |    39 +
 examples/src/main/r/ml/prefixSpan.R                |    43 +
 examples/src/main/r/ml/randomForest.R              |    65 +
 examples/src/main/r/ml/survreg.R                   |    46 +
 examples/src/main/r/ml/svmLinear.R                 |    42 +
 .../r/streaming/structured_network_wordcount.R     |    57 +
 ...apache.spark.sql.SparkSessionExtensionsProvider |    18 +
 ...rg.apache.spark.sql.jdbc.JdbcConnectionProvider |    18 +
 .../src/main/resources/dir1/dir2/file2.parquet     |   Bin 0 -> 520 bytes
 examples/src/main/resources/dir1/file1.parquet     |   Bin 0 -> 520 bytes
 examples/src/main/resources/dir1/file3.json        |     1 +
 examples/src/main/resources/employees.json         |     4 +
 examples/src/main/resources/full_user.avsc         |     1 +
 examples/src/main/resources/kv1.txt                |   500 +
 examples/src/main/resources/people.csv             |     3 +
 examples/src/main/resources/people.json            |     3 +
 examples/src/main/resources/people.txt             |     3 +
 examples/src/main/resources/people.xml             |    15 +
 examples/src/main/resources/user.avsc              |     8 +
 examples/src/main/resources/users.avro             |   Bin 0 -> 334 bytes
 examples/src/main/resources/users.orc              |   Bin 0 -> 547 bytes
 examples/src/main/resources/users.parquet          |   Bin 0 -> 615 bytes
 .../spark/examples/AccumulatorMetricsTest.scala    |    77 +
 .../org/apache/spark/examples/BroadcastTest.scala  |    58 +
 .../apache/spark/examples/DFSReadWriteTest.scala   |   144 +
 .../spark/examples/DriverSubmissionTest.scala      |    54 +
 .../spark/examples/ExceptionHandlingTest.scala     |    37 +
 .../org/apache/spark/examples/GroupByTest.scala    |    58 +
 .../scala/org/apache/spark/examples/HdfsTest.scala |    51 +
 .../scala/org/apache/spark/examples/LocalALS.scala |   143 +
 .../org/apache/spark/examples/LocalFileLR.scala    |    77 +
 .../org/apache/spark/examples/LocalKMeans.scala    |   120 +
 .../scala/org/apache/spark/examples/LocalLR.scala  |    79 +
 .../scala/org/apache/spark/examples/LocalPi.scala  |    34 +
 .../scala/org/apache/spark/examples/LogQuery.scala |    86 +
 .../apache/spark/examples/MiniReadWriteTest.scala  |   139 +
 .../apache/spark/examples/MultiBroadcastTest.scala |    60 +
 .../spark/examples/SimpleSkewedGroupByTest.scala   |    67 +
 .../apache/spark/examples/SkewedGroupByTest.scala  |    62 +
 .../scala/org/apache/spark/examples/SparkALS.scala |   150 +
 .../org/apache/spark/examples/SparkHdfsLR.scala    |    96 +
 .../org/apache/spark/examples/SparkKMeans.scala    |   112 +
 .../scala/org/apache/spark/examples/SparkLR.scala  |    92 +
 .../org/apache/spark/examples/SparkPageRank.scala  |    85 +
 .../scala/org/apache/spark/examples/SparkPi.scala  |    43 +
 .../spark/examples/SparkRemoteFileTest.scala       |    48 +
 .../scala/org/apache/spark/examples/SparkTC.scala  |    75 +
 .../spark/examples/extensions/AgeExample.scala     |    32 +
 .../extensions/SessionExtensionsWithLoader.scala   |    31 +
 .../SessionExtensionsWithoutLoader.scala           |    31 +
 .../extensions/SparkSessionExtensionsTest.scala    |    38 +
 .../examples/graphx/AggregateMessagesExample.scala |    72 +
 .../apache/spark/examples/graphx/Analytics.scala   |   159 +
 .../examples/graphx/ComprehensiveExample.scala     |    80 +
 .../graphx/ConnectedComponentsExample.scala        |    68 +
 .../examples/graphx/LiveJournalPageRank.scala      |    46 +
 .../spark/examples/graphx/PageRankExample.scala    |    61 +
 .../apache/spark/examples/graphx/SSSPExample.scala |    69 +
 .../spark/examples/graphx/SynthBenchmark.scala     |   132 +
 .../examples/graphx/TriangleCountingExample.scala  |    70 +
 .../examples/ml/AFTSurvivalRegressionExample.scala |    67 +
 .../org/apache/spark/examples/ml/ALSExample.scala  |   100 +
 .../spark/examples/ml/BinarizerExample.scala       |    53 +
 .../spark/examples/ml/BisectingKMeansExample.scala |    71 +
 .../ml/BucketedRandomProjectionLSHExample.scala    |    91 +
 .../spark/examples/ml/BucketizerExample.scala      |    93 +
 .../spark/examples/ml/ChiSqSelectorExample.scala   |    59 +
 .../spark/examples/ml/ChiSquareTestExample.scala   |    63 +
 .../spark/examples/ml/CorrelationExample.scala     |    63 +
 .../spark/examples/ml/CountVectorizerExample.scala |    60 +
 .../org/apache/spark/examples/ml/DCTExample.scala  |    55 +
 .../spark/examples/ml/DataFrameExample.scala       |   102 +
 .../ml/DecisionTreeClassificationExample.scala     |    95 +
 .../spark/examples/ml/DecisionTreeExample.scala    |   347 +
 .../ml/DecisionTreeRegressionExample.scala         |    85 +
 .../spark/examples/ml/DeveloperApiExample.scala    |   190 +
 .../examples/ml/ElementwiseProductExample.scala    |    53 +
 .../ml/EstimatorTransformerParamExample.scala      |   100 +
 .../spark/examples/ml/FMClassifierExample.scala    |    96 +
 .../spark/examples/ml/FMRegressorExample.scala     |    84 +
 .../apache/spark/examples/ml/FPGrowthExample.scala |    64 +
 .../spark/examples/ml/FeatureHasherExample.scala   |    50 +
 .../org/apache/spark/examples/ml/GBTExample.scala  |   242 +
 .../spark/examples/ml/GaussianMixtureExample.scala |    60 +
 .../ml/GeneralizedLinearRegressionExample.scala    |    78 +
 .../ml/GradientBoostedTreeClassifierExample.scala  |    98 +
 .../ml/GradientBoostedTreeRegressorExample.scala   |    85 +
 .../apache/spark/examples/ml/ImputerExample.scala  |    56 +
 .../spark/examples/ml/IndexToStringExample.scala   |    72 +
 .../spark/examples/ml/InteractionExample.scala     |    68 +
 .../examples/ml/IsotonicRegressionExample.scala    |    60 +
 .../apache/spark/examples/ml/KMeansExample.scala   |    68 +
 .../org/apache/spark/examples/ml/LDAExample.scala  |    68 +
 .../examples/ml/LinearRegressionExample.scala      |   140 +
 .../ml/LinearRegressionWithElasticNetExample.scala |    62 +
 .../spark/examples/ml/LinearSVCExample.scala       |    52 +
 .../examples/ml/LogisticRegressionExample.scala    |   161 +
 .../ml/LogisticRegressionSummaryExample.scala      |    73 +
 .../LogisticRegressionWithElasticNetExample.scala  |    66 +
 .../spark/examples/ml/MaxAbsScalerExample.scala    |    54 +
 .../spark/examples/ml/MinHashLSHExample.scala      |    90 +
 .../spark/examples/ml/MinMaxScalerExample.scala    |    57 +
 .../ModelSelectionViaCrossValidationExample.scala  |   121 +
 ...elSelectionViaTrainValidationSplitExample.scala |    83 +
 ...ssLogisticRegressionWithElasticNetExample.scala |   100 +
 .../ml/MultilayerPerceptronClassifierExample.scala |    75 +
 .../apache/spark/examples/ml/NGramExample.scala    |    49 +
 .../spark/examples/ml/NaiveBayesExample.scala      |    61 +
 .../spark/examples/ml/NormalizerExample.scala      |    60 +
 .../spark/examples/ml/OneHotEncoderExample.scala   |    56 +
 .../spark/examples/ml/OneVsRestExample.scala       |    79 +
 .../org/apache/spark/examples/ml/PCAExample.scala  |    58 +
 .../apache/spark/examples/ml/PipelineExample.scala |    93 +
 .../examples/ml/PolynomialExpansionExample.scala   |    57 +
 .../ml/PowerIterationClusteringExample.scala       |    56 +
 .../spark/examples/ml/PrefixSpanExample.scala      |    59 +
 .../examples/ml/QuantileDiscretizerExample.scala   |    55 +
 .../apache/spark/examples/ml/RFormulaExample.scala |    52 +
 .../ml/RandomForestClassifierExample.scala         |    97 +
 .../spark/examples/ml/RandomForestExample.scala    |   250 +
 .../examples/ml/RandomForestRegressorExample.scala |    84 +
 .../spark/examples/ml/RobustScalerExample.scala    |    55 +
 .../spark/examples/ml/SQLTransformerExample.scala  |    46 +
 .../spark/examples/ml/StandardScalerExample.scala  |    53 +
 .../examples/ml/StopWordsRemoverExample.scala      |    49 +
 .../spark/examples/ml/StringIndexerExample.scala   |    49 +
 .../spark/examples/ml/SummarizerExample.scala      |    61 +
 .../spark/examples/ml/TargetEncoderExample.scala   |    71 +
 .../apache/spark/examples/ml/TfIdfExample.scala    |    60 +
 .../spark/examples/ml/TokenizerExample.scala       |    61 +
 .../examples/ml/UnaryTransformerExample.scala      |   122 +
 .../ml/UnivariateFeatureSelectorExample.scala      |    73 +
 .../ml/VarianceThresholdSelectorExample.scala      |    69 +
 .../spark/examples/ml/VectorAssemblerExample.scala |    51 +
 .../spark/examples/ml/VectorIndexerExample.scala   |    55 +
 .../spark/examples/ml/VectorSizeHintExample.scala  |    63 +
 .../spark/examples/ml/VectorSlicerExample.scala    |    62 +
 .../apache/spark/examples/ml/Word2VecExample.scala |    59 +
 .../spark/examples/mllib/AbstractParams.scala      |    53 +
 .../examples/mllib/AssociationRulesExample.scala   |    55 +
 .../examples/mllib/BinaryClassification.scala      |   157 +
 .../mllib/BinaryClassificationMetricsExample.scala |   104 +
 .../examples/mllib/BisectingKMeansExample.scala    |    60 +
 .../examples/mllib/ChiSqSelectorExample.scala      |    61 +
 .../apache/spark/examples/mllib/Correlations.scala |    93 +
 .../spark/examples/mllib/CorrelationsExample.scala |    66 +
 .../spark/examples/mllib/CosineSimilarity.scala    |   108 +
 .../mllib/DecisionTreeClassificationExample.scala  |    69 +
 .../mllib/DecisionTreeRegressionExample.scala      |    68 +
 .../spark/examples/mllib/DecisionTreeRunner.scala  |   363 +
 .../apache/spark/examples/mllib/DenseKMeans.scala  |   111 +
 .../examples/mllib/ElementwiseProductExample.scala |    56 +
 .../spark/examples/mllib/FPGrowthExample.scala     |    83 +
 .../examples/mllib/GaussianMixtureExample.scala    |    57 +
 .../mllib/GradientBoostedTreesRunner.scala         |   156 +
 .../GradientBoostingClassificationExample.scala    |    71 +
 .../mllib/GradientBoostingRegressionExample.scala  |    68 +
 .../examples/mllib/HypothesisTestingExample.scala  |    80 +
 ...thesisTestingKolmogorovSmirnovTestExample.scala |    54 +
 .../examples/mllib/IsotonicRegressionExample.scala |    69 +
 .../spark/examples/mllib/KMeansExample.scala       |    56 +
 .../mllib/KernelDensityEstimationExample.scala     |    54 +
 .../apache/spark/examples/mllib/LBFGSExample.scala |    91 +
 .../apache/spark/examples/mllib/LDAExample.scala   |   240 +
 .../mllib/LatentDirichletAllocationExample.scala   |    64 +
 .../mllib/LogisticRegressionWithLBFGSExample.scala |    69 +
 .../apache/spark/examples/mllib/MovieLensALS.scala |   192 +
 .../examples/mllib/MultiLabelMetricsExample.scala  |    71 +
 .../examples/mllib/MulticlassMetricsExample.scala  |    97 +
 .../examples/mllib/MultivariateSummarizer.scala    |    99 +
 .../spark/examples/mllib/NaiveBayesExample.scala   |    53 +
 .../spark/examples/mllib/NormalizerExample.scala   |    57 +
 .../examples/mllib/PCAOnRowMatrixExample.scala     |    62 +
 .../examples/mllib/PCAOnSourceVectorExample.scala  |    59 +
 .../examples/mllib/PMMLModelExportExample.scala    |    59 +
 .../mllib/PowerIterationClusteringExample.scala    |   152 +
 .../spark/examples/mllib/PrefixSpanExample.scala   |    53 +
 .../mllib/RandomForestClassificationExample.scala  |    69 +
 .../mllib/RandomForestRegressionExample.scala      |    70 +
 .../spark/examples/mllib/RandomRDDGeneration.scala |    61 +
 .../examples/mllib/RankingMetricsExample.scala     |   117 +
 .../examples/mllib/RecommendationExample.scala     |    69 +
 .../apache/spark/examples/mllib/SVDExample.scala   |    68 +
 .../spark/examples/mllib/SVMWithSGDExample.scala   |    70 +
 .../apache/spark/examples/mllib/SampledRDDs.scala  |   127 +
 .../spark/examples/mllib/SimpleFPGrowth.scala      |    58 +
 .../spark/examples/mllib/SparseNaiveBayes.scala    |   104 +
 .../examples/mllib/StandardScalerExample.scala     |    60 +
 .../examples/mllib/StratifiedSamplingExample.scala |    53 +
 .../examples/mllib/StreamingKMeansExample.scala    |    83 +
 .../mllib/StreamingLinearRegressionExample.scala   |    77 +
 .../mllib/StreamingLogisticRegression.scala        |    75 +
 .../examples/mllib/StreamingTestExample.scala      |    92 +
 .../examples/mllib/SummaryStatisticsExample.scala  |    53 +
 .../apache/spark/examples/mllib/TFIDFExample.scala |    66 +
 .../spark/examples/mllib/TallSkinnyPCA.scala       |    62 +
 .../spark/examples/mllib/TallSkinnySVD.scala       |    62 +
 .../spark/examples/mllib/Word2VecExample.scala     |    55 +
 .../examples/pythonconverters/AvroConverters.scala |   148 +
 .../apache/spark/examples/sql/RDDRelation.scala    |    82 +
 .../spark/examples/sql/SQLDataSourceExample.scala  |   471 +
 .../spark/examples/sql/SimpleTypedAggregator.scala |   160 +
 .../spark/examples/sql/SparkDataFramePi.scala      |    42 +
 .../spark/examples/sql/SparkSQLExample.scala       |   274 +
 .../spark/examples/sql/UserDefinedScalar.scala     |    80 +
 .../examples/sql/UserDefinedTypedAggregation.scala |    89 +
 .../sql/UserDefinedUntypedAggregation.scala        |    89 +
 .../spark/examples/sql/hive/SparkHiveExample.scala |   154 +
 .../sql/jdbc/ExampleJdbcConnectionProvider.scala   |    38 +
 .../StructuredComplexSessionization.scala          |   301 +
 .../sql/streaming/StructuredKafkaWordCount.scala   |    93 +
 .../StructuredKerberizedKafkaWordCount.scala       |   131 +
 .../sql/streaming/StructuredNetworkWordCount.scala |    75 +
 .../StructuredNetworkWordCountWindowed.scala       |   103 +
 .../sql/streaming/StructuredSessionization.scala   |    89 +
 .../spark/examples/streaming/CustomReceiver.scala  |   106 +
 .../examples/streaming/DirectKafkaWordCount.scala  |    83 +
 .../streaming/DirectKerberizedKafkaWordCount.scala |   123 +
 .../spark/examples/streaming/HdfsWordCount.scala   |    57 +
 .../examples/streaming/NetworkWordCount.scala      |    61 +
 .../spark/examples/streaming/QueueStream.scala     |    55 +
 .../spark/examples/streaming/RawNetworkGrep.scala  |    64 +
 .../streaming/RecoverableNetworkWordCount.scala    |   165 +
 .../examples/streaming/SqlNetworkWordCount.scala   |   104 +
 .../streaming/StatefulNetworkWordCount.scala       |    76 +
 .../examples/streaming/StreamingExamples.scala     |    38 +
 .../streaming/clickstream/PageViewGenerator.scala  |   113 +
 .../streaming/clickstream/PageViewStream.scala     |   115 +
 examples/src/main/scripts/getGpusResources.sh      |    33 +
 examples/src/main/scripts/submit-pi.sh             |    50 +
 examples/src/main/scripts/submit-sql.sh            |    39 +
 graphx/pom.xml                                     |    97 +
 .../org/apache/spark/graphx/TripletFields.java     |    72 +
 .../apache/spark/graphx/impl/EdgeActiveness.java   |    34 +
 .../main/scala/org/apache/spark/graphx/Edge.scala  |    98 +
 .../org/apache/spark/graphx/EdgeContext.scala      |    67 +
 .../org/apache/spark/graphx/EdgeDirection.scala    |    61 +
 .../scala/org/apache/spark/graphx/EdgeRDD.scala    |   125 +
 .../org/apache/spark/graphx/EdgeTriplet.scala      |    68 +
 .../main/scala/org/apache/spark/graphx/Graph.scala |   543 +
 .../org/apache/spark/graphx/GraphLoader.scala      |   107 +
 .../scala/org/apache/spark/graphx/GraphOps.scala   |   477 +
 .../org/apache/spark/graphx/GraphXUtils.scala      |    74 +
 .../apache/spark/graphx/PartitionStrategy.scala    |   143 +
 .../scala/org/apache/spark/graphx/Pregel.scala     |   179 +
 .../scala/org/apache/spark/graphx/VertexRDD.scala  |   365 +
 .../apache/spark/graphx/impl/EdgePartition.scala   |   542 +
 .../spark/graphx/impl/EdgePartitionBuilder.scala   |   171 +
 .../org/apache/spark/graphx/impl/EdgeRDDImpl.scala |   139 +
 .../org/apache/spark/graphx/impl/GraphImpl.scala   |   357 +
 .../spark/graphx/impl/ReplicatedVertexView.scala   |   120 +
 .../spark/graphx/impl/RoutingTablePartition.scala  |   139 +
 .../graphx/impl/ShippableVertexPartition.scala     |   162 +
 .../apache/spark/graphx/impl/VertexPartition.scala |    75 +
 .../spark/graphx/impl/VertexPartitionBase.scala    |    90 +
 .../spark/graphx/impl/VertexPartitionBaseOps.scala |   258 +
 .../apache/spark/graphx/impl/VertexRDDImpl.scala   |   254 +
 .../org/apache/spark/graphx/impl/package.scala     |    24 +
 .../spark/graphx/lib/ConnectedComponents.scala     |    75 +
 .../apache/spark/graphx/lib/LabelPropagation.scala |    73 +
 .../org/apache/spark/graphx/lib/PageRank.scala     |   513 +
 .../org/apache/spark/graphx/lib/SVDPlusPlus.scala  |   214 +
 .../apache/spark/graphx/lib/ShortestPaths.scala    |    77 +
 .../graphx/lib/StronglyConnectedComponents.scala   |   127 +
 .../apache/spark/graphx/lib/TriangleCount.scala    |   116 +
 .../org/apache/spark/graphx/lib/package-info.java  |    21 +
 .../org/apache/spark/graphx/lib/package.scala      |    23 +
 .../org/apache/spark/graphx/package-info.java      |    22 +
 .../scala/org/apache/spark/graphx/package.scala    |    38 +
 .../apache/spark/graphx/util/GraphGenerators.scala |   264 +
 .../graphx/util/PeriodicGraphCheckpointer.scala    |   106 +
 .../collection/GraphXPrimitiveKeyOpenHashMap.scala |   148 +
 .../org/apache/spark/graphx/util/package-info.java |    21 +
 .../org/apache/spark/graphx/util/package.scala     |    23 +
 graphx/src/test/resources/als-test.data            |    16 +
 graphx/src/test/resources/log4j2.properties        |    31 +
 .../org/apache/spark/graphx/EdgeRDDSuite.scala     |    73 +
 .../scala/org/apache/spark/graphx/EdgeSuite.scala  |    39 +
 .../org/apache/spark/graphx/GraphLoaderSuite.scala |    48 +
 .../org/apache/spark/graphx/GraphOpsSuite.scala    |   230 +
 .../scala/org/apache/spark/graphx/GraphSuite.scala |   417 +
 .../apache/spark/graphx/LocalSparkContext.scala    |    39 +
 .../org/apache/spark/graphx/PregelSuite.scala      |    55 +
 .../org/apache/spark/graphx/VertexRDDSuite.scala   |   237 +
 .../spark/graphx/impl/EdgePartitionSuite.scala     |   124 +
 .../spark/graphx/impl/VertexPartitionSuite.scala   |   134 +
 .../graphx/lib/ConnectedComponentsSuite.scala      |   130 +
 .../spark/graphx/lib/LabelPropagationSuite.scala   |    44 +
 .../apache/spark/graphx/lib/PageRankSuite.scala    |   448 +
 .../apache/spark/graphx/lib/SVDPlusPlusSuite.scala |    52 +
 .../spark/graphx/lib/ShortestPathsSuite.scala      |    43 +
 .../lib/StronglyConnectedComponentsSuite.scala     |    71 +
 .../spark/graphx/lib/TriangleCountSuite.scala      |    87 +
 .../spark/graphx/util/GraphGeneratorsSuite.scala   |   122 +
 .../util/PeriodicGraphCheckpointerSuite.scala      |   195 +
 hadoop-cloud/README.md                             |    20 +
 hadoop-cloud/pom.xml                               |   295 +
 ...AbortableStreamBasedCheckpointFileManager.scala |    98 +
 .../io/cloud/BindingParquetOutputCommitter.scala   |   126 +
 .../io/cloud/PathOutputCommitProtocol.scala        |   223 +
 .../internal/io/cloud/IntegrationTestSuite.java    |    29 +
 .../io/cloud/abortable/AbortableFileSystem.java    |   113 +
 .../abortable/AbstractAbortableFileSystem.java     |    44 +
 hadoop-cloud/src/test/resources/log4j2.properties  |    40 +
 .../scala/org/apache/spark/SparkContextSuite.scala |    79 +
 ...ableStreamBasedCheckpointFileManagerSuite.scala |    83 +
 .../internal/io/cloud/CommitterBindingSuite.scala  |   268 +
 .../io/cloud/StubPathOutputCommitter.scala         |   164 +
 launcher/pom.xml                                   |   121 +
 .../apache/spark/launcher/AbstractAppHandle.java   |   166 +
 .../spark/launcher/AbstractCommandBuilder.java     |   428 +
 .../apache/spark/launcher/AbstractLauncher.java    |   323 +
 .../apache/spark/launcher/ChildProcAppHandle.java  |   137 +
 .../apache/spark/launcher/CommandBuilderUtils.java |   348 +
 .../spark/launcher/FilteredObjectInputStream.java  |    53 +
 .../apache/spark/launcher/InProcessAppHandle.java  |    89 +
 .../apache/spark/launcher/InProcessLauncher.java   |   118 +
 .../apache/spark/launcher/JavaModuleOptions.java   |    65 +
 .../apache/spark/launcher/LauncherConnection.java  |   109 +
 .../apache/spark/launcher/LauncherProtocol.java    |    93 +
 .../org/apache/spark/launcher/LauncherServer.java  |   401 +
 .../main/java/org/apache/spark/launcher/Main.java  |   191 +
 .../apache/spark/launcher/NamedThreadFactory.java  |    40 +
 .../apache/spark/launcher/OutputRedirector.java    |   113 +
 .../org/apache/spark/launcher/SparkAppHandle.java  |   136 +
 .../spark/launcher/SparkClassCommandBuilder.java   |   111 +
 .../org/apache/spark/launcher/SparkLauncher.java   |   506 +
 .../spark/launcher/SparkSubmitCommandBuilder.java  |   641 +
 .../spark/launcher/SparkSubmitOptionParser.java    |   240 +
 .../org/apache/spark/launcher/package-info.java    |    93 +
 .../java/org/apache/spark/launcher/BaseSuite.java  |    92 +
 .../spark/launcher/ChildProcAppHandleSuite.java    |   283 +
 .../spark/launcher/CommandBuilderUtilsSuite.java   |   121 +
 .../spark/launcher/InProcessLauncherSuite.java     |   159 +
 .../apache/spark/launcher/LauncherServerSuite.java |   281 +
 .../launcher/SparkClassCommandBuilderSuite.java    |    40 +
 .../launcher/SparkSubmitCommandBuilderSuite.java   |   547 +
 .../launcher/SparkSubmitOptionParserSuite.java     |   111 +
 launcher/src/test/resources/log4j2.properties      |    43 +
 launcher/src/test/resources/spark-defaults.conf    |    22 +
 licenses-binary/LICENSE-AnchorJS.txt               |    21 +
 licenses-binary/LICENSE-CC0.txt                    |   121 +
 licenses-binary/LICENSE-JLargeArrays.txt           |    23 +
 licenses-binary/LICENSE-JTransforms.txt            |    23 +
 licenses-binary/LICENSE-antlr.txt                  |     8 +
 licenses-binary/LICENSE-arpack.txt                 |     8 +
 licenses-binary/LICENSE-automaton.txt              |    24 +
 licenses-binary/LICENSE-blas.txt                   |    25 +
 licenses-binary/LICENSE-bootstrap.txt              |    22 +
 licenses-binary/LICENSE-cloudpickle.txt            |    28 +
 licenses-binary/LICENSE-d3.min.js.txt              |    13 +
 licenses-binary/LICENSE-dagre-d3.txt               |    19 +
 licenses-binary/LICENSE-datatables.txt             |    24 +
 licenses-binary/LICENSE-dnsjava.txt                |    24 +
 licenses-binary/LICENSE-f2j.txt                    |     8 +
 licenses-binary/LICENSE-graphlib-dot.txt           |    19 +
 licenses-binary/LICENSE-icu4j.txt                  |   519 +
 licenses-binary/LICENSE-istack-commons-runtime.txt |    11 +
 licenses-binary/LICENSE-jakarta-annotation-api     |   277 +
 licenses-binary/LICENSE-jakarta-servlet-api.txt    |   277 +
 licenses-binary/LICENSE-jakarta-ws-rs-api          |   277 +
 licenses-binary/LICENSE-jakarta.activation-api.txt |    11 +
 licenses-binary/LICENSE-jakarta.xml.bind-api.txt   |    11 +
 licenses-binary/LICENSE-janino.txt                 |    31 +
 licenses-binary/LICENSE-javassist.html             |   373 +
 .../LICENSE-javax-transaction-transaction-api.txt  |   119 +
 licenses-binary/LICENSE-javolution.txt             |    27 +
 licenses-binary/LICENSE-jaxb-runtime.txt           |    11 +
 licenses-binary/LICENSE-jline.txt                  |    32 +
 licenses-binary/LICENSE-jline3.txt                 |    34 +
 licenses-binary/LICENSE-join.txt                   |    30 +
 licenses-binary/LICENSE-jquery.txt                 |    20 +
 licenses-binary/LICENSE-json-formatter.txt         |     6 +
 licenses-binary/LICENSE-jsp-api.txt                |   759 +
 licenses-binary/LICENSE-kryo.txt                   |    10 +
 licenses-binary/LICENSE-leveldbjni.txt             |    27 +
 licenses-binary/LICENSE-loose-version.txt          |   279 +
 licenses-binary/LICENSE-machinist.txt              |    19 +
 licenses-binary/LICENSE-matchMedia-polyfill.txt    |     1 +
 licenses-binary/LICENSE-minlog.txt                 |    10 +
 licenses-binary/LICENSE-modernizr.txt              |    21 +
 licenses-binary/LICENSE-mustache.txt               |    11 +
 licenses-binary/LICENSE-paranamer.txt              |    28 +
 licenses-binary/LICENSE-pmml-model.txt             |    10 +
 licenses-binary/LICENSE-protobuf.txt               |    32 +
 licenses-binary/LICENSE-py4j.txt                   |    27 +
 licenses-binary/LICENSE-pyrolite.txt               |    28 +
 licenses-binary/LICENSE-re2j.txt                   |    32 +
 licenses-binary/LICENSE-reflectasm.txt             |    10 +
 licenses-binary/LICENSE-respond.txt                |    22 +
 licenses-binary/LICENSE-sbt-launch-lib.txt         |    26 +
 licenses-binary/LICENSE-scopt.txt                  |     9 +
 licenses-binary/LICENSE-slf4j.txt                  |    21 +
 licenses-binary/LICENSE-sorttable.js.txt           |    16 +
 licenses-binary/LICENSE-spire.txt                  |    19 +
 licenses-binary/LICENSE-vis-timeline.txt           |    23 +
 licenses-binary/LICENSE-xmlenc.txt                 |    27 +
 licenses-binary/LICENSE-xz.txt                     |    11 +
 licenses-binary/LICENSE-zstd-jni.txt               |    26 +
 licenses-binary/LICENSE-zstd.txt                   |    30 +
 licenses/LICENSE-AnchorJS.txt                      |    21 +
 licenses/LICENSE-CC0.txt                           |   121 +
 licenses/LICENSE-bootstrap.txt                     |    22 +
 licenses/LICENSE-cloudpickle.txt                   |    28 +
 licenses/LICENSE-d3.min.js.txt                     |    13 +
 licenses/LICENSE-dagre-d3.txt                      |    19 +
 licenses/LICENSE-datatables.txt                    |    24 +
 licenses/LICENSE-graphlib-dot.txt                  |    19 +
 licenses/LICENSE-jdom.txt                          |    54 +
 licenses/LICENSE-join.txt                          |    30 +
 licenses/LICENSE-jquery.txt                        |    20 +
 licenses/LICENSE-json-formatter.txt                |     6 +
 licenses/LICENSE-loose-version.txt                 |   279 +
 licenses/LICENSE-matchMedia-polyfill.txt           |     1 +
 licenses/LICENSE-modernizr.txt                     |    21 +
 licenses/LICENSE-mustache.txt                      |    11 +
 licenses/LICENSE-py4j.txt                          |    27 +
 licenses/LICENSE-respond.txt                       |    22 +
 licenses/LICENSE-sbt-launch-lib.txt                |    26 +
 licenses/LICENSE-sorttable.js.txt                  |    16 +
 licenses/LICENSE-vis-timeline.txt                  |    23 +
 licenses/LICENSE-xz.txt                            |    11 +
 .../benchmarks/BLASBenchmark-jdk21-results.txt     |   312 +
 mllib-local/benchmarks/BLASBenchmark-results.txt   |   312 +
 mllib-local/pom.xml                                |    98 +
 .../scala/org/apache/spark/ml/impl/Utils.scala     |   140 +
 .../scala/org/apache/spark/ml/linalg/BLAS.scala    |   869 +
 .../org/apache/spark/ml/linalg/Matrices.scala      |  1353 +
 .../scala/org/apache/spark/ml/linalg/Vectors.scala |   881 +
 .../stat/distribution/MultivariateGaussian.scala   |   133 +
 .../scala/scala/collection/compat/package.scala    |    26 +
 .../org/apache/spark/ml/SparkMLFunSuite.scala      |    31 +
 .../org/apache/spark/ml/impl/UtilsSuite.scala      |    30 +
 .../org/apache/spark/ml/linalg/BLASBenchmark.scala |   446 +
 .../org/apache/spark/ml/linalg/BLASSuite.scala     |   476 +
 .../ml/linalg/BreezeMatrixConversionSuite.scala    |    71 +
 .../ml/linalg/BreezeVectorConversionSuite.scala    |    67 +
 .../org/apache/spark/ml/linalg/MatricesSuite.scala |   906 +
 .../org/apache/spark/ml/linalg/VectorsSuite.scala  |   484 +
 .../distribution/MultivariateGaussianSuite.scala   |    91 +
 .../org/apache/spark/ml/util/TestingUtils.scala    |   246 +
 .../apache/spark/ml/util/TestingUtilsSuite.scala   |   460 +
 .../UDTSerializationBenchmark-jdk21-results.txt    |    12 +
 .../UDTSerializationBenchmark-results.txt          |    12 +
 mllib/pom.xml                                      |   211 +
 .../services/org.apache.spark.ml.Estimator         |    73 +
 .../services/org.apache.spark.ml.Transformer       |    95 +
 .../org.apache.spark.ml.evaluation.Evaluator       |    26 +
 .../org.apache.spark.ml.util.MLFormatRegister      |    21 +
 ...apache.spark.sql.SparkSessionExtensionsProvider |    18 +
 ...org.apache.spark.sql.sources.DataSourceRegister |    19 +
 .../org/apache/spark/ml/feature/stopwords/README   |    12 +
 .../apache/spark/ml/feature/stopwords/danish.txt   |    94 +
 .../apache/spark/ml/feature/stopwords/dutch.txt    |   101 +
 .../apache/spark/ml/feature/stopwords/english.txt  |   181 +
 .../apache/spark/ml/feature/stopwords/finnish.txt  |   235 +
 .../apache/spark/ml/feature/stopwords/french.txt   |   156 +
 .../apache/spark/ml/feature/stopwords/german.txt   |   231 +
 .../spark/ml/feature/stopwords/hungarian.txt       |   199 +
 .../apache/spark/ml/feature/stopwords/italian.txt  |   279 +
 .../spark/ml/feature/stopwords/norwegian.txt       |   176 +
 .../spark/ml/feature/stopwords/portuguese.txt      |   203 +
 .../apache/spark/ml/feature/stopwords/russian.txt  |   151 +
 .../apache/spark/ml/feature/stopwords/spanish.txt  |   313 +
 .../apache/spark/ml/feature/stopwords/swedish.txt  |   114 +
 .../apache/spark/ml/feature/stopwords/turkish.txt  |    53 +
 .../main/scala/org/apache/spark/ml/Estimator.scala |   115 +
 .../src/main/scala/org/apache/spark/ml/Model.scala |    61 +
 .../main/scala/org/apache/spark/ml/Pipeline.scala  |   390 +
 .../main/scala/org/apache/spark/ml/Predictor.scala |   217 +
 .../scala/org/apache/spark/ml/Transformer.scala    |   125 +
 .../scala/org/apache/spark/ml/ann/BreezeUtil.scala |    65 +
 .../main/scala/org/apache/spark/ml/ann/Layer.scala |   864 +
 .../org/apache/spark/ml/ann/LossFunction.scala     |   124 +
 .../apache/spark/ml/attribute/AttributeGroup.scala |   247 +
 .../apache/spark/ml/attribute/AttributeKeys.scala  |    37 +
 .../apache/spark/ml/attribute/AttributeType.scala  |    70 +
 .../org/apache/spark/ml/attribute/attributes.scala |   599 +
 .../apache/spark/ml/attribute/package-info.java    |    41 +
 .../org/apache/spark/ml/attribute/package.scala    |    41 +
 .../ml/classification/ClassificationSummary.scala  |   247 +
 .../spark/ml/classification/Classifier.scala       |   214 +
 .../ml/classification/DecisionTreeClassifier.scala |   343 +
 .../spark/ml/classification/FMClassifier.scala     |   482 +
 .../spark/ml/classification/GBTClassifier.scala    |   468 +
 .../apache/spark/ml/classification/LinearSVC.scala |   578 +
 .../ml/classification/LogisticRegression.scala     |  1584 +
 .../MultilayerPerceptronClassifier.scala           |   509 +
 .../spark/ml/classification/NaiveBayes.scala       |   675 +
 .../apache/spark/ml/classification/OneVsRest.scala |   516 +
 .../classification/ProbabilisticClassifier.scala   |   282 +
 .../ml/classification/RandomForestClassifier.scala |   590 +
 .../spark/ml/clustering/BisectingKMeans.scala      |   359 +
 .../spark/ml/clustering/ClusteringSummary.scala    |    55 +
 .../spark/ml/clustering/GaussianMixture.scala      |   792 +
 .../org/apache/spark/ml/clustering/KMeans.scala    |   846 +
 .../scala/org/apache/spark/ml/clustering/LDA.scala |  1085 +
 .../ml/clustering/PowerIterationClustering.scala   |   223 +
 .../evaluation/BinaryClassificationEvaluator.scala |   159 +
 .../spark/ml/evaluation/ClusteringEvaluator.scala  |   159 +
 .../spark/ml/evaluation/ClusteringMetrics.scala    |   625 +
 .../org/apache/spark/ml/evaluation/Evaluator.scala |    63 +
 .../MulticlassClassificationEvaluator.scala        |   234 +
 .../MultilabelClassificationEvaluator.scala        |   169 +
 .../spark/ml/evaluation/RankingEvaluator.scala     |   150 +
 .../spark/ml/evaluation/RegressionEvaluator.scala  |   153 +
 .../main/scala/org/apache/spark/ml/events.scala    |   180 +
 .../org/apache/spark/ml/feature/Binarizer.scala    |   249 +
 .../ml/feature/BucketedRandomProjectionLSH.scala   |   267 +
 .../org/apache/spark/ml/feature/Bucketizer.scala   |   305 +
 .../apache/spark/ml/feature/ChiSqSelector.scala    |   217 +
 .../apache/spark/ml/feature/CountVectorizer.scala  |   416 +
 .../scala/org/apache/spark/ml/feature/DCT.scala    |   104 +
 .../spark/ml/feature/ElementwiseProduct.scala      |   108 +
 .../apache/spark/ml/feature/FeatureHasher.scala    |   265 +
 .../org/apache/spark/ml/feature/HashingTF.scala    |   179 +
 .../scala/org/apache/spark/ml/feature/IDF.scala    |   259 +
 .../org/apache/spark/ml/feature/Imputer.scala      |   366 +
 .../org/apache/spark/ml/feature/Instance.scala     |   211 +
 .../org/apache/spark/ml/feature/Interaction.scala  |   302 +
 .../scala/org/apache/spark/ml/feature/LSH.scala    |   353 +
 .../org/apache/spark/ml/feature/LabeledPoint.scala |    49 +
 .../org/apache/spark/ml/feature/MaxAbsScaler.scala |   206 +
 .../org/apache/spark/ml/feature/MinHashLSH.scala   |   264 +
 .../org/apache/spark/ml/feature/MinMaxScaler.scala |   292 +
 .../scala/org/apache/spark/ml/feature/NGram.scala  |    86 +
 .../org/apache/spark/ml/feature/Normalizer.scala   |    96 +
 .../apache/spark/ml/feature/OneHotEncoder.scala    |   578 +
 .../scala/org/apache/spark/ml/feature/PCA.scala    |   247 +
 .../spark/ml/feature/PolynomialExpansion.scala     |   204 +
 .../spark/ml/feature/QuantileDiscretizer.scala     |   277 +
 .../org/apache/spark/ml/feature/RFormula.scala     |   662 +
 .../apache/spark/ml/feature/RFormulaParser.scala   |   339 +
 .../org/apache/spark/ml/feature/RobustScaler.scala |   327 +
 .../apache/spark/ml/feature/SQLTransformer.scala   |   106 +
 .../org/apache/spark/ml/feature/Selector.scala     |   388 +
 .../apache/spark/ml/feature/StandardScaler.scala   |   338 +
 .../apache/spark/ml/feature/StopWordsRemover.scala |   240 +
 .../apache/spark/ml/feature/StringIndexer.scala    |   625 +
 .../apache/spark/ml/feature/TargetEncoder.scala    |   468 +
 .../org/apache/spark/ml/feature/Tokenizer.scala    |   180 +
 .../ml/feature/UnivariateFeatureSelector.scala     |   469 +
 .../ml/feature/VarianceThresholdSelector.scala     |   227 +
 .../apache/spark/ml/feature/VectorAssembler.scala  |   308 +
 .../apache/spark/ml/feature/VectorIndexer.scala    |   591 +
 .../apache/spark/ml/feature/VectorSizeHint.scala   |   191 +
 .../org/apache/spark/ml/feature/VectorSlicer.scala |   197 +
 .../org/apache/spark/ml/feature/Word2Vec.scala     |   450 +
 .../org/apache/spark/ml/feature/package-info.java  |   108 +
 .../org/apache/spark/ml/feature/package.scala      |    87 +
 .../scala/org/apache/spark/ml/fpm/FPGrowth.scala   |   432 +
 .../scala/org/apache/spark/ml/fpm/PrefixSpan.scala |   170 +
 .../main/scala/org/apache/spark/ml/functions.scala |    88 +
 .../org/apache/spark/ml/image/HadoopUtils.scala    |   121 +
 .../org/apache/spark/ml/image/ImageSchema.scala    |   194 +
 .../spark/ml/linalg/JsonMatrixConverter.scala      |    83 +
 .../spark/ml/linalg/JsonVectorConverter.scala      |    66 +
 .../org/apache/spark/ml/linalg/MatrixUDT.scala     |   115 +
 .../org/apache/spark/ml/linalg/SQLDataTypes.scala  |    34 +
 .../org/apache/spark/ml/linalg/VectorUDT.scala     |   104 +
 .../optim/IterativelyReweightedLeastSquares.scala  |   122 +
 .../spark/ml/optim/NormalEquationSolver.scala      |   164 +
 .../spark/ml/optim/WeightedLeastSquares.scala      |   520 +
 .../ml/optim/aggregator/AFTBlockAggregator.scala   |   138 +
 .../aggregator/BinaryLogisticBlockAggregator.scala |   154 +
 .../aggregator/DifferentiableLossAggregator.scala  |    81 +
 .../ml/optim/aggregator/HingeBlockAggregator.scala |   148 +
 .../ml/optim/aggregator/HuberBlockAggregator.scala |   160 +
 .../aggregator/LeastSquaresBlockAggregator.scala   |   134 +
 .../MultinomialLogisticBlockAggregator.scala       |   221 +
 .../optim/loss/DifferentiableRegularization.scala  |    88 +
 .../spark/ml/optim/loss/RDDLossFunction.scala      |    71 +
 .../scala/org/apache/spark/ml/package-info.java    |    22 +
 .../main/scala/org/apache/spark/ml/package.scala   |    50 +
 .../scala/org/apache/spark/ml/param/params.scala   |  1084 +
 .../spark/ml/param/shared/HasParallelism.scala     |    59 +
 .../ml/param/shared/SharedParamsCodeGen.scala      |   264 +
 .../spark/ml/param/shared/sharedParams.scala       |   583 +
 .../scala/org/apache/spark/ml/python/MLSerDe.scala |   218 +
 .../scala/org/apache/spark/ml/python/MLUtil.scala  |    43 +
 .../spark/ml/r/AFTSurvivalRegressionWrapper.scala  |   156 +
 .../scala/org/apache/spark/ml/r/ALSWrapper.scala   |   122 +
 .../apache/spark/ml/r/BisectingKMeansWrapper.scala |   147 +
 .../spark/ml/r/DecisionTreeClassifierWrapper.scala |   158 +
 .../spark/ml/r/DecisionTreeRegressorWrapper.scala  |   141 +
 .../apache/spark/ml/r/FMClassifierWrapper.scala    |   179 +
 .../org/apache/spark/ml/r/FMRegressorWrapper.scala |   159 +
 .../org/apache/spark/ml/r/FPGrowthWrapper.scala    |    87 +
 .../apache/spark/ml/r/GBTClassifierWrapper.scala   |   165 +
 .../apache/spark/ml/r/GBTRegressorWrapper.scala    |   149 +
 .../apache/spark/ml/r/GaussianMixtureWrapper.scala |   139 +
 .../ml/r/GeneralizedLinearRegressionWrapper.scala  |   209 +
 .../spark/ml/r/IsotonicRegressionWrapper.scala     |   126 +
 .../org/apache/spark/ml/r/KMeansWrapper.scala      |   149 +
 .../org/apache/spark/ml/r/KSTestWrapper.scala      |    58 +
 .../scala/org/apache/spark/ml/r/LDAWrapper.scala   |   227 +
 .../spark/ml/r/LinearRegressionWrapper.scala       |   154 +
 .../org/apache/spark/ml/r/LinearSVCWrapper.scala   |   166 +
 .../spark/ml/r/LogisticRegressionWrapper.scala     |   220 +
 .../r/MultilayerPerceptronClassifierWrapper.scala  |   152 +
 .../org/apache/spark/ml/r/NaiveBayesWrapper.scala  |   130 +
 .../ml/r/PowerIterationClusteringWrapper.scala     |    39 +
 .../org/apache/spark/ml/r/PrefixSpanWrapper.scala  |    34 +
 .../org/apache/spark/ml/r/RWrapperUtils.scala      |    74 +
 .../scala/org/apache/spark/ml/r/RWrappers.scala    |    88 +
 .../spark/ml/r/RandomForestClassifierWrapper.scala |   169 +
 .../spark/ml/r/RandomForestRegressorWrapper.scala  |   152 +
 .../org/apache/spark/ml/recommendation/ALS.scala   |  1913 ++
 .../ml/regression/AFTSurvivalRegression.scala      |   570 +
 .../ml/regression/DecisionTreeRegressor.scala      |   350 +
 .../apache/spark/ml/regression/FMRegressor.scala   |   853 +
 .../apache/spark/ml/regression/GBTRegressor.scala  |   421 +
 .../regression/GeneralizedLinearRegression.scala   |  1672 ++
 .../spark/ml/regression/IsotonicRegression.scala   |   351 +
 .../spark/ml/regression/LinearRegression.scala     |  1163 +
 .../ml/regression/RandomForestRegressor.scala      |   373 +
 .../org/apache/spark/ml/regression/Regressor.scala |    49 +
 .../spark/ml/source/image/ImageDataSource.scala    |    56 +
 .../spark/ml/source/image/ImageFileFormat.scala    |    99 +
 .../spark/ml/source/image/ImageOptions.scala       |    32 +
 .../spark/ml/source/libsvm/LibSVMDataSource.scala  |    53 +
 .../spark/ml/source/libsvm/LibSVMOptions.scala     |    51 +
 .../spark/ml/source/libsvm/LibSVMRelation.scala    |   188 +
 .../scala/org/apache/spark/ml/stat/ANOVATest.scala |   229 +
 .../org/apache/spark/ml/stat/ChiSquareTest.scala   |   101 +
 .../org/apache/spark/ml/stat/Correlation.scala     |    84 +
 .../org/apache/spark/ml/stat/FValueTest.scala      |   150 +
 .../spark/ml/stat/KolmogorovSmirnovTest.scala      |   116 +
 .../spark/ml/stat/MultiClassSummarizer.scala       |    98 +
 .../org/apache/spark/ml/stat/Summarizer.scala      |   798 +
 .../main/scala/org/apache/spark/ml/tree/Node.scala |   446 +
 .../scala/org/apache/spark/ml/tree/Split.scala     |   191 +
 .../apache/spark/ml/tree/impl/BaggedPoint.scala    |   135 +
 .../spark/ml/tree/impl/DTStatsAggregator.scala     |   187 +
 .../spark/ml/tree/impl/DecisionTreeMetadata.scala  |   247 +
 .../spark/ml/tree/impl/GradientBoostedTrees.scala  |   541 +
 .../apache/spark/ml/tree/impl/RandomForest.scala   |  1368 +
 .../apache/spark/ml/tree/impl/TimeTracker.scala    |    70 +
 .../org/apache/spark/ml/tree/impl/TreePoint.scala  |   135 +
 .../org/apache/spark/ml/tree/treeModels.scala      |   666 +
 .../org/apache/spark/ml/tree/treeParams.scala      |   612 +
 .../apache/spark/ml/tuning/CrossValidator.scala    |   477 +
 .../apache/spark/ml/tuning/ParamGridBuilder.scala  |   120 +
 .../spark/ml/tuning/TrainValidationSplit.scala     |   434 +
 .../apache/spark/ml/tuning/ValidatorParams.scala   |   214 +
 .../org/apache/spark/ml/util/ConnectHelper.scala   |   126 +
 .../org/apache/spark/ml/util/DatasetUtils.scala    |   215 +
 .../apache/spark/ml/util/HasTrainingSummary.scala  |    63 +
 .../org/apache/spark/ml/util/Identifiable.scala    |    48 +
 .../org/apache/spark/ml/util/Instrumentation.scala |   296 +
 .../org/apache/spark/ml/util/MetadataUtils.scala   |   112 +
 .../scala/org/apache/spark/ml/util/ReadWrite.scala |  1174 +
 .../org/apache/spark/ml/util/SchemaUtils.scala     |   252 +
 .../scala/org/apache/spark/ml/util/Summary.scala   |    28 +
 .../org/apache/spark/ml/util/stopwatches.scala     |   156 +
 .../scala/org/apache/spark/mllib/JavaPackage.java  |    31 +
 .../mllib/api/python/FPGrowthModelWrapper.scala    |    32 +
 .../api/python/GaussianMixtureModelWrapper.scala   |    49 +
 .../spark/mllib/api/python/LDAModelWrapper.scala   |    47 +
 .../python/MatrixFactorizationModelWrapper.scala   |    53 +
 .../PowerIterationClusteringModelWrapper.scala     |    32 +
 .../mllib/api/python/PrefixSpanModelWrapper.scala  |    32 +
 .../spark/mllib/api/python/PythonMLLibAPI.scala    |  1608 +
 .../mllib/api/python/Word2VecModelWrapper.scala    |    81 +
 .../apache/spark/mllib/api/python/package.scala    |    25 +
 .../mllib/classification/ClassificationModel.scala |    71 +
 .../mllib/classification/LogisticRegression.scala  |   356 +
 .../spark/mllib/classification/NaiveBayes.scala    |   460 +
 .../apache/spark/mllib/classification/SVM.scala    |   255 +
 .../StreamingLogisticRegressionWithSGD.scala       |   102 +
 .../impl/GLMClassificationModel.scala              |    94 +
 .../spark/mllib/clustering/BisectingKMeans.scala   |   566 +
 .../mllib/clustering/BisectingKMeansModel.scala    |   281 +
 .../spark/mllib/clustering/DistanceMeasure.scala   |   537 +
 .../spark/mllib/clustering/GaussianMixture.scala   |   351 +
 .../mllib/clustering/GaussianMixtureModel.scala    |   199 +
 .../org/apache/spark/mllib/clustering/KMeans.scala |   556 +
 .../spark/mllib/clustering/KMeansModel.scala       |   234 +
 .../org/apache/spark/mllib/clustering/LDA.scala    |   445 +
 .../apache/spark/mllib/clustering/LDAModel.scala   |  1024 +
 .../spark/mllib/clustering/LDAOptimizer.scala      |   660 +
 .../apache/spark/mllib/clustering/LDAUtils.scala   |    55 +
 .../spark/mllib/clustering/LocalKMeans.scala       |   135 +
 .../clustering/PowerIterationClustering.scala      |   451 +
 .../spark/mllib/clustering/StreamingKMeans.scala   |   351 +
 .../spark/mllib/evaluation/AreaUnderCurve.scala    |    85 +
 .../evaluation/BinaryClassificationMetrics.scala   |   279 +
 .../spark/mllib/evaluation/MulticlassMetrics.scala |   294 +
 .../spark/mllib/evaluation/MultilabelMetrics.scala |   283 +
 .../spark/mllib/evaluation/RankingMetrics.scala    |   295 +
 .../spark/mllib/evaluation/RegressionMetrics.scala |   138 +
 .../BinaryClassificationMetricComputers.scala      |    78 +
 .../evaluation/binary/BinaryConfusionMatrix.scala  |    72 +
 .../evaluation/binary/BinaryLabelCounter.scala     |    58 +
 .../apache/spark/mllib/feature/ChiSqSelector.scala |   315 +
 .../spark/mllib/feature/ElementwiseProduct.scala   |    79 +
 .../org/apache/spark/mllib/feature/HashingTF.scala |   173 +
 .../scala/org/apache/spark/mllib/feature/IDF.scala |   259 +
 .../apache/spark/mllib/feature/Normalizer.scala    |    84 +
 .../scala/org/apache/spark/mllib/feature/PCA.scala |   132 +
 .../spark/mllib/feature/StandardScaler.scala       |   177 +
 .../spark/mllib/feature/VectorTransformer.scala    |    64 +
 .../org/apache/spark/mllib/feature/Word2Vec.scala  |   738 +
 .../apache/spark/mllib/fpm/AssociationRules.scala  |   174 +
 .../org/apache/spark/mllib/fpm/FPGrowth.scala      |   345 +
 .../scala/org/apache/spark/mllib/fpm/FPTree.scala  |   134 +
 .../apache/spark/mllib/fpm/LocalPrefixSpan.scala   |   110 +
 .../org/apache/spark/mllib/fpm/PrefixSpan.scala    |   697 +
 .../org/apache/spark/mllib/linalg/ARPACK.scala     |    44 +
 .../scala/org/apache/spark/mllib/linalg/BLAS.scala |   735 +
 .../spark/mllib/linalg/CholeskyDecomposition.scala |    71 +
 .../mllib/linalg/EigenValueDecomposition.scala     |   153 +
 .../org/apache/spark/mllib/linalg/LAPACK.scala     |    44 +
 .../org/apache/spark/mllib/linalg/Matrices.scala   |  1260 +
 .../mllib/linalg/SingularValueDecomposition.scala  |    33 +
 .../org/apache/spark/mllib/linalg/Vectors.scala    |  1051 +
 .../mllib/linalg/distributed/BlockMatrix.scala     |   550 +
 .../linalg/distributed/CoordinateMatrix.scala      |   176 +
 .../linalg/distributed/DistributedMatrix.scala     |    40 +
 .../linalg/distributed/IndexedRowMatrix.scala      |   280 +
 .../spark/mllib/linalg/distributed/RowMatrix.scala |   889 +
 .../apache/spark/mllib/optimization/Gradient.scala |   335 +
 .../spark/mllib/optimization/GradientDescent.scala |   341 +
 .../apache/spark/mllib/optimization/LBFGS.scala    |   300 +
 .../org/apache/spark/mllib/optimization/NNLS.scala |   174 +
 .../spark/mllib/optimization/Optimizer.scala       |    32 +
 .../apache/spark/mllib/optimization/Updater.scala  |   148 +
 .../scala/org/apache/spark/mllib/package-info.java |    41 +
 .../scala/org/apache/spark/mllib/package.scala     |    38 +
 .../apache/spark/mllib/pmml/PMMLExportable.scala   |    81 +
 .../BinaryClassificationPMMLModelExport.scala      |    92 +
 .../export/GeneralizedLinearPMMLModelExport.scala  |    77 +
 .../mllib/pmml/export/KMeansPMMLModelExport.scala  |    85 +
 .../spark/mllib/pmml/export/PMMLModelExport.scala  |    49 +
 .../mllib/pmml/export/PMMLModelExportFactory.scala |    64 +
 .../spark/mllib/random/RandomDataGenerator.scala   |   204 +
 .../org/apache/spark/mllib/random/RandomRDDs.scala |   893 +
 .../spark/mllib/rdd/MLPairRDDFunctions.scala       |    73 +
 .../org/apache/spark/mllib/rdd/RDDFunctions.scala  |    59 +
 .../org/apache/spark/mllib/rdd/RandomRDD.scala     |   120 +
 .../org/apache/spark/mllib/rdd/SlidingRDD.scala    |   112 +
 .../apache/spark/mllib/recommendation/ALS.scala    |   467 +
 .../recommendation/MatrixFactorizationModel.scala  |   422 +
 .../regression/GeneralizedLinearAlgorithm.scala    |   370 +
 .../mllib/regression/IsotonicRegression.scala      |   533 +
 .../spark/mllib/regression/LabeledPoint.scala      |    81 +
 .../org/apache/spark/mllib/regression/Lasso.scala  |   104 +
 .../spark/mllib/regression/LinearRegression.scala  |   106 +
 .../spark/mllib/regression/RegressionModel.scala   |    70 +
 .../spark/mllib/regression/RidgeRegression.scala   |   104 +
 .../regression/StreamingLinearAlgorithm.scala      |   160 +
 .../StreamingLinearRegressionWithSGD.scala         |   118 +
 .../mllib/regression/impl/GLMRegressionModel.scala |    86 +
 .../apache/spark/mllib/stat/KernelDensity.scala    |   122 +
 .../mllib/stat/MultivariateOnlineSummarizer.scala  |   305 +
 .../stat/MultivariateStatisticalSummary.scala      |    82 +
 .../org/apache/spark/mllib/stat/Statistics.scala   |   253 +
 .../spark/mllib/stat/correlation/Correlation.scala |    96 +
 .../stat/correlation/PearsonCorrelation.scala      |   107 +
 .../stat/correlation/SpearmanCorrelation.scala     |    90 +
 .../stat/distribution/MultivariateGaussian.scala   |   142 +
 .../apache/spark/mllib/stat/test/ChiSqTest.scala   |   321 +
 .../mllib/stat/test/KolmogorovSmirnovTest.scala    |   195 +
 .../spark/mllib/stat/test/StreamingTest.scala      |   174 +
 .../mllib/stat/test/StreamingTestMethod.scala      |   167 +
 .../apache/spark/mllib/stat/test/TestResult.scala  |   131 +
 .../org/apache/spark/mllib/tree/DecisionTree.scala |   278 +
 .../spark/mllib/tree/GradientBoostedTrees.scala    |   150 +
 .../org/apache/spark/mllib/tree/RandomForest.scala |   290 +
 .../spark/mllib/tree/configuration/Algo.scala      |    37 +
 .../tree/configuration/BoostingStrategy.scala      |   113 +
 .../configuration/EnsembleCombiningStrategy.scala  |    26 +
 .../mllib/tree/configuration/FeatureType.scala     |    31 +
 .../tree/configuration/QuantileStrategy.scala      |    31 +
 .../spark/mllib/tree/configuration/Strategy.scala  |   234 +
 .../apache/spark/mllib/tree/impurity/Entropy.scala |   171 +
 .../apache/spark/mllib/tree/impurity/Gini.scala    |   166 +
 .../spark/mllib/tree/impurity/Impurities.scala     |    32 +
 .../spark/mllib/tree/impurity/Impurity.scala       |   209 +
 .../spark/mllib/tree/impurity/Variance.scala       |   144 +
 .../spark/mllib/tree/loss/AbsoluteError.scala      |    50 +
 .../org/apache/spark/mllib/tree/loss/LogLoss.scala |    59 +
 .../org/apache/spark/mllib/tree/loss/Loss.scala    |    73 +
 .../org/apache/spark/mllib/tree/loss/Losses.scala  |    33 +
 .../spark/mllib/tree/loss/SquaredError.scala       |    50 +
 .../spark/mllib/tree/model/DecisionTreeModel.scala |   334 +
 .../mllib/tree/model/InformationGainStats.scala    |   126 +
 .../org/apache/spark/mllib/tree/model/Node.scala   |   245 +
 .../apache/spark/mllib/tree/model/Predict.scala    |    46 +
 .../org/apache/spark/mllib/tree/model/Split.scala  |    67 +
 .../mllib/tree/model/treeEnsembleModels.scala      |   469 +
 .../org/apache/spark/mllib/tree/package.scala      |    29 +
 .../apache/spark/mllib/util/DataValidators.scala   |    64 +
 .../spark/mllib/util/KMeansDataGenerator.scala     |    91 +
 .../spark/mllib/util/LinearDataGenerator.scala     |   211 +
 .../util/LogisticRegressionDataGenerator.scala     |    89 +
 .../apache/spark/mllib/util/MFDataGenerator.scala  |   117 +
 .../org/apache/spark/mllib/util/MLUtils.scala      |   607 +
 .../apache/spark/mllib/util/NumericParser.scala    |   123 +
 .../apache/spark/mllib/util/SVMDataGenerator.scala |    73 +
 .../apache/spark/mllib/util/modelSaveLoad.scala    |   126 +
 .../sql/ml/InternalFunctionRegistration.scala      |   107 +
 .../java/org/apache/spark/SharedSparkSession.java  |    53 +
 .../org/apache/spark/ml/JavaPipelineSuite.java     |    64 +
 .../ml/attribute/JavaAttributeGroupSuite.java      |    45 +
 .../spark/ml/attribute/JavaAttributeSuite.java     |    55 +
 .../JavaDecisionTreeClassifierSuite.java           |    79 +
 .../ml/classification/JavaGBTClassifierSuite.java  |    83 +
 .../JavaLogisticRegressionSuite.java               |   149 +
 .../JavaMultilayerPerceptronClassifierSuite.java   |    56 +
 .../ml/classification/JavaNaiveBayesSuite.java     |    79 +
 .../ml/classification/JavaOneVsRestSuite.java      |    74 +
 .../JavaRandomForestClassifierSuite.java           |   103 +
 .../spark/ml/clustering/JavaKMeansSuite.java       |    61 +
 .../spark/ml/feature/JavaBucketizerSuite.java      |    99 +
 .../org/apache/spark/ml/feature/JavaDCTSuite.java  |    62 +
 .../spark/ml/feature/JavaHashingTFSuite.java       |    70 +
 .../spark/ml/feature/JavaNormalizerSuite.java      |    54 +
 .../org/apache/spark/ml/feature/JavaPCASuite.java  |   101 +
 .../ml/feature/JavaPolynomialExpansionSuite.java   |    75 +
 .../spark/ml/feature/JavaStandardScalerSuite.java  |    55 +
 .../ml/feature/JavaStopWordsRemoverSuite.java      |    55 +
 .../spark/ml/feature/JavaStringIndexerSuite.java   |    63 +
 .../spark/ml/feature/JavaTargetEncoderSuite.java   |   157 +
 .../spark/ml/feature/JavaTokenizerSuite.java       |    58 +
 .../spark/ml/feature/JavaVectorAssemblerSuite.java |    61 +
 .../spark/ml/feature/JavaVectorIndexerSuite.java   |    55 +
 .../spark/ml/feature/JavaVectorSlicerSuite.java    |    69 +
 .../apache/spark/ml/feature/JavaWord2VecSuite.java |    59 +
 .../spark/ml/linalg/JavaSQLDataTypesSuite.java     |    31 +
 .../org/apache/spark/ml/param/JavaParamsSuite.java |    51 +
 .../org/apache/spark/ml/param/JavaTestParams.java  |   126 +
 .../regression/JavaDecisionTreeRegressorSuite.java |    80 +
 .../spark/ml/regression/JavaGBTRegressorSuite.java |    84 +
 .../ml/regression/JavaLinearRegressionSuite.java   |    81 +
 .../regression/JavaRandomForestRegressorSuite.java |   104 +
 .../ml/source/libsvm/JavaLibSVMRelationSuite.java  |    72 +
 .../ml/stat/JavaKolmogorovSmirnovTestSuite.java    |    80 +
 .../apache/spark/ml/stat/JavaSummarizerSuite.java  |    66 +
 .../spark/ml/tuning/JavaCrossValidatorSuite.java   |    67 +
 .../spark/ml/util/JavaDefaultReadWriteSuite.java   |    63 +
 .../JavaLogisticRegressionSuite.java               |    78 +
 .../mllib/classification/JavaNaiveBayesSuite.java  |    94 +
 .../spark/mllib/classification/JavaSVMSuite.java   |    80 +
 .../JavaStreamingLogisticRegressionSuite.java      |    79 +
 .../mllib/clustering/JavaBisectingKMeansSuite.java |    58 +
 .../mllib/clustering/JavaGaussianMixtureSuite.java |    49 +
 .../spark/mllib/clustering/JavaKMeansSuite.java    |    89 +
 .../spark/mllib/clustering/JavaLDASuite.java       |   194 +
 .../mllib/clustering/JavaStreamingKMeansSuite.java |    79 +
 .../mllib/evaluation/JavaRankingMetricsSuite.java  |    83 +
 .../apache/spark/mllib/feature/JavaTfIdfSuite.java |    70 +
 .../spark/mllib/feature/JavaWord2VecSuite.java     |    49 +
 .../spark/mllib/fpm/JavaAssociationRulesSuite.java |    40 +
 .../apache/spark/mllib/fpm/JavaFPGrowthSuite.java  |    98 +
 .../spark/mllib/fpm/JavaPrefixSpanSuite.java       |    92 +
 .../spark/mllib/linalg/JavaMatricesSuite.java      |   164 +
 .../spark/mllib/linalg/JavaVectorsSuite.java       |    43 +
 .../linalg/distributed/JavaRowMatrixSuite.java     |    44 +
 .../spark/mllib/random/JavaRandomRDDsSuite.java    |   262 +
 .../spark/mllib/recommendation/JavaALSSuite.java   |   188 +
 .../regression/JavaIsotonicRegressionSuite.java    |    75 +
 .../spark/mllib/regression/JavaLassoSuite.java     |    78 +
 .../regression/JavaLinearRegressionSuite.java      |    96 +
 .../mllib/regression/JavaRidgeRegressionSuite.java |    94 +
 .../JavaStreamingLinearRegressionSuite.java        |    78 +
 .../spark/mllib/stat/JavaStatisticsSuite.java      |   114 +
 .../spark/mllib/tree/JavaDecisionTreeSuite.java    |    90 +
 .../apache/spark/mllib/util/JavaMLUtilsSuite.java  |    74 +
 .../org.apache.spark.ml.util.MLFormatRegister      |    20 +
 .../date=2018-01/29.5.a_b_EGDP022204.jpg           |   Bin 0 -> 27295 bytes
 .../cls=kittens/date=2018-01/not-image.txt         |     1 +
 .../partitioned/cls=kittens/date=2018-02/54893.jpg |   Bin 0 -> 35914 bytes
 .../cls=kittens/date=2018-02/DP153539.jpg          |   Bin 0 -> 26354 bytes
 .../cls=kittens/date=2018-02/DP802813.jpg          |   Bin 0 -> 30432 bytes
 .../cls=multichannel/date=2018-01/BGRA.png         |   Bin 0 -> 683 bytes
 .../date=2018-01/BGRA_alpha_60.png                 |   Bin 0 -> 747 bytes
 .../cls=multichannel/date=2018-02/chr30.4.184.jpg  |   Bin 0 -> 59472 bytes
 .../cls=multichannel/date=2018-02/grayscale.jpg    |   Bin 0 -> 36728 bytes
 mllib/src/test/resources/iris_libsvm.txt           |   150 +
 mllib/src/test/resources/log4j2.properties         |    32 +
 .../ml-models/dtc-2.4.7/data/._SUCCESS.crc         |   Bin 0 -> 8 bytes
 ...-406c-894c-ca4eac67c690-c000.snappy.parquet.crc |   Bin 0 -> 36 bytes
 .../resources/ml-models/dtc-2.4.7/data/_SUCCESS    |     0
 ...c890-406c-894c-ca4eac67c690-c000.snappy.parquet |   Bin 0 -> 3242 bytes
 .../ml-models/dtc-2.4.7/metadata/._SUCCESS.crc     |   Bin 0 -> 8 bytes
 .../ml-models/dtc-2.4.7/metadata/.part-00000.crc   |   Bin 0 -> 16 bytes
 .../ml-models/dtc-2.4.7/metadata/_SUCCESS          |     0
 .../ml-models/dtc-2.4.7/metadata/part-00000        |     1 +
 .../ml-models/dtr-2.4.7/data/._SUCCESS.crc         |   Bin 0 -> 8 bytes
 ...-4b3d-84af-d861adcb9ca8-c000.snappy.parquet.crc |   Bin 0 -> 36 bytes
 .../resources/ml-models/dtr-2.4.7/data/_SUCCESS    |     0
 ...a437-4b3d-84af-d861adcb9ca8-c000.snappy.parquet |   Bin 0 -> 3264 bytes
 .../ml-models/dtr-2.4.7/metadata/._SUCCESS.crc     |   Bin 0 -> 8 bytes
 .../ml-models/dtr-2.4.7/metadata/.part-00000.crc   |   Bin 0 -> 12 bytes
 .../ml-models/dtr-2.4.7/metadata/_SUCCESS          |     0
 .../ml-models/dtr-2.4.7/metadata/part-00000        |     1 +
 .../ml-models/gbtc-2.4.7/data/._SUCCESS.crc        |   Bin 0 -> 8 bytes
 ...-41c7-91c0-6da8cc01fb43-c000.snappy.parquet.crc |   Bin 0 -> 44 bytes
 .../resources/ml-models/gbtc-2.4.7/data/_SUCCESS   |     0
 ...c861-41c7-91c0-6da8cc01fb43-c000.snappy.parquet |   Bin 0 -> 4542 bytes
 .../ml-models/gbtc-2.4.7/metadata/._SUCCESS.crc    |   Bin 0 -> 8 bytes
 .../ml-models/gbtc-2.4.7/metadata/.part-00000.crc  |   Bin 0 -> 16 bytes
 .../ml-models/gbtc-2.4.7/metadata/_SUCCESS         |     0
 .../ml-models/gbtc-2.4.7/metadata/part-00000       |     1 +
 .../gbtc-2.4.7/treesMetadata/._SUCCESS.crc         |   Bin 0 -> 8 bytes
 ...-4a90-813c-ddc394101e21-c000.snappy.parquet.crc |   Bin 0 -> 36 bytes
 .../ml-models/gbtc-2.4.7/treesMetadata/_SUCCESS    |     0
 ...31e3-4a90-813c-ddc394101e21-c000.snappy.parquet |   Bin 0 -> 3075 bytes
 .../ml-models/gbtr-2.4.7/data/._SUCCESS.crc        |   Bin 0 -> 8 bytes
 ...-4511-9aab-639288bfae6d-c000.snappy.parquet.crc |   Bin 0 -> 40 bytes
 .../resources/ml-models/gbtr-2.4.7/data/_SUCCESS   |     0
 ...d346-4511-9aab-639288bfae6d-c000.snappy.parquet |   Bin 0 -> 3740 bytes
 .../ml-models/gbtr-2.4.7/metadata/._SUCCESS.crc    |   Bin 0 -> 8 bytes
 .../ml-models/gbtr-2.4.7/metadata/.part-00000.crc  |   Bin 0 -> 16 bytes
 .../ml-models/gbtr-2.4.7/metadata/_SUCCESS         |     0
 .../ml-models/gbtr-2.4.7/metadata/part-00000       |     1 +
 .../gbtr-2.4.7/treesMetadata/._SUCCESS.crc         |   Bin 0 -> 8 bytes
 ...-4fd8-ad9c-4be239c2215a-c000.snappy.parquet.crc |   Bin 0 -> 32 bytes
 .../ml-models/gbtr-2.4.7/treesMetadata/_SUCCESS    |     0
 ...87fe-4fd8-ad9c-4be239c2215a-c000.snappy.parquet |   Bin 0 -> 3038 bytes
 .../hashingTF-2.4.4/metadata/._SUCCESS.crc         |   Bin 0 -> 8 bytes
 .../hashingTF-2.4.4/metadata/.part-00000.crc       |   Bin 0 -> 12 bytes
 .../ml-models/hashingTF-2.4.4/metadata/_SUCCESS    |     0
 .../ml-models/hashingTF-2.4.4/metadata/part-00000  |     1 +
 .../ml-models/mlp-2.4.4/data/._SUCCESS.crc         |   Bin 0 -> 8 bytes
 ...-4b90-8231-eb5f6ac12138-c000.snappy.parquet.crc |   Bin 0 -> 24 bytes
 .../resources/ml-models/mlp-2.4.4/data/_SUCCESS    |     0
 ...d8df-4b90-8231-eb5f6ac12138-c000.snappy.parquet |   Bin 0 -> 2004 bytes
 .../ml-models/mlp-2.4.4/metadata/._SUCCESS.crc     |   Bin 0 -> 8 bytes
 .../ml-models/mlp-2.4.4/metadata/.part-00000.crc   |   Bin 0 -> 12 bytes
 .../ml-models/mlp-2.4.4/metadata/_SUCCESS          |     0
 .../ml-models/mlp-2.4.4/metadata/part-00000        |     1 +
 .../ml-models/rfc-2.4.7/data/._SUCCESS.crc         |   Bin 0 -> 8 bytes
 ...-4485-b112-25b4b11c9009-c000.snappy.parquet.crc |   Bin 0 -> 40 bytes
 .../resources/ml-models/rfc-2.4.7/data/_SUCCESS    |     0
 ...91f8-4485-b112-25b4b11c9009-c000.snappy.parquet |   Bin 0 -> 3836 bytes
 .../ml-models/rfc-2.4.7/metadata/._SUCCESS.crc     |   Bin 0 -> 8 bytes
 .../ml-models/rfc-2.4.7/metadata/.part-00000.crc   |   Bin 0 -> 16 bytes
 .../ml-models/rfc-2.4.7/metadata/_SUCCESS          |     0
 .../ml-models/rfc-2.4.7/metadata/part-00000        |     1 +
 .../rfc-2.4.7/treesMetadata/._SUCCESS.crc          |   Bin 0 -> 8 bytes
 ...-4c4e-a823-70c7afdcbdc5-c000.snappy.parquet.crc |   Bin 0 -> 36 bytes
 .../ml-models/rfc-2.4.7/treesMetadata/_SUCCESS     |     0
 ...b666-4c4e-a823-70c7afdcbdc5-c000.snappy.parquet |   Bin 0 -> 3391 bytes
 .../ml-models/rfr-2.4.7/data/._SUCCESS.crc         |   Bin 0 -> 8 bytes
 ...-40fc-b681-981caaeca996-c000.snappy.parquet.crc |   Bin 0 -> 40 bytes
 .../resources/ml-models/rfr-2.4.7/data/_SUCCESS    |     0
 ...6edb-40fc-b681-981caaeca996-c000.snappy.parquet |   Bin 0 -> 3797 bytes
 .../ml-models/rfr-2.4.7/metadata/._SUCCESS.crc     |   Bin 0 -> 8 bytes
 .../ml-models/rfr-2.4.7/metadata/.part-00000.crc   |   Bin 0 -> 16 bytes
 .../ml-models/rfr-2.4.7/metadata/_SUCCESS          |     0
 .../ml-models/rfr-2.4.7/metadata/part-00000        |     1 +
 .../rfr-2.4.7/treesMetadata/._SUCCESS.crc          |   Bin 0 -> 8 bytes
 ...-447a-9b86-d95edaabcde8-c000.snappy.parquet.crc |   Bin 0 -> 32 bytes
 .../ml-models/rfr-2.4.7/treesMetadata/_SUCCESS     |     0
 ...d349-447a-9b86-d95edaabcde8-c000.snappy.parquet |   Bin 0 -> 3055 bytes
 .../strIndexerModel-2.4.4/data/._SUCCESS.crc       |   Bin 0 -> 8 bytes
 ...-4756-b9ca-c5e505dcd898-c000.snappy.parquet.crc |   Bin 0 -> 12 bytes
 .../ml-models/strIndexerModel-2.4.4/data/_SUCCESS  |     0
 ...6e17-4756-b9ca-c5e505dcd898-c000.snappy.parquet |   Bin 0 -> 478 bytes
 .../strIndexerModel-2.4.4/metadata/._SUCCESS.crc   |   Bin 0 -> 8 bytes
 .../strIndexerModel-2.4.4/metadata/.part-00000.crc |   Bin 0 -> 12 bytes
 .../strIndexerModel-2.4.4/metadata/_SUCCESS        |     0
 .../strIndexerModel-2.4.4/metadata/part-00000      |     1 +
 .../scala/org/apache/spark/ml/FunctionsSuite.scala |   133 +
 .../scala/org/apache/spark/ml/MLEventsSuite.scala  |   333 +
 .../scala/org/apache/spark/ml/PipelineSuite.scala  |   276 +
 .../scala/org/apache/spark/ml/PredictorSuite.scala |    90 +
 .../scala/org/apache/spark/ml/ann/ANNSuite.scala   |    92 +
 .../org/apache/spark/ml/ann/GradientSuite.scala    |    76 +
 .../spark/ml/attribute/AttributeGroupSuite.scala   |    65 +
 .../apache/spark/ml/attribute/AttributeSuite.scala |   242 +
 .../spark/ml/classification/ClassifierSuite.scala  |   106 +
 .../DecisionTreeClassifierSuite.scala              |   502 +
 .../ml/classification/FMClassifierSuite.scala      |   328 +
 .../ml/classification/GBTClassifierSuite.scala     |   567 +
 .../spark/ml/classification/LinearSVCSuite.scala   |   430 +
 .../classification/LogisticRegressionSuite.scala   |  3278 ++
 .../MultilayerPerceptronClassifierSuite.scala      |   320 +
 .../spark/ml/classification/NaiveBayesSuite.scala  |   751 +
 .../spark/ml/classification/OneVsRestSuite.scala   |   370 +
 .../ProbabilisticClassifierSuite.scala             |   169 +
 .../RandomForestClassifierSuite.scala              |   487 +
 .../spark/ml/clustering/BisectingKMeansSuite.scala |   352 +
 .../spark/ml/clustering/GaussianMixtureSuite.scala |   346 +
 .../apache/spark/ml/clustering/KMeansSuite.scala   |   528 +
 .../org/apache/spark/ml/clustering/LDASuite.scala  |   343 +
 .../clustering/PowerIterationClusteringSuite.scala |   302 +
 .../BinaryClassificationEvaluatorSuite.scala       |   128 +
 .../ml/evaluation/ClusteringEvaluatorSuite.scala   |   205 +
 .../MulticlassClassificationEvaluatorSuite.scala   |   112 +
 .../MultilabelClassificationEvaluatorSuite.scala   |   110 +
 .../ml/evaluation/RankingEvaluatorSuite.scala      |   100 +
 .../ml/evaluation/RegressionEvaluatorSuite.scala   |   129 +
 .../apache/spark/ml/feature/BinarizerSuite.scala   |   270 +
 .../feature/BucketedRandomProjectionLSHSuite.scala |   238 +
 .../apache/spark/ml/feature/BucketizerSuite.scala  |   505 +
 .../spark/ml/feature/ChiSqSelectorSuite.scala      |   208 +
 .../spark/ml/feature/CountVectorizerSuite.scala    |   353 +
 .../org/apache/spark/ml/feature/DCTSuite.scala     |    98 +
 .../spark/ml/feature/ElementwiseProductSuite.scala |    53 +
 .../spark/ml/feature/FeatureHasherSuite.scala      |   248 +
 .../apache/spark/ml/feature/HashingTFSuite.scala   |   125 +
 .../org/apache/spark/ml/feature/IDFSuite.scala     |   131 +
 .../org/apache/spark/ml/feature/ImputerSuite.scala |   535 +
 .../apache/spark/ml/feature/InstanceSuite.scala    |   132 +
 .../apache/spark/ml/feature/InteractionSuite.scala |   205 +
 .../org/apache/spark/ml/feature/LSHTest.scala      |   169 +
 .../spark/ml/feature/LabeledPointSuite.scala       |    40 +
 .../spark/ml/feature/MaxAbsScalerSuite.scala       |    75 +
 .../apache/spark/ml/feature/MinHashLSHSuite.scala  |   193 +
 .../spark/ml/feature/MinMaxScalerSuite.scala       |   121 +
 .../org/apache/spark/ml/feature/NGramSuite.scala   |    90 +
 .../apache/spark/ml/feature/NormalizerSuite.scala  |   122 +
 .../spark/ml/feature/OneHotEncoderSuite.scala      |   632 +
 .../org/apache/spark/ml/feature/PCASuite.scala     |   124 +
 .../ml/feature/PolynomialExpansionSuite.scala      |   145 +
 .../ml/feature/QuantileDiscretizerSuite.scala      |   576 +
 .../spark/ml/feature/RFormulaParserSuite.scala     |   201 +
 .../apache/spark/ml/feature/RFormulaSuite.scala    |   641 +
 .../spark/ml/feature/RobustScalerSuite.scala       |   278 +
 .../spark/ml/feature/SQLTransformerSuite.scala     |    80 +
 .../spark/ml/feature/StandardScalerSuite.scala     |   153 +
 .../spark/ml/feature/StopWordsRemoverSuite.scala   |   370 +
 .../spark/ml/feature/StringIndexerSuite.scala      |   533 +
 .../spark/ml/feature/TargetEncoderSuite.scala      |   524 +
 .../apache/spark/ml/feature/TokenizerSuite.scala   |   110 +
 .../feature/UnivariateFeatureSelectorSuite.scala   |   689 +
 .../feature/VarianceThresholdSelectorSuite.scala   |   141 +
 .../spark/ml/feature/VectorAssemblerSuite.scala    |   291 +
 .../spark/ml/feature/VectorIndexerSuite.scala      |   346 +
 .../spark/ml/feature/VectorSizeHintSuite.scala     |   221 +
 .../spark/ml/feature/VectorSlicerSuite.scala       |   122 +
 .../apache/spark/ml/feature/Word2VecSuite.scala    |   229 +
 .../org/apache/spark/ml/fpm/FPGrowthSuite.scala    |   194 +
 .../org/apache/spark/ml/fpm/PrefixSpanSuite.scala  |   147 +
 .../spark/ml/linalg/JsonMatrixConverterSuite.scala |    45 +
 .../spark/ml/linalg/JsonVectorConverterSuite.scala |    41 +
 .../apache/spark/ml/linalg/MatrixUDTSuite.scala    |    41 +
 .../apache/spark/ml/linalg/SQLDataTypesSuite.scala |    27 +
 .../apache/spark/ml/linalg/VectorUDTSuite.scala    |    47 +
 .../IterativelyReweightedLeastSquaresSuite.scala   |   200 +
 .../spark/ml/optim/WeightedLeastSquaresSuite.scala |   546 +
 .../BinaryLogisticBlockAggregatorSuite.scala       |   305 +
 .../DifferentiableLossAggregatorSuite.scala        |   160 +
 .../aggregator/HingeBlockAggregatorSuite.scala     |   260 +
 .../aggregator/HuberBlockAggregatorSuite.scala     |   291 +
 .../LeastSquaresBlockAggregatorSuite.scala         |   180 +
 .../MultinomialLogisticBlockAggregatorSuite.scala  |   389 +
 .../loss/DifferentiableRegularizationSuite.scala   |    62 +
 .../spark/ml/optim/loss/RDDLossFunctionSuite.scala |    83 +
 .../org/apache/spark/ml/param/ParamsSuite.scala    |   455 +
 .../org/apache/spark/ml/param/TestParams.scala     |    38 +
 .../spark/ml/param/shared/SharedParamsSuite.scala  |    36 +
 .../org/apache/spark/ml/python/MLSerDeSuite.scala  |    72 +
 .../org/apache/spark/ml/r/RWrapperUtilsSuite.scala |    48 +
 .../apache/spark/ml/recommendation/ALSSuite.scala  |  1263 +
 .../spark/ml/recommendation/CollectTopKSuite.scala |   112 +
 .../ml/regression/AFTSurvivalRegressionSuite.scala |   527 +
 .../ml/regression/DecisionTreeRegressorSuite.scala |   284 +
 .../spark/ml/regression/FMRegressorSuite.scala     |   299 +
 .../spark/ml/regression/GBTRegressorSuite.scala    |   421 +
 .../GeneralizedLinearRegressionSuite.scala         |  1811 ++
 .../ml/regression/IsotonicRegressionSuite.scala    |   228 +
 .../ml/regression/LinearRegressionSuite.scala      |  1448 +
 .../ml/regression/RandomForestRegressorSuite.scala |   268 +
 .../ml/source/image/ImageFileFormatSuite.scala     |   148 +
 .../ml/source/libsvm/LibSVMRelationSuite.scala     |   226 +
 .../org/apache/spark/ml/stat/ANOVATestSuite.scala  |   174 +
 .../apache/spark/ml/stat/ChiSquareTestSuite.scala  |   133 +
 .../apache/spark/ml/stat/CorrelationSuite.scala    |    75 +
 .../org/apache/spark/ml/stat/FValueTestSuite.scala |   171 +
 .../spark/ml/stat/KolmogorovSmirnovTestSuite.scala |   142 +
 .../spark/ml/stat/MultiClassSummarizerSuite.scala  |    91 +
 .../org/apache/spark/ml/stat/SummarizerSuite.scala |   687 +
 .../spark/ml/tree/impl/BaggedPointSuite.scala      |   115 +
 .../ml/tree/impl/GradientBoostedTreesSuite.scala   |    86 +
 .../spark/ml/tree/impl/RandomForestSuite.scala     |   799 +
 .../apache/spark/ml/tree/impl/TreePointSuite.scala |    36 +
 .../org/apache/spark/ml/tree/impl/TreeTests.scala  |   312 +
 .../spark/ml/tuning/CrossValidatorSuite.scala      |   557 +
 .../spark/ml/tuning/ParamGridBuilderSuite.scala    |    62 +
 .../ml/tuning/TrainValidationSplitSuite.scala      |   354 +
 .../ml/tuning/ValidatorParamsSuiteHelpers.scala    |    86 +
 .../spark/ml/util/DefaultReadWriteTest.scala       |   276 +
 .../apache/spark/ml/util/IdentifiableSuite.scala   |    41 +
 .../scala/org/apache/spark/ml/util/MLTest.scala    |   389 +
 .../org/apache/spark/ml/util/MLTestSuite.scala     |    55 +
 .../org/apache/spark/ml/util/MLTestingUtils.scala  |   287 +
 .../apache/spark/ml/util/PMMLReadWriteTest.scala   |    52 +
 .../scala/org/apache/spark/ml/util/PMMLUtils.scala |    43 +
 .../org/apache/spark/ml/util/ReadWriteSuite.scala  |   132 +
 .../org/apache/spark/ml/util/StopwatchSuite.scala  |   126 +
 .../org/apache/spark/ml/util/TempDirectory.scala   |    50 +
 .../mllib/api/python/PythonMLLibAPISuite.scala     |   103 +
 .../classification/LogisticRegressionSuite.scala   |   942 +
 .../mllib/classification/NaiveBayesSuite.scala     |   367 +
 .../spark/mllib/classification/SVMSuite.scala      |   247 +
 .../StreamingLogisticRegressionSuite.scala         |   181 +
 .../mllib/clustering/BisectingKMeansSuite.scala    |   202 +
 .../mllib/clustering/DistanceMeasureSuite.scala    |    77 +
 .../mllib/clustering/GaussianMixtureSuite.scala    |   205 +
 .../spark/mllib/clustering/KMeansSuite.scala       |   376 +
 .../apache/spark/mllib/clustering/LDASuite.scala   |   634 +
 .../clustering/PowerIterationClusteringSuite.scala |   196 +
 .../mllib/clustering/StreamingKMeansSuite.scala    |   171 +
 .../mllib/evaluation/AreaUnderCurveSuite.scala     |    46 +
 .../BinaryClassificationMetricsSuite.scala         |   200 +
 .../mllib/evaluation/MulticlassMetricsSuite.scala  |   288 +
 .../mllib/evaluation/MultilabelMetricsSuite.scala  |   103 +
 .../mllib/evaluation/RankingMetricsSuite.scala     |   124 +
 .../mllib/evaluation/RegressionMetricsSuite.scala  |   186 +
 .../spark/mllib/feature/ChiSqSelectorSuite.scala   |   198 +
 .../mllib/feature/ElementwiseProductSuite.scala    |    61 +
 .../spark/mllib/feature/HashingTFSuite.scala       |    64 +
 .../org/apache/spark/mllib/feature/IDFSuite.scala  |   109 +
 .../spark/mllib/feature/NormalizerSuite.scala      |   122 +
 .../org/apache/spark/mllib/feature/PCASuite.scala  |    80 +
 .../spark/mllib/feature/StandardScalerSuite.scala  |   374 +
 .../apache/spark/mllib/feature/Word2VecSuite.scala |   178 +
 .../spark/mllib/fpm/AssociationRulesSuite.scala    |    91 +
 .../org/apache/spark/mllib/fpm/FPGrowthSuite.scala |   346 +
 .../org/apache/spark/mllib/fpm/FPTreeSuite.scala   |   112 +
 .../apache/spark/mllib/fpm/PrefixSpanSuite.scala   |   464 +
 .../org/apache/spark/mllib/linalg/BLASSuite.scala  |   431 +
 .../mllib/linalg/BreezeMatrixConversionSuite.scala |    71 +
 .../mllib/linalg/BreezeVectorConversionSuite.scala |    67 +
 .../apache/spark/mllib/linalg/MatricesSuite.scala  |   661 +
 .../mllib/linalg/UDTSerializationBenchmark.scala   |    76 +
 .../apache/spark/mllib/linalg/VectorsSuite.scala   |   609 +
 .../linalg/distributed/BlockMatrixSuite.scala      |   402 +
 .../linalg/distributed/CoordinateMatrixSuite.scala |   116 +
 .../linalg/distributed/IndexedRowMatrixSuite.scala |   250 +
 .../mllib/linalg/distributed/RowMatrixSuite.scala  |   375 +
 .../mllib/optimization/GradientDescentSuite.scala  |   204 +
 .../spark/mllib/optimization/LBFGSSuite.scala      |   273 +
 .../spark/mllib/optimization/NNLSSuite.scala       |   106 +
 .../BinaryClassificationPMMLModelExportSuite.scala |    85 +
 .../GeneralizedLinearPMMLModelExportSuite.scala    |    84 +
 .../pmml/export/KMeansPMMLModelExportSuite.scala   |    49 +
 .../pmml/export/PMMLModelExportFactorySuite.scala  |    94 +
 .../mllib/random/RandomDataGeneratorSuite.scala    |   153 +
 .../spark/mllib/random/RandomRDDsSuite.scala       |   201 +
 .../spark/mllib/rdd/MLPairRDDFunctionsSuite.scala  |    36 +
 .../apache/spark/mllib/rdd/RDDFunctionsSuite.scala |    51 +
 .../spark/mllib/recommendation/ALSSuite.scala      |   297 +
 .../MatrixFactorizationModelSuite.scala            |   115 +
 .../mllib/regression/IsotonicRegressionSuite.scala |   381 +
 .../spark/mllib/regression/LabeledPointSuite.scala |    73 +
 .../apache/spark/mllib/regression/LassoSuite.scala |   161 +
 .../mllib/regression/LinearRegressionSuite.scala   |   170 +
 .../mllib/regression/RidgeRegressionSuite.scala    |   114 +
 .../StreamingLinearRegressionSuite.scala           |   191 +
 .../apache/spark/mllib/stat/CorrelationSuite.scala |   165 +
 .../spark/mllib/stat/HypothesisTestSuite.scala     |   261 +
 .../spark/mllib/stat/KernelDensitySuite.scala      |    50 +
 .../stat/MultivariateOnlineSummarizerSuite.scala   |   291 +
 .../spark/mllib/stat/StreamingTestSuite.scala      |   244 +
 .../distribution/MultivariateGaussianSuite.scala   |   104 +
 .../spark/mllib/tree/DecisionTreeSuite.scala       |   643 +
 .../spark/mllib/tree/EnsembleTestHelper.scala      |   107 +
 .../mllib/tree/GradientBoostedTreesSuite.scala     |   199 +
 .../apache/spark/mllib/tree/ImpuritySuite.scala    |    81 +
 .../spark/mllib/tree/RandomForestSuite.scala       |   161 +
 .../mllib/util/LocalClusterSparkContext.scala      |    48 +
 .../org/apache/spark/mllib/util/MLUtilsSuite.scala |   389 +
 .../spark/mllib/util/MLlibTestSparkContext.scala   |    83 +
 .../spark/mllib/util/NumericParserSuite.scala      |    47 +
 .../org/apache/spark/mllib/util/TestingUtils.scala |   238 +
 .../spark/mllib/util/TestingUtilsSuite.scala       |   479 +
 pom.xml                                            |  3697 +++
 project/MimaBuild.scala                            |   100 +
 project/MimaExcludes.scala                         |   139 +
 project/SparkBuild.scala                           |  1967 ++
 project/build.properties                           |    17 +
 project/plugins.sbt                                |    50 +
 pyproject.toml                                     |    77 +
 python/.coveragerc                                 |    44 +
 python/.gitignore                                  |     9 +
 python/MANIFEST.in                                 |    31 +
 python/README.md                                   |    33 +
 python/conf_viztracer/daemon_viztracer.py          |    48 +
 python/conf_viztracer/spark-defaults.conf          |    21 +
 python/conf_vscode/README.md                       |    43 +
 python/conf_vscode/sitecustomize.py                |    52 +
 python/docs/Makefile                               |    35 +
 python/docs/make.bat                               |    23 +
 python/docs/make2.bat                              |    51 +
 python/docs/source/_static/css/pyspark.css         |    93 +
 .../_templates/autosummary/accessor_attribute.rst  |     6 +
 .../_templates/autosummary/accessor_method.rst     |     6 +
 .../docs/source/_templates/autosummary/class.rst   |    38 +
 .../_templates/autosummary/class_with_docs.rst     |    81 +
 .../source/_templates/autosummary/plot_class.rst   |    53 +
 python/docs/source/_templates/layout.html          |    30 +
 python/docs/source/_templates/spark_footer.html    |     3 +
 python/docs/source/conf.py                         |   454 +
 python/docs/source/development/contributing.rst    |   335 +
 python/docs/source/development/debugging.rst       |   676 +
 python/docs/source/development/index.rst           |    31 +
 python/docs/source/development/logger.rst          |   169 +
 python/docs/source/development/setting_ide.rst     |    62 +
 python/docs/source/development/testing.rst         |    91 +
 python/docs/source/getting_started/index.rst       |    43 +
 python/docs/source/getting_started/install.rst     |   320 +
 .../getting_started/quickstart_connect.ipynb       |   144 +
 .../source/getting_started/quickstart_df.ipynb     |  1144 +
 .../source/getting_started/quickstart_ps.ipynb     | 14493 +++++++++
 .../source/getting_started/testing_pyspark.ipynb   |   485 +
 python/docs/source/index.rst                       |   202 +
 python/docs/source/migration_guide/index.rst       |    51 +
 .../source/migration_guide/koalas_to_pyspark.rst   |    36 +
 .../source/migration_guide/pyspark_upgrade.rst     |   205 +
 python/docs/source/reference/index.rst             |    43 +
 python/docs/source/reference/pyspark.errors.rst    |    78 +
 python/docs/source/reference/pyspark.logger.rst    |    47 +
 python/docs/source/reference/pyspark.ml.rst        |   389 +
 python/docs/source/reference/pyspark.mllib.rst     |   254 +
 .../source/reference/pyspark.pandas/extensions.rst |    39 +
 .../docs/source/reference/pyspark.pandas/frame.rst |   356 +
 .../reference/pyspark.pandas/general_functions.rst |    66 +
 .../source/reference/pyspark.pandas/groupby.rst    |   110 +
 .../docs/source/reference/pyspark.pandas/index.rst |    42 +
 .../source/reference/pyspark.pandas/indexing.rst   |   408 +
 python/docs/source/reference/pyspark.pandas/io.rst |   127 +
 python/docs/source/reference/pyspark.pandas/ml.rst |    42 +
 .../source/reference/pyspark.pandas/options.rst    |    37 +
 .../source/reference/pyspark.pandas/resampling.rst |    41 +
 .../source/reference/pyspark.pandas/series.rst     |   502 +
 .../source/reference/pyspark.pandas/testing.rst    |    33 +
 .../source/reference/pyspark.pandas/window.rst     |    62 +
 python/docs/source/reference/pyspark.pipelines.rst |    33 +
 python/docs/source/reference/pyspark.resource.rst  |    38 +
 python/docs/source/reference/pyspark.rst           |   298 +
 python/docs/source/reference/pyspark.sql/avro.rst  |    28 +
 .../docs/source/reference/pyspark.sql/catalog.rst  |    54 +
 .../docs/source/reference/pyspark.sql/column.rst   |    64 +
 .../source/reference/pyspark.sql/configuration.rst |    27 +
 .../source/reference/pyspark.sql/core_classes.rst  |    54 +
 .../source/reference/pyspark.sql/data_types.rst    |    55 +
 .../source/reference/pyspark.sql/dataframe.rst     |   191 +
 .../source/reference/pyspark.sql/datasource.rst    |    44 +
 .../source/reference/pyspark.sql/functions.rst     |   722 +
 .../docs/source/reference/pyspark.sql/grouping.rst |    42 +
 python/docs/source/reference/pyspark.sql/index.rst |    47 +
 python/docs/source/reference/pyspark.sql/io.rst    |    70 +
 .../source/reference/pyspark.sql/observation.rst   |    27 +
 .../docs/source/reference/pyspark.sql/protobuf.rst |    28 +
 python/docs/source/reference/pyspark.sql/row.rst   |    27 +
 .../source/reference/pyspark.sql/spark_session.rst |    93 +
 .../reference/pyspark.sql/stateful_processor.rst   |    30 +
 python/docs/source/reference/pyspark.sql/udf.rst   |    32 +
 python/docs/source/reference/pyspark.sql/udtf.rst  |    30 +
 .../source/reference/pyspark.sql/variant_val.rst   |    29 +
 .../docs/source/reference/pyspark.sql/window.rst   |    38 +
 .../source/reference/pyspark.ss/core_classes.rst   |    32 +
 python/docs/source/reference/pyspark.ss/index.rst  |    30 +
 python/docs/source/reference/pyspark.ss/io.rst     |    49 +
 .../reference/pyspark.ss/query_management.rst      |    45 +
 python/docs/source/reference/pyspark.streaming.rst |   130 +
 python/docs/source/reference/pyspark.testing.rst   |    29 +
 python/docs/source/tutorial/arrow_pandas.rst       |    24 +
 python/docs/source/tutorial/index.rst              |    37 +
 .../tutorial/pandas_on_spark/best_practices.rst    |   405 +
 .../docs/source/tutorial/pandas_on_spark/faq.rst   |    57 +
 .../tutorial/pandas_on_spark/from_to_dbms.rst      |   125 +
 .../docs/source/tutorial/pandas_on_spark/index.rst |    34 +
 .../source/tutorial/pandas_on_spark/options.rst    |   343 +
 .../tutorial/pandas_on_spark/pandas_pyspark.rst    |   136 +
 .../tutorial/pandas_on_spark/transform_apply.rst   |   137 +
 .../source/tutorial/pandas_on_spark/typehints.rst  |   264 +
 .../docs/source/tutorial/pandas_on_spark/types.rst |   242 +
 python/docs/source/tutorial/python_packaging.rst   |   253 +
 python/docs/source/tutorial/sql/arrow_pandas.rst   |   461 +
 .../docs/source/tutorial/sql/arrow_python_udtf.rst |   655 +
 python/docs/source/tutorial/sql/index.rst          |    31 +
 .../source/tutorial/sql/python_data_source.rst     |   568 +
 python/docs/source/tutorial/sql/python_udtf.rst    |   558 +
 .../docs/source/tutorial/sql/type_conversions.rst  |   254 +
 .../source/user_guide/ansi_migration_guide.ipynb   |   272 +
 .../user_guide/assets/pyspark-udf-profile.png      |   Bin 0 -> 281384 bytes
 .../source/user_guide/assets/pyspark-ui-print.png  |   Bin 0 -> 206336 bytes
 .../user_guide/assets/pyspark-ui-sql-broadcast.png |   Bin 0 -> 84476 bytes
 .../source/user_guide/assets/pyspark-ui-sql.png    |   Bin 0 -> 365322 bytes
 python/docs/source/user_guide/bugbusting.ipynb     |  1307 +
 python/docs/source/user_guide/dataframes.ipynb     |   800 +
 python/docs/source/user_guide/dataprep.ipynb       |  1420 +
 python/docs/source/user_guide/index.rst            |    38 +
 python/docs/source/user_guide/loadandbehold.ipynb  |   488 +
 python/docs/source/user_guide/sql.ipynb            |  1192 +
 python/docs/source/user_guide/touroftypes.ipynb    |  1190 +
 python/docs/source/user_guide/udfandudtf.ipynb     |  1040 +
 python/lib/PY4J_LICENSE.txt                        |    27 +
 python/lib/py4j-0.10.9.9-src.zip                   |   Bin 0 -> 43108 bytes
 python/mypy.ini                                    |   205 +
 python/packaging/classic/setup.cfg                 |    22 +
 python/packaging/classic/setup.py                  |   421 +
 python/packaging/client/setup.cfg                  |    22 +
 python/packaging/client/setup.py                   |   236 +
 .../packaging/connect/pyspark_connect/__init__.py  |    26 +
 python/packaging/connect/setup.cfg                 |    22 +
 python/packaging/connect/setup.py                  |   148 +
 python/pyspark/__init__.py                         |   156 +
 python/pyspark/_globals.py                         |    71 +
 python/pyspark/_typing.pyi                         |    43 +
 python/pyspark/accumulators.py                     |   412 +
 python/pyspark/cloudpickle/__init__.py             |    18 +
 python/pyspark/cloudpickle/cloudpickle.py          |  1552 +
 python/pyspark/cloudpickle/cloudpickle_fast.py     |    14 +
 python/pyspark/conf.py                             |   282 +
 python/pyspark/core/__init__.py                    |    16 +
 python/pyspark/core/broadcast.py                   |   382 +
 python/pyspark/core/context.py                     |  2671 ++
 python/pyspark/core/files.py                       |   165 +
 python/pyspark/core/rdd.py                         |  5411 ++++
 python/pyspark/core/status.py                      |   112 +
 python/pyspark/daemon.py                           |   287 +
 python/pyspark/errors/__init__.py                  |    90 +
 python/pyspark/errors/error-conditions.json        |  1496 +
 python/pyspark/errors/error_classes.py             |    31 +
 python/pyspark/errors/exceptions/__init__.py       |    33 +
 python/pyspark/errors/exceptions/base.py           |   494 +
 python/pyspark/errors/exceptions/captured.py       |   461 +
 python/pyspark/errors/exceptions/connect.py        |   545 +
 python/pyspark/errors/exceptions/tblib.py          |   342 +
 python/pyspark/errors/tests/__init__.py            |    16 +
 python/pyspark/errors/tests/connect/__init__.py    |    16 +
 .../errors/tests/connect/test_parity_traceback.py  |    28 +
 .../errors/tests/test_connect_errors_conversion.py |   412 +
 python/pyspark/errors/tests/test_errors.py         |   125 +
 python/pyspark/errors/tests/test_traceback.py      |   277 +
 python/pyspark/errors/utils.py                     |   381 +
 python/pyspark/errors_doc_gen.py                   |    62 +
 python/pyspark/find_spark_home.py                  |    95 +
 python/pyspark/install.py                          |   203 +
 python/pyspark/instrumentation_utils.py            |   187 +
 python/pyspark/java_gateway.py                     |   196 +
 python/pyspark/join.py                             |   119 +
 python/pyspark/logger/__init__.py                  |    23 +
 python/pyspark/logger/logger.py                    |   330 +
 python/pyspark/logger/tests/__init__.py            |    16 +
 python/pyspark/logger/tests/connect/__init__.py    |    16 +
 .../logger/tests/connect/test_parity_logger.py     |    29 +
 python/pyspark/logger/tests/test_logger.py         |   219 +
 python/pyspark/logger/worker_io.py                 |   297 +
 python/pyspark/loose_version.py                    |    75 +
 python/pyspark/ml/__init__.py                      |    71 +
 python/pyspark/ml/_typing.pyi                      |    86 +
 python/pyspark/ml/base.py                          |   412 +
 python/pyspark/ml/classification.py                |  4364 +++
 python/pyspark/ml/clustering.py                    |  2206 ++
 python/pyspark/ml/common.py                        |   163 +
 python/pyspark/ml/connect/__init__.py              |    42 +
 python/pyspark/ml/connect/base.py                  |   360 +
 python/pyspark/ml/connect/classification.py        |   398 +
 python/pyspark/ml/connect/evaluation.py            |   300 +
 python/pyspark/ml/connect/feature.py               |   428 +
 python/pyspark/ml/connect/functions.py             |    92 +
 python/pyspark/ml/connect/io_utils.py              |   342 +
 python/pyspark/ml/connect/pipeline.py              |   273 +
 python/pyspark/ml/connect/proto.py                 |    89 +
 python/pyspark/ml/connect/readwrite.py             |   370 +
 python/pyspark/ml/connect/serialize.py             |   227 +
 python/pyspark/ml/connect/summarizer.py            |   121 +
 python/pyspark/ml/connect/tuning.py                |   586 +
 python/pyspark/ml/connect/util.py                  |   192 +
 python/pyspark/ml/deepspeed/__init__.py            |    16 +
 .../pyspark/ml/deepspeed/deepspeed_distributor.py  |   164 +
 python/pyspark/ml/deepspeed/tests/__init__.py      |    16 +
 .../deepspeed/tests/test_deepspeed_distributor.py  |   295 +
 python/pyspark/ml/dl_util.py                       |   150 +
 python/pyspark/ml/evaluation.py                    |  1195 +
 python/pyspark/ml/feature.py                       |  7868 +++++
 python/pyspark/ml/fpm.py                           |   560 +
 python/pyspark/ml/functions.py                     |   872 +
 python/pyspark/ml/image.py                         |   249 +
 python/pyspark/ml/linalg/__init__.py               |  1384 +
 python/pyspark/ml/model_cache.py                   |    55 +
 python/pyspark/ml/param/__init__.py                |   604 +
 python/pyspark/ml/param/_shared_params_code_gen.py |   375 +
 python/pyspark/ml/param/shared.py                  |   900 +
 python/pyspark/ml/pipeline.py                      |   457 +
 python/pyspark/ml/recommendation.py                |   754 +
 python/pyspark/ml/regression.py                    |  3342 +++
 python/pyspark/ml/stat.py                          |   585 +
 python/pyspark/ml/tests/__init__.py                |    16 +
 python/pyspark/ml/tests/connect/__init__.py        |    16 +
 .../pyspark/ml/tests/connect/test_connect_cache.py |   124 +
 .../tests/connect/test_connect_classification.py   |    50 +
 .../ml/tests/connect/test_connect_evaluation.py    |    37 +
 .../ml/tests/connect/test_connect_feature.py       |    41 +
 .../ml/tests/connect/test_connect_function.py      |    72 +
 .../tests/connect/test_connect_model_offloading.py |   312 +
 .../ml/tests/connect/test_connect_pipeline.py      |    51 +
 .../ml/tests/connect/test_connect_summarizer.py    |    37 +
 .../ml/tests/connect/test_connect_tuning.py        |    51 +
 .../connect/test_legacy_mode_classification.py     |   253 +
 .../tests/connect/test_legacy_mode_evaluation.py   |   190 +
 .../ml/tests/connect/test_legacy_mode_feature.py   |   200 +
 .../ml/tests/connect/test_legacy_mode_pipeline.py  |   196 +
 .../tests/connect/test_legacy_mode_summarizer.py   |    79 +
 .../ml/tests/connect/test_legacy_mode_tuning.py    |   314 +
 python/pyspark/ml/tests/connect/test_parity_als.py |    30 +
 .../ml/tests/connect/test_parity_classification.py |    30 +
 .../ml/tests/connect/test_parity_clustering.py     |    30 +
 .../ml/tests/connect/test_parity_evaluation.py     |    30 +
 .../ml/tests/connect/test_parity_feature.py        |    30 +
 python/pyspark/ml/tests/connect/test_parity_fpm.py |    30 +
 .../ml/tests/connect/test_parity_functions.py      |    48 +
 python/pyspark/ml/tests/connect/test_parity_ovr.py |    30 +
 .../ml/tests/connect/test_parity_pipeline.py       |    30 +
 .../ml/tests/connect/test_parity_regression.py     |    30 +
 .../pyspark/ml/tests/connect/test_parity_stat.py   |    30 +
 .../tests/connect/test_parity_torch_data_loader.py |    43 +
 .../tests/connect/test_parity_torch_distributor.py |   161 +
 .../pyspark/ml/tests/connect/test_parity_tuning.py |    30 +
 python/pyspark/ml/tests/test_algorithms.py         |   384 +
 python/pyspark/ml/tests/test_als.py                |   131 +
 python/pyspark/ml/tests/test_base.py               |    90 +
 python/pyspark/ml/tests/test_classification.py     |  1019 +
 python/pyspark/ml/tests/test_clustering.py         |   519 +
 python/pyspark/ml/tests/test_dl_util.py            |   180 +
 python/pyspark/ml/tests/test_evaluation.py         |   384 +
 python/pyspark/ml/tests/test_feature.py            |  1783 ++
 python/pyspark/ml/tests/test_fpm.py                |   109 +
 python/pyspark/ml/tests/test_functions.py          |   578 +
 python/pyspark/ml/tests/test_image.py              |    76 +
 python/pyspark/ml/tests/test_linalg.py             |   406 +
 python/pyspark/ml/tests/test_model_cache.py        |    48 +
 python/pyspark/ml/tests/test_ovr.py                |   125 +
 python/pyspark/ml/tests/test_param.py              |   434 +
 python/pyspark/ml/tests/test_persistence.py        |   516 +
 python/pyspark/ml/tests/test_pipeline.py           |   306 +
 python/pyspark/ml/tests/test_regression.py         |   698 +
 python/pyspark/ml/tests/test_stat.py               |   215 +
 python/pyspark/ml/tests/test_training_summary.py   |   396 +
 python/pyspark/ml/tests/test_tuning.py             |   284 +
 python/pyspark/ml/tests/test_util.py               |    78 +
 python/pyspark/ml/tests/test_wrapper.py            |   131 +
 python/pyspark/ml/tests/tuning/__init__.py         |    16 +
 python/pyspark/ml/tests/tuning/test_cv_io_basic.py |   144 +
 .../pyspark/ml/tests/tuning/test_cv_io_nested.py   |    90 +
 .../pyspark/ml/tests/tuning/test_cv_io_pipeline.py |   144 +
 python/pyspark/ml/tests/tuning/test_tuning.py      |   413 +
 .../pyspark/ml/tests/tuning/test_tvs_io_basic.py   |   130 +
 .../pyspark/ml/tests/tuning/test_tvs_io_nested.py  |    89 +
 .../ml/tests/tuning/test_tvs_io_pipeline.py        |   135 +
 .../ml/tests/typing/test_classification.yml        |    38 +
 .../pyspark/ml/tests/typing/test_clustering.yaml   |    33 +
 python/pyspark/ml/tests/typing/test_evaluation.yml |    28 +
 python/pyspark/ml/tests/typing/test_feature.yml    |    55 +
 python/pyspark/ml/tests/typing/test_param.yml      |    30 +
 python/pyspark/ml/tests/typing/test_readable.yml   |    28 +
 python/pyspark/ml/tests/typing/test_regression.yml |    53 +
 python/pyspark/ml/torch/__init__.py                |    16 +
 python/pyspark/ml/torch/data.py                    |   102 +
 python/pyspark/ml/torch/distributor.py             |  1105 +
 python/pyspark/ml/torch/log_communication.py       |   198 +
 python/pyspark/ml/torch/tests/__init__.py          |    16 +
 python/pyspark/ml/torch/tests/test_data_loader.py  |   127 +
 python/pyspark/ml/torch/tests/test_distributor.py  |   578 +
 .../ml/torch/tests/test_log_communication.py       |   166 +
 .../pyspark/ml/torch/torch_run_process_wrapper.py  |    83 +
 python/pyspark/ml/tree.py                          |   434 +
 python/pyspark/ml/tuning.py                        |  1764 ++
 python/pyspark/ml/util.py                          |  1253 +
 python/pyspark/ml/wrapper.py                       |   505 +
 python/pyspark/mllib/__init__.py                   |    44 +
 python/pyspark/mllib/_typing.pyi                   |    37 +
 python/pyspark/mllib/classification.py             |   991 +
 python/pyspark/mllib/clustering.py                 |  1318 +
 python/pyspark/mllib/common.py                     |   174 +
 python/pyspark/mllib/evaluation.py                 |   690 +
 python/pyspark/mllib/feature.py                    |  1083 +
 python/pyspark/mllib/fpm.py                        |   233 +
 python/pyspark/mllib/linalg/__init__.py            |  1654 ++
 python/pyspark/mllib/linalg/distributed.py         |  1662 ++
 python/pyspark/mllib/random.py                     |   698 +
 python/pyspark/mllib/recommendation.py             |   389 +
 python/pyspark/mllib/regression.py                 |  1067 +
 python/pyspark/mllib/stat/KernelDensity.py         |    59 +
 python/pyspark/mllib/stat/__init__.py              |    34 +
 python/pyspark/mllib/stat/_statistics.py           |   410 +
 python/pyspark/mllib/stat/distribution.py          |    39 +
 python/pyspark/mllib/stat/test.py                  |    86 +
 python/pyspark/mllib/tests/__init__.py             |    16 +
 python/pyspark/mllib/tests/test_algorithms.py      |   347 +
 python/pyspark/mllib/tests/test_feature.py         |   184 +
 python/pyspark/mllib/tests/test_linalg.py          |   672 +
 python/pyspark/mllib/tests/test_stat.py            |   200 +
 .../mllib/tests/test_streaming_algorithms.py       |   470 +
 python/pyspark/mllib/tests/test_util.py            |   101 +
 python/pyspark/mllib/tree.py                       |   888 +
 python/pyspark/mllib/util.py                       |   655 +
 python/pyspark/pandas/__init__.py                  |   144 +
 python/pyspark/pandas/_typing.py                   |    52 +
 python/pyspark/pandas/accessors.py                 |   988 +
 python/pyspark/pandas/base.py                      |  1759 ++
 python/pyspark/pandas/categorical.py               |   798 +
 python/pyspark/pandas/config.py                    |   585 +
 python/pyspark/pandas/correlation.py               |   283 +
 python/pyspark/pandas/data_type_ops/__init__.py    |    16 +
 python/pyspark/pandas/data_type_ops/base.py        |   550 +
 python/pyspark/pandas/data_type_ops/binary_ops.py  |    98 +
 python/pyspark/pandas/data_type_ops/boolean_ops.py |   466 +
 .../pandas/data_type_ops/categorical_ops.py        |   141 +
 python/pyspark/pandas/data_type_ops/complex_ops.py |   145 +
 python/pyspark/pandas/data_type_ops/date_ops.py    |   121 +
 .../pyspark/pandas/data_type_ops/datetime_ops.py   |   177 +
 python/pyspark/pandas/data_type_ops/null_ops.py    |    83 +
 python/pyspark/pandas/data_type_ops/num_ops.py     |   773 +
 python/pyspark/pandas/data_type_ops/string_ops.py  |   154 +
 .../pyspark/pandas/data_type_ops/timedelta_ops.py  |   109 +
 python/pyspark/pandas/data_type_ops/udt_ops.py     |    29 +
 python/pyspark/pandas/datetimes.py                 |   925 +
 python/pyspark/pandas/exceptions.py                |   140 +
 python/pyspark/pandas/extensions.py                |   388 +
 python/pyspark/pandas/frame.py                     | 13967 +++++++++
 python/pyspark/pandas/generic.py                   |  3649 +++
 python/pyspark/pandas/groupby.py                   |  4647 +++
 python/pyspark/pandas/indexes/__init__.py          |    20 +
 python/pyspark/pandas/indexes/base.py              |  2679 ++
 python/pyspark/pandas/indexes/category.py          |   668 +
 python/pyspark/pandas/indexes/datetimes.py         |   882 +
 python/pyspark/pandas/indexes/multi.py             |  1292 +
 python/pyspark/pandas/indexes/timedelta.py         |   205 +
 python/pyspark/pandas/indexing.py                  |  1873 ++
 python/pyspark/pandas/internal.py                  |  1632 +
 python/pyspark/pandas/missing/__init__.py          |    48 +
 python/pyspark/pandas/missing/common.py            |    76 +
 python/pyspark/pandas/missing/frame.py             |    62 +
 python/pyspark/pandas/missing/general_functions.py |    43 +
 python/pyspark/pandas/missing/groupby.py           |    92 +
 python/pyspark/pandas/missing/indexes.py           |   184 +
 python/pyspark/pandas/missing/resample.py          |   101 +
 python/pyspark/pandas/missing/scalars.py           |    29 +
 python/pyspark/pandas/missing/series.py            |    68 +
 python/pyspark/pandas/missing/window.py            |   168 +
 python/pyspark/pandas/mlflow.py                    |   236 +
 python/pyspark/pandas/namespace.py                 |  3929 +++
 python/pyspark/pandas/numpy_compat.py              |   259 +
 python/pyspark/pandas/plot/__init__.py             |    17 +
 python/pyspark/pandas/plot/core.py                 |  1223 +
 python/pyspark/pandas/plot/matplotlib.py           |   972 +
 python/pyspark/pandas/plot/plotly.py               |   255 +
 python/pyspark/pandas/resample.py                  |   794 +
 python/pyspark/pandas/series.py                    |  7448 +++++
 python/pyspark/pandas/spark/__init__.py            |    16 +
 python/pyspark/pandas/spark/accessors.py           |  1274 +
 python/pyspark/pandas/spark/utils.py               |   202 +
 python/pyspark/pandas/sql_formatter.py             |   334 +
 python/pyspark/pandas/sql_processor.py             |   389 +
 python/pyspark/pandas/strings.py                   |  2394 ++
 python/pyspark/pandas/supported_api_gen.py         |   441 +
 python/pyspark/pandas/testing.py                   |   328 +
 python/pyspark/pandas/tests/__init__.py            |    16 +
 .../pyspark/pandas/tests/computation/__init__.py   |    16 +
 .../pandas/tests/computation/test_any_all.py       |   264 +
 .../pandas/tests/computation/test_apply_func.py    |   570 +
 .../pandas/tests/computation/test_binary_ops.py    |   316 +
 .../pandas/tests/computation/test_combine.py       |   673 +
 .../pandas/tests/computation/test_compute.py       |   445 +
 .../pyspark/pandas/tests/computation/test_corr.py  |   220 +
 .../pandas/tests/computation/test_corrwith.py      |    77 +
 .../pyspark/pandas/tests/computation/test_cov.py   |   113 +
 .../pandas/tests/computation/test_cumulative.py    |   123 +
 .../pandas/tests/computation/test_describe.py      |   248 +
 .../pyspark/pandas/tests/computation/test_eval.py  |    73 +
 .../pyspark/pandas/tests/computation/test_melt.py  |   179 +
 .../pandas/tests/computation/test_missing_data.py  |   477 +
 .../pyspark/pandas/tests/computation/test_pivot.py |   202 +
 .../pandas/tests/computation/test_pivot_table.py   |    86 +
 .../tests/computation/test_pivot_table_adv.py      |    86 +
 .../computation/test_pivot_table_multi_idx.py      |    84 +
 .../computation/test_pivot_table_multi_idx_adv.py  |    86 +
 .../pyspark/pandas/tests/computation/test_stats.py |   320 +
 python/pyspark/pandas/tests/connect/__init__.py    |    16 +
 .../pandas/tests/connect/computation/__init__.py   |    16 +
 .../connect/computation/test_parity_any_all.py     |    34 +
 .../connect/computation/test_parity_apply_func.py  |    34 +
 .../connect/computation/test_parity_binary_ops.py  |    34 +
 .../connect/computation/test_parity_combine.py     |    34 +
 .../connect/computation/test_parity_compute.py     |    34 +
 .../tests/connect/computation/test_parity_corr.py  |    34 +
 .../connect/computation/test_parity_corrwith.py    |    34 +
 .../tests/connect/computation/test_parity_cov.py   |    34 +
 .../connect/computation/test_parity_cumulative.py  |    34 +
 .../connect/computation/test_parity_describe.py    |    30 +
 .../tests/connect/computation/test_parity_eval.py  |    34 +
 .../tests/connect/computation/test_parity_melt.py  |    34 +
 .../computation/test_parity_missing_data.py        |    34 +
 .../tests/connect/computation/test_parity_pivot.py |    34 +
 .../connect/computation/test_parity_pivot_table.py |    34 +
 .../computation/test_parity_pivot_table_adv.py     |    34 +
 .../test_parity_pivot_table_multi_idx.py           |    34 +
 .../test_parity_pivot_table_multi_idx_adv.py       |    36 +
 .../tests/connect/computation/test_parity_stats.py |    30 +
 .../pandas/tests/connect/data_type_ops/__init__.py |    16 +
 .../connect/data_type_ops/test_parity_as_type.py   |    36 +
 .../connect/data_type_ops/test_parity_base.py      |    32 +
 .../data_type_ops/test_parity_binary_ops.py        |    36 +
 .../data_type_ops/test_parity_boolean_ops.py       |    36 +
 .../data_type_ops/test_parity_categorical_ops.py   |    36 +
 .../data_type_ops/test_parity_complex_ops.py       |    36 +
 .../connect/data_type_ops/test_parity_date_ops.py  |    36 +
 .../data_type_ops/test_parity_datetime_ops.py      |    36 +
 .../connect/data_type_ops/test_parity_null_ops.py  |    36 +
 .../data_type_ops/test_parity_num_arithmetic.py    |    36 +
 .../connect/data_type_ops/test_parity_num_mod.py   |    36 +
 .../data_type_ops/test_parity_num_mul_div.py       |    36 +
 .../connect/data_type_ops/test_parity_num_ops.py   |    36 +
 .../connect/data_type_ops/test_parity_num_pow.py   |    36 +
 .../data_type_ops/test_parity_num_reverse.py       |    36 +
 .../data_type_ops/test_parity_string_ops.py        |    36 +
 .../data_type_ops/test_parity_timedelta_ops.py     |    36 +
 .../connect/data_type_ops/test_parity_udt_ops.py   |    36 +
 .../tests/connect/diff_frames_ops/__init__.py      |    16 +
 .../connect/diff_frames_ops/test_parity_align.py   |    32 +
 .../diff_frames_ops/test_parity_arithmetic.py      |    34 +
 .../test_parity_arithmetic_chain.py                |    34 +
 .../test_parity_arithmetic_chain_ext.py            |    34 +
 .../test_parity_arithmetic_chain_ext_float.py      |    36 +
 .../diff_frames_ops/test_parity_arithmetic_ext.py  |    34 +
 .../test_parity_arithmetic_ext_float.py            |    34 +
 .../diff_frames_ops/test_parity_assign_frame.py    |    34 +
 .../diff_frames_ops/test_parity_assign_series.py   |    34 +
 .../connect/diff_frames_ops/test_parity_basic.py   |    34 +
 .../diff_frames_ops/test_parity_basic_slow.py      |    32 +
 .../connect/diff_frames_ops/test_parity_bitwise.py |    34 +
 .../diff_frames_ops/test_parity_combine_first.py   |    34 +
 .../diff_frames_ops/test_parity_compare_series.py  |    34 +
 .../diff_frames_ops/test_parity_concat_inner.py    |    34 +
 .../diff_frames_ops/test_parity_concat_outer.py    |    34 +
 .../diff_frames_ops/test_parity_corrwith.py        |    34 +
 .../connect/diff_frames_ops/test_parity_cov.py     |    35 +
 .../diff_frames_ops/test_parity_dot_frame.py       |    32 +
 .../diff_frames_ops/test_parity_dot_series.py      |    32 +
 .../connect/diff_frames_ops/test_parity_error.py   |    32 +
 .../connect/diff_frames_ops/test_parity_groupby.py |    34 +
 .../test_parity_groupby_aggregate.py               |    34 +
 .../diff_frames_ops/test_parity_groupby_apply.py   |    34 +
 .../test_parity_groupby_cumulative.py              |    34 +
 .../diff_frames_ops/test_parity_groupby_diff.py    |    34 +
 .../test_parity_groupby_diff_len.py                |    34 +
 .../test_parity_groupby_expanding.py               |    34 +
 .../test_parity_groupby_expanding_adv.py           |    34 +
 .../test_parity_groupby_expanding_count.py         |    36 +
 .../diff_frames_ops/test_parity_groupby_fillna.py  |    34 +
 .../diff_frames_ops/test_parity_groupby_filter.py  |    34 +
 .../diff_frames_ops/test_parity_groupby_rolling.py |    34 +
 .../test_parity_groupby_rolling_adv.py             |    34 +
 .../test_parity_groupby_rolling_count.py           |    34 +
 .../diff_frames_ops/test_parity_groupby_shift.py   |    34 +
 .../test_parity_groupby_split_apply_combine.py     |    34 +
 .../test_parity_groupby_transform.py               |    34 +
 .../connect/diff_frames_ops/test_parity_index.py   |    32 +
 .../connect/diff_frames_ops/test_parity_series.py  |    32 +
 .../diff_frames_ops/test_parity_setitem_frame.py   |    32 +
 .../diff_frames_ops/test_parity_setitem_series.py  |    32 +
 .../pyspark/pandas/tests/connect/frame/__init__.py |    16 +
 .../tests/connect/frame/test_parity_asfreq.py      |    34 +
 .../pandas/tests/connect/frame/test_parity_asof.py |    34 +
 .../tests/connect/frame/test_parity_attrs.py       |    34 +
 .../pandas/tests/connect/frame/test_parity_axis.py |    34 +
 .../tests/connect/frame/test_parity_constructor.py |    34 +
 .../tests/connect/frame/test_parity_conversion.py  |    34 +
 .../tests/connect/frame/test_parity_interpolate.py |    34 +
 .../connect/frame/test_parity_interpolate_error.py |    34 +
 .../tests/connect/frame/test_parity_reindexing.py  |    34 +
 .../tests/connect/frame/test_parity_reshaping.py   |    34 +
 .../tests/connect/frame/test_parity_spark.py       |    34 +
 .../pandas/tests/connect/frame/test_parity_take.py |    37 +
 .../tests/connect/frame/test_parity_take_adv.py    |    37 +
 .../tests/connect/frame/test_parity_time_series.py |    34 +
 .../tests/connect/frame/test_parity_truncate.py    |    34 +
 .../pandas/tests/connect/groupby/__init__.py       |    16 +
 .../tests/connect/groupby/test_parity_aggregate.py |    34 +
 .../connect/groupby/test_parity_apply_func.py      |    37 +
 .../tests/connect/groupby/test_parity_corr.py      |    34 +
 .../connect/groupby/test_parity_cumulative.py      |    34 +
 .../tests/connect/groupby/test_parity_describe.py  |    34 +
 .../tests/connect/groupby/test_parity_groupby.py   |    35 +
 .../tests/connect/groupby/test_parity_grouping.py  |    35 +
 .../tests/connect/groupby/test_parity_head_tail.py |    34 +
 .../tests/connect/groupby/test_parity_index.py     |    34 +
 .../tests/connect/groupby/test_parity_missing.py   |    35 +
 .../connect/groupby/test_parity_missing_data.py    |    34 +
 .../groupby/test_parity_nlargest_nsmallest.py      |    35 +
 .../tests/connect/groupby/test_parity_raises.py    |    35 +
 .../tests/connect/groupby/test_parity_rank.py      |    35 +
 .../tests/connect/groupby/test_parity_size.py      |    35 +
 .../connect/groupby/test_parity_split_apply.py     |    34 +
 .../groupby/test_parity_split_apply_count.py       |    34 +
 .../groupby/test_parity_split_apply_first.py       |    34 +
 .../groupby/test_parity_split_apply_last.py        |    34 +
 .../groupby/test_parity_split_apply_min_max.py     |    34 +
 .../groupby/test_parity_split_apply_skew.py        |    34 +
 .../connect/groupby/test_parity_split_apply_std.py |    34 +
 .../connect/groupby/test_parity_split_apply_var.py |    34 +
 .../tests/connect/groupby/test_parity_stat.py      |    34 +
 .../tests/connect/groupby/test_parity_stat_adv.py  |    34 +
 .../tests/connect/groupby/test_parity_stat_ddof.py |    34 +
 .../tests/connect/groupby/test_parity_stat_func.py |    34 +
 .../tests/connect/groupby/test_parity_stat_prod.py |    34 +
 .../connect/groupby/test_parity_value_counts.py    |    35 +
 .../pandas/tests/connect/indexes/__init__.py       |    16 +
 .../tests/connect/indexes/test_parity_align.py     |    34 +
 .../tests/connect/indexes/test_parity_append.py    |    34 +
 .../tests/connect/indexes/test_parity_asof.py      |    34 +
 .../tests/connect/indexes/test_parity_astype.py    |    34 +
 .../tests/connect/indexes/test_parity_basic.py     |    34 +
 .../tests/connect/indexes/test_parity_category.py  |    32 +
 .../connect/indexes/test_parity_conversion.py      |    34 +
 .../tests/connect/indexes/test_parity_datetime.py  |    34 +
 .../connect/indexes/test_parity_datetime_at.py     |    34 +
 .../indexes/test_parity_datetime_between.py        |    34 +
 .../connect/indexes/test_parity_datetime_ceil.py   |    34 +
 .../connect/indexes/test_parity_datetime_floor.py  |    34 +
 .../connect/indexes/test_parity_datetime_iso.py    |    34 +
 .../connect/indexes/test_parity_datetime_map.py    |    34 +
 .../indexes/test_parity_datetime_property.py       |    34 +
 .../connect/indexes/test_parity_datetime_round.py  |    34 +
 .../tests/connect/indexes/test_parity_default.py   |    38 +
 .../tests/connect/indexes/test_parity_delete.py    |    34 +
 .../tests/connect/indexes/test_parity_diff.py      |    34 +
 .../tests/connect/indexes/test_parity_drop.py      |    34 +
 .../tests/connect/indexes/test_parity_getattr.py   |    34 +
 .../tests/connect/indexes/test_parity_indexing.py  |    34 +
 .../connect/indexes/test_parity_indexing_adv.py    |    36 +
 .../connect/indexes/test_parity_indexing_basic.py  |    34 +
 .../connect/indexes/test_parity_indexing_iloc.py   |    34 +
 .../connect/indexes/test_parity_indexing_loc.py    |    34 +
 .../connect/indexes/test_parity_indexing_loc_2d.py |    34 +
 .../indexes/test_parity_indexing_loc_multi_idx.py  |    34 +
 .../tests/connect/indexes/test_parity_insert.py    |    34 +
 .../connect/indexes/test_parity_intersection.py    |    34 +
 .../tests/connect/indexes/test_parity_level.py     |    34 +
 .../tests/connect/indexes/test_parity_map.py       |    34 +
 .../tests/connect/indexes/test_parity_missing.py   |    34 +
 .../tests/connect/indexes/test_parity_monotonic.py |    34 +
 .../tests/connect/indexes/test_parity_name.py      |    34 +
 .../tests/connect/indexes/test_parity_reindex.py   |    34 +
 .../tests/connect/indexes/test_parity_rename.py    |    34 +
 .../tests/connect/indexes/test_parity_repeat.py    |    34 +
 .../connect/indexes/test_parity_reset_index.py     |    34 +
 .../tests/connect/indexes/test_parity_sort.py      |    34 +
 .../tests/connect/indexes/test_parity_stat.py      |    34 +
 .../connect/indexes/test_parity_symmetric_diff.py  |    34 +
 .../tests/connect/indexes/test_parity_take.py      |    34 +
 .../tests/connect/indexes/test_parity_timedelta.py |    32 +
 .../tests/connect/indexes/test_parity_union.py     |    34 +
 .../tests/connect/indexes/test_parity_unique.py    |    34 +
 python/pyspark/pandas/tests/connect/io/__init__.py |    16 +
 .../pandas/tests/connect/io/test_parity_csv.py     |    35 +
 .../connect/io/test_parity_dataframe_conversion.py |    35 +
 .../connect/io/test_parity_dataframe_spark_io.py   |    35 +
 .../pandas/tests/connect/io/test_parity_feather.py |    35 +
 .../pandas/tests/connect/io/test_parity_io.py      |    34 +
 .../connect/io/test_parity_series_conversion.py    |    34 +
 .../pandas/tests/connect/io/test_parity_stata.py   |    35 +
 .../pyspark/pandas/tests/connect/plot/__init__.py  |    16 +
 .../tests/connect/plot/test_parity_frame_plot.py   |    32 +
 .../plot/test_parity_frame_plot_matplotlib.py      |    32 +
 .../connect/plot/test_parity_frame_plot_plotly.py  |    32 +
 .../tests/connect/plot/test_parity_series_plot.py  |    30 +
 .../plot/test_parity_series_plot_matplotlib.py     |    32 +
 .../connect/plot/test_parity_series_plot_plotly.py |    32 +
 .../pandas/tests/connect/resample/__init__.py      |    16 +
 .../tests/connect/resample/test_parity_error.py    |    30 +
 .../tests/connect/resample/test_parity_frame.py    |    32 +
 .../tests/connect/resample/test_parity_missing.py  |    32 +
 .../tests/connect/resample/test_parity_on.py       |    30 +
 .../tests/connect/resample/test_parity_series.py   |    32 +
 .../tests/connect/resample/test_parity_timezone.py |    32 +
 .../pandas/tests/connect/reshape/__init__.py       |    16 +
 .../connect/reshape/test_parity_get_dummies.py     |    34 +
 .../reshape/test_parity_get_dummies_kwargs.py      |    34 +
 .../reshape/test_parity_get_dummies_multiindex.py  |    34 +
 .../reshape/test_parity_get_dummies_object.py      |    34 +
 .../reshape/test_parity_get_dummies_prefix.py      |    34 +
 .../connect/reshape/test_parity_merge_asof.py      |    34 +
 .../pandas/tests/connect/series/__init__.py        |    16 +
 .../tests/connect/series/test_parity_all_any.py    |    34 +
 .../tests/connect/series/test_parity_arg_ops.py    |    34 +
 .../tests/connect/series/test_parity_as_of.py      |    34 +
 .../tests/connect/series/test_parity_as_type.py    |    34 +
 .../tests/connect/series/test_parity_compute.py    |    34 +
 .../tests/connect/series/test_parity_conversion.py |    34 +
 .../tests/connect/series/test_parity_cumulative.py |    34 +
 .../tests/connect/series/test_parity_datetime.py   |    34 +
 .../tests/connect/series/test_parity_index.py      |    34 +
 .../connect/series/test_parity_interpolate.py      |    34 +
 .../connect/series/test_parity_missing_data.py     |    34 +
 .../tests/connect/series/test_parity_series.py     |    34 +
 .../tests/connect/series/test_parity_sort.py       |    34 +
 .../tests/connect/series/test_parity_stat.py       |    34 +
 .../connect/series/test_parity_string_ops_adv.py   |    34 +
 .../connect/series/test_parity_string_ops_basic.py |    34 +
 .../tests/connect/test_parity_arrow_interface.py   |    32 +
 .../tests/connect/test_parity_categorical.py       |    35 +
 .../pandas/tests/connect/test_parity_config.py     |    30 +
 .../pandas/tests/connect/test_parity_extension.py  |    34 +
 .../tests/connect/test_parity_frame_spark.py       |    47 +
 .../tests/connect/test_parity_generic_functions.py |    32 +
 .../tests/connect/test_parity_indexops_spark.py    |    32 +
 .../pandas/tests/connect/test_parity_internal.py   |    32 +
 .../pandas/tests/connect/test_parity_namespace.py  |    30 +
 .../tests/connect/test_parity_numpy_compat.py      |    34 +
 .../pandas/tests/connect/test_parity_repr.py       |    30 +
 .../pandas/tests/connect/test_parity_scalars.py    |    30 +
 .../tests/connect/test_parity_spark_functions.py   |    32 +
 .../pandas/tests/connect/test_parity_sql.py        |    30 +
 .../pandas/tests/connect/test_parity_typedef.py    |    30 +
 .../pandas/tests/connect/test_parity_utils.py      |    30 +
 .../pandas/tests/connect/window/__init__.py        |    16 +
 .../tests/connect/window/test_parity_ewm_error.py  |    35 +
 .../tests/connect/window/test_parity_ewm_mean.py   |    35 +
 .../tests/connect/window/test_parity_expanding.py  |    34 +
 .../connect/window/test_parity_expanding_adv.py    |    34 +
 .../connect/window/test_parity_expanding_error.py  |    34 +
 .../connect/window/test_parity_groupby_ewm_mean.py |    35 +
 .../window/test_parity_groupby_expanding.py        |    34 +
 .../window/test_parity_groupby_expanding_adv.py    |    34 +
 .../connect/window/test_parity_groupby_rolling.py  |    34 +
 .../window/test_parity_groupby_rolling_adv.py      |    34 +
 .../window/test_parity_groupby_rolling_count.py    |    34 +
 .../tests/connect/window/test_parity_missing.py    |    34 +
 .../tests/connect/window/test_parity_rolling.py    |    34 +
 .../connect/window/test_parity_rolling_adv.py      |    34 +
 .../connect/window/test_parity_rolling_count.py    |    34 +
 .../connect/window/test_parity_rolling_error.py    |    34 +
 .../pyspark/pandas/tests/data_type_ops/__init__.py |    16 +
 .../pandas/tests/data_type_ops/test_as_type.py     |    91 +
 .../pandas/tests/data_type_ops/test_base.py        |   104 +
 .../pandas/tests/data_type_ops/test_binary_ops.py  |   222 +
 .../pandas/tests/data_type_ops/test_boolean_ops.py |   833 +
 .../tests/data_type_ops/test_categorical_ops.py    |   554 +
 .../pandas/tests/data_type_ops/test_complex_ops.py |   370 +
 .../pandas/tests/data_type_ops/test_date_ops.py    |   255 +
 .../tests/data_type_ops/test_datetime_ops.py       |   258 +
 .../pandas/tests/data_type_ops/test_null_ops.py    |   176 +
 .../tests/data_type_ops/test_num_arithmetic.py     |    93 +
 .../pandas/tests/data_type_ops/test_num_mod.py     |    73 +
 .../pandas/tests/data_type_ops/test_num_mul_div.py |   154 +
 .../pandas/tests/data_type_ops/test_num_ops.py     |   441 +
 .../pandas/tests/data_type_ops/test_num_pow.py     |    65 +
 .../pandas/tests/data_type_ops/test_num_reverse.py |   138 +
 .../pandas/tests/data_type_ops/test_string_ops.py  |   348 +
 .../tests/data_type_ops/test_timedelta_ops.py      |   217 +
 .../pandas/tests/data_type_ops/test_udt_ops.py     |   210 +
 .../pandas/tests/data_type_ops/testing_utils.py    |   221 +
 .../pandas/tests/diff_frames_ops/__init__.py       |    16 +
 .../pandas/tests/diff_frames_ops/test_align.py     |   111 +
 .../tests/diff_frames_ops/test_arithmetic.py       |   194 +
 .../tests/diff_frames_ops/test_arithmetic_chain.py |   187 +
 .../diff_frames_ops/test_arithmetic_chain_ext.py   |   113 +
 .../test_arithmetic_chain_ext_float.py             |   115 +
 .../tests/diff_frames_ops/test_arithmetic_ext.py   |    92 +
 .../diff_frames_ops/test_arithmetic_ext_float.py   |    92 +
 .../tests/diff_frames_ops/test_assign_frame.py     |   236 +
 .../tests/diff_frames_ops/test_assign_series.py    |   234 +
 .../pandas/tests/diff_frames_ops/test_basic.py     |   232 +
 .../tests/diff_frames_ops/test_basic_slow.py       |   191 +
 .../pandas/tests/diff_frames_ops/test_bitwise.py   |   103 +
 .../tests/diff_frames_ops/test_combine_first.py    |   103 +
 .../tests/diff_frames_ops/test_compare_series.py   |   148 +
 .../tests/diff_frames_ops/test_concat_inner.py     |   116 +
 .../tests/diff_frames_ops/test_concat_outer.py     |    74 +
 .../pandas/tests/diff_frames_ops/test_corrwith.py  |   128 +
 .../pandas/tests/diff_frames_ops/test_cov.py       |    86 +
 .../pandas/tests/diff_frames_ops/test_dot_frame.py |    95 +
 .../tests/diff_frames_ops/test_dot_series.py       |   134 +
 .../pandas/tests/diff_frames_ops/test_error.py     |   247 +
 .../pandas/tests/diff_frames_ops/test_groupby.py   |   120 +
 .../diff_frames_ops/test_groupby_aggregate.py      |   132 +
 .../tests/diff_frames_ops/test_groupby_apply.py    |    75 +
 .../diff_frames_ops/test_groupby_cumulative.py     |   175 +
 .../tests/diff_frames_ops/test_groupby_diff.py     |    75 +
 .../tests/diff_frames_ops/test_groupby_diff_len.py |    93 +
 .../diff_frames_ops/test_groupby_expanding.py      |    93 +
 .../diff_frames_ops/test_groupby_expanding_adv.py  |    55 +
 .../test_groupby_expanding_count.py                |    52 +
 .../tests/diff_frames_ops/test_groupby_fillna.py   |    98 +
 .../tests/diff_frames_ops/test_groupby_filter.py   |    75 +
 .../tests/diff_frames_ops/test_groupby_rolling.py  |    93 +
 .../diff_frames_ops/test_groupby_rolling_adv.py    |    53 +
 .../diff_frames_ops/test_groupby_rolling_count.py  |    79 +
 .../tests/diff_frames_ops/test_groupby_shift.py    |    78 +
 .../test_groupby_split_apply_combine.py            |   106 +
 .../diff_frames_ops/test_groupby_transform.py      |    75 +
 .../pandas/tests/diff_frames_ops/test_index.py     |   102 +
 .../pandas/tests/diff_frames_ops/test_series.py    |   129 +
 .../tests/diff_frames_ops/test_setitem_frame.py    |   118 +
 .../tests/diff_frames_ops/test_setitem_series.py   |   175 +
 python/pyspark/pandas/tests/frame/__init__.py      |    16 +
 python/pyspark/pandas/tests/frame/test_asfreq.py   |    80 +
 python/pyspark/pandas/tests/frame/test_asof.py     |    99 +
 python/pyspark/pandas/tests/frame/test_attrs.py    |   358 +
 python/pyspark/pandas/tests/frame/test_axis.py     |   132 +
 .../pyspark/pandas/tests/frame/test_constructor.py |   587 +
 .../pyspark/pandas/tests/frame/test_conversion.py  |    68 +
 .../pyspark/pandas/tests/frame/test_interpolate.py |    84 +
 .../pandas/tests/frame/test_interpolate_error.py   |    70 +
 .../pyspark/pandas/tests/frame/test_reindexing.py  |   855 +
 .../pyspark/pandas/tests/frame/test_reshaping.py   |   477 +
 python/pyspark/pandas/tests/frame/test_spark.py    |   304 +
 python/pyspark/pandas/tests/frame/test_take.py     |    86 +
 python/pyspark/pandas/tests/frame/test_take_adv.py |    97 +
 .../pyspark/pandas/tests/frame/test_time_series.py |   133 +
 python/pyspark/pandas/tests/frame/test_truncate.py |   126 +
 python/pyspark/pandas/tests/groupby/__init__.py    |    16 +
 .../pyspark/pandas/tests/groupby/test_aggregate.py |   283 +
 .../pandas/tests/groupby/test_apply_func.py        |   500 +
 python/pyspark/pandas/tests/groupby/test_corr.py   |    77 +
 .../pandas/tests/groupby/test_cumulative.py        |   401 +
 .../pyspark/pandas/tests/groupby/test_describe.py  |   126 +
 .../pyspark/pandas/tests/groupby/test_groupby.py   |   486 +
 .../pyspark/pandas/tests/groupby/test_grouping.py  |   167 +
 .../pyspark/pandas/tests/groupby/test_head_tail.py |   215 +
 python/pyspark/pandas/tests/groupby/test_index.py  |   177 +
 .../pyspark/pandas/tests/groupby/test_missing.py   |   140 +
 .../pandas/tests/groupby/test_missing_data.py      |   316 +
 .../tests/groupby/test_nlargest_nsmallest.py       |   115 +
 python/pyspark/pandas/tests/groupby/test_raises.py |    48 +
 python/pyspark/pandas/tests/groupby/test_rank.py   |    87 +
 python/pyspark/pandas/tests/groupby/test_size.py   |    66 +
 .../pandas/tests/groupby/test_split_apply.py       |   180 +
 .../pandas/tests/groupby/test_split_apply_count.py |    42 +
 .../pandas/tests/groupby/test_split_apply_first.py |    42 +
 .../pandas/tests/groupby/test_split_apply_last.py  |    42 +
 .../tests/groupby/test_split_apply_min_max.py      |    42 +
 .../pandas/tests/groupby/test_split_apply_skew.py  |    42 +
 .../pandas/tests/groupby/test_split_apply_std.py   |    42 +
 .../pandas/tests/groupby/test_split_apply_var.py   |    42 +
 python/pyspark/pandas/tests/groupby/test_stat.py   |   136 +
 .../pyspark/pandas/tests/groupby/test_stat_adv.py  |   154 +
 .../pyspark/pandas/tests/groupby/test_stat_ddof.py |    84 +
 .../pyspark/pandas/tests/groupby/test_stat_func.py |   111 +
 .../pyspark/pandas/tests/groupby/test_stat_prod.py |    79 +
 .../pandas/tests/groupby/test_value_counts.py      |   100 +
 python/pyspark/pandas/tests/indexes/__init__.py    |    16 +
 python/pyspark/pandas/tests/indexes/test_align.py  |    83 +
 python/pyspark/pandas/tests/indexes/test_append.py |   122 +
 python/pyspark/pandas/tests/indexes/test_asof.py   |    82 +
 python/pyspark/pandas/tests/indexes/test_astype.py |    97 +
 python/pyspark/pandas/tests/indexes/test_basic.py  |   213 +
 .../pyspark/pandas/tests/indexes/test_category.py  |   406 +
 .../pandas/tests/indexes/test_conversion.py        |   317 +
 .../pyspark/pandas/tests/indexes/test_datetime.py  |   136 +
 .../pandas/tests/indexes/test_datetime_at.py       |    61 +
 .../pandas/tests/indexes/test_datetime_between.py  |    72 +
 .../pandas/tests/indexes/test_datetime_ceil.py     |    41 +
 .../pandas/tests/indexes/test_datetime_floor.py    |    41 +
 .../pandas/tests/indexes/test_datetime_iso.py      |    40 +
 .../pandas/tests/indexes/test_datetime_map.py      |    58 +
 .../pandas/tests/indexes/test_datetime_property.py |    62 +
 .../pandas/tests/indexes/test_datetime_round.py    |    41 +
 .../pyspark/pandas/tests/indexes/test_default.py   |   104 +
 python/pyspark/pandas/tests/indexes/test_delete.py |    80 +
 python/pyspark/pandas/tests/indexes/test_diff.py   |   119 +
 python/pyspark/pandas/tests/indexes/test_drop.py   |   169 +
 .../pyspark/pandas/tests/indexes/test_getattr.py   |    72 +
 .../pyspark/pandas/tests/indexes/test_indexing.py  |   432 +
 .../pandas/tests/indexes/test_indexing_adv.py      |   399 +
 .../pandas/tests/indexes/test_indexing_basic.py    |   164 +
 .../pandas/tests/indexes/test_indexing_iloc.py     |   296 +
 .../pandas/tests/indexes/test_indexing_loc.py      |   388 +
 .../pandas/tests/indexes/test_indexing_loc_2d.py   |   241 +
 .../tests/indexes/test_indexing_loc_multi_idx.py   |   142 +
 python/pyspark/pandas/tests/indexes/test_insert.py |   102 +
 .../pandas/tests/indexes/test_intersection.py      |   160 +
 python/pyspark/pandas/tests/indexes/test_level.py  |   167 +
 python/pyspark/pandas/tests/indexes/test_map.py    |   111 +
 .../pyspark/pandas/tests/indexes/test_missing.py   |   237 +
 .../pyspark/pandas/tests/indexes/test_monotonic.py |   177 +
 python/pyspark/pandas/tests/indexes/test_name.py   |   169 +
 .../pyspark/pandas/tests/indexes/test_reindex.py   |   291 +
 python/pyspark/pandas/tests/indexes/test_rename.py |   311 +
 python/pyspark/pandas/tests/indexes/test_repeat.py |    58 +
 .../pandas/tests/indexes/test_reset_index.py       |   158 +
 python/pyspark/pandas/tests/indexes/test_sort.py   |    92 +
 python/pyspark/pandas/tests/indexes/test_stat.py   |   174 +
 .../pandas/tests/indexes/test_symmetric_diff.py    |   110 +
 python/pyspark/pandas/tests/indexes/test_take.py   |    77 +
 .../pyspark/pandas/tests/indexes/test_timedelta.py |   119 +
 python/pyspark/pandas/tests/indexes/test_union.py  |   201 +
 python/pyspark/pandas/tests/indexes/test_unique.py |   157 +
 python/pyspark/pandas/tests/io/__init__.py         |    16 +
 python/pyspark/pandas/tests/io/test_csv.py         |   422 +
 .../pandas/tests/io/test_dataframe_conversion.py   |   278 +
 .../pandas/tests/io/test_dataframe_spark_io.py     |   459 +
 python/pyspark/pandas/tests/io/test_feather.py     |    61 +
 python/pyspark/pandas/tests/io/test_io.py          |   162 +
 .../pandas/tests/io/test_series_conversion.py      |    79 +
 python/pyspark/pandas/tests/io/test_stata.py       |    60 +
 python/pyspark/pandas/tests/plot/__init__.py       |    16 +
 .../pyspark/pandas/tests/plot/test_frame_plot.py   |   144 +
 .../tests/plot/test_frame_plot_matplotlib.py       |   481 +
 .../pandas/tests/plot/test_frame_plot_plotly.py    |   276 +
 .../pyspark/pandas/tests/plot/test_series_plot.py  |    94 +
 .../tests/plot/test_series_plot_matplotlib.py      |   399 +
 .../pandas/tests/plot/test_series_plot_plotly.py   |   237 +
 python/pyspark/pandas/tests/resample/__init__.py   |    16 +
 python/pyspark/pandas/tests/resample/test_error.py |    96 +
 python/pyspark/pandas/tests/resample/test_frame.py |   147 +
 .../pyspark/pandas/tests/resample/test_missing.py  |   134 +
 python/pyspark/pandas/tests/resample/test_on.py    |    59 +
 .../pyspark/pandas/tests/resample/test_series.py   |   146 +
 .../pyspark/pandas/tests/resample/test_timezone.py |    76 +
 python/pyspark/pandas/tests/reshape/__init__.py    |    16 +
 .../pandas/tests/reshape/test_get_dummies.py       |   119 +
 .../tests/reshape/test_get_dummies_kwargs.py       |    62 +
 .../tests/reshape/test_get_dummies_multiindex.py   |   101 +
 .../tests/reshape/test_get_dummies_object.py       |    77 +
 .../tests/reshape/test_get_dummies_prefix.py       |    86 +
 .../pandas/tests/reshape/test_merge_asof.py        |   213 +
 python/pyspark/pandas/tests/series/__init__.py     |    16 +
 python/pyspark/pandas/tests/series/test_all_any.py |    94 +
 python/pyspark/pandas/tests/series/test_arg_ops.py |   186 +
 python/pyspark/pandas/tests/series/test_as_of.py   |   126 +
 python/pyspark/pandas/tests/series/test_as_type.py |   182 +
 python/pyspark/pandas/tests/series/test_compute.py |   588 +
 .../pyspark/pandas/tests/series/test_conversion.py |    86 +
 .../pyspark/pandas/tests/series/test_cumulative.py |   127 +
 .../pyspark/pandas/tests/series/test_datetime.py   |   294 +
 python/pyspark/pandas/tests/series/test_index.py   |   464 +
 .../pandas/tests/series/test_interpolate.py        |    99 +
 .../pandas/tests/series/test_missing_data.py       |   242 +
 python/pyspark/pandas/tests/series/test_series.py  |   811 +
 python/pyspark/pandas/tests/series/test_sort.py    |   151 +
 python/pyspark/pandas/tests/series/test_stat.py    |   710 +
 .../pandas/tests/series/test_string_ops_adv.py     |   230 +
 .../pandas/tests/series/test_string_ops_basic.py   |   181 +
 .../pyspark/pandas/tests/test_arrow_interface.py   |    70 +
 python/pyspark/pandas/tests/test_categorical.py    |   675 +
 python/pyspark/pandas/tests/test_config.py         |   153 +
 python/pyspark/pandas/tests/test_extension.py      |   152 +
 python/pyspark/pandas/tests/test_frame_spark.py    |   155 +
 .../pyspark/pandas/tests/test_generic_functions.py |   104 +
 python/pyspark/pandas/tests/test_indexops_spark.py |    75 +
 python/pyspark/pandas/tests/test_internal.py       |   144 +
 python/pyspark/pandas/tests/test_namespace.py      |   687 +
 python/pyspark/pandas/tests/test_numpy_compat.py   |   202 +
 python/pyspark/pandas/tests/test_repr.py           |   185 +
 python/pyspark/pandas/tests/test_scalars.py        |    52 +
 .../pyspark/pandas/tests/test_spark_functions.py   |    34 +
 python/pyspark/pandas/tests/test_sql.py            |   116 +
 python/pyspark/pandas/tests/test_typedef.py        |   474 +
 python/pyspark/pandas/tests/test_utils.py          |   258 +
 python/pyspark/pandas/tests/window/__init__.py     |    16 +
 .../pyspark/pandas/tests/window/test_ewm_error.py  |    90 +
 .../pyspark/pandas/tests/window/test_ewm_mean.py   |   261 +
 .../pyspark/pandas/tests/window/test_expanding.py  |    89 +
 .../pandas/tests/window/test_expanding_adv.py      |    49 +
 .../pandas/tests/window/test_expanding_error.py    |    44 +
 .../pandas/tests/window/test_groupby_ewm_mean.py   |   226 +
 .../pandas/tests/window/test_groupby_expanding.py  |   128 +
 .../tests/window/test_groupby_expanding_adv.py     |    51 +
 .../pandas/tests/window/test_groupby_rolling.py    |   125 +
 .../tests/window/test_groupby_rolling_adv.py       |    53 +
 .../tests/window/test_groupby_rolling_count.py     |   106 +
 python/pyspark/pandas/tests/window/test_missing.py |   461 +
 python/pyspark/pandas/tests/window/test_rolling.py |    84 +
 .../pandas/tests/window/test_rolling_adv.py        |    49 +
 .../pandas/tests/window/test_rolling_count.py      |    65 +
 .../pandas/tests/window/test_rolling_error.py      |    47 +
 python/pyspark/pandas/typedef/__init__.py          |    18 +
 python/pyspark/pandas/typedef/typehints.py         |   919 +
 python/pyspark/pandas/usage_logging/__init__.py    |   140 +
 .../pyspark/pandas/usage_logging/usage_logger.py   |   132 +
 python/pyspark/pandas/utils.py                     |  1200 +
 python/pyspark/pandas/window.py                    |  2705 ++
 python/pyspark/pipelines/__init__.py               |    33 +
 .../pipelines/add_pipeline_analysis_context.py     |    48 +
 python/pyspark/pipelines/api.py                    |   506 +
 .../pyspark/pipelines/block_session_mutations.py   |   135 +
 python/pyspark/pipelines/cli.py                    |   447 +
 python/pyspark/pipelines/flow.py                   |    43 +
 python/pyspark/pipelines/graph_element_registry.py |    83 +
 python/pyspark/pipelines/init_cli.py               |    83 +
 python/pyspark/pipelines/logging_utils.py          |    42 +
 python/pyspark/pipelines/output.py                 |    87 +
 python/pyspark/pipelines/source_code_location.py   |    72 +
 .../spark_connect_graph_element_registry.py        |   152 +
 python/pyspark/pipelines/spark_connect_pipeline.py |    99 +
 python/pyspark/pipelines/tests/__init__.py         |    16 +
 .../tests/local_graph_element_registry.py          |    58 +
 .../tests/test_add_pipeline_analysis_context.py    |    96 +
 .../tests/test_block_session_mutations.py          |   256 +
 python/pyspark/pipelines/tests/test_cli.py         |   555 +
 python/pyspark/pipelines/tests/test_decorators.py  |    68 +
 .../pipelines/tests/test_graph_element_registry.py |   136 +
 python/pyspark/pipelines/tests/test_init_cli.py    |    94 +
 .../pyspark/pipelines/tests/test_spark_connect.py  |    97 +
 python/pyspark/pipelines/type_error_utils.py       |    51 +
 python/pyspark/profiler.py                         |   535 +
 python/pyspark/py.typed                            |     1 +
 python/pyspark/rddsampler.py                       |   115 +
 python/pyspark/resource/__init__.py                |    38 +
 python/pyspark/resource/information.py             |    69 +
 python/pyspark/resource/profile.py                 |   342 +
 python/pyspark/resource/requests.py                |   550 +
 python/pyspark/resource/tests/__init__.py          |    16 +
 .../resource/tests/test_connect_resources.py       |    82 +
 python/pyspark/resource/tests/test_resources.py    |    99 +
 python/pyspark/resultiterable.py                   |    45 +
 python/pyspark/serializers.py                      |   670 +
 python/pyspark/shell.py                            |   143 +
 python/pyspark/shuffle.py                          |   854 +
 python/pyspark/sql/__init__.py                     |    82 +
 python/pyspark/sql/_typing.pyi                     |    88 +
 python/pyspark/sql/avro/__init__.py                |    18 +
 python/pyspark/sql/avro/functions.py               |   224 +
 python/pyspark/sql/catalog.py                      |  1281 +
 python/pyspark/sql/classic/__init__.py             |    18 +
 python/pyspark/sql/classic/column.py               |   658 +
 python/pyspark/sql/classic/dataframe.py            |  2041 ++
 python/pyspark/sql/classic/table_arg.py            |    52 +
 python/pyspark/sql/classic/window.py               |   150 +
 python/pyspark/sql/column.py                       |  1645 ++
 python/pyspark/sql/conf.py                         |   175 +
 python/pyspark/sql/connect/__init__.py             |    18 +
 python/pyspark/sql/connect/_typing.py              |    92 +
 python/pyspark/sql/connect/avro/__init__.py        |    18 +
 python/pyspark/sql/connect/avro/functions.py       |   139 +
 python/pyspark/sql/connect/catalog.py              |   365 +
 python/pyspark/sql/connect/client/__init__.py      |    23 +
 python/pyspark/sql/connect/client/artifact.py      |   509 +
 python/pyspark/sql/connect/client/core.py          |  2407 ++
 python/pyspark/sql/connect/client/reattach.py      |   345 +
 python/pyspark/sql/connect/client/retries.py       |   392 +
 python/pyspark/sql/connect/column.py               |   634 +
 python/pyspark/sql/connect/conf.py                 |   167 +
 python/pyspark/sql/connect/conversion.py           |    64 +
 python/pyspark/sql/connect/dataframe.py            |  2415 ++
 python/pyspark/sql/connect/datasource.py           |    47 +
 python/pyspark/sql/connect/expressions.py          |  1366 +
 python/pyspark/sql/connect/functions/__init__.py   |    23 +
 python/pyspark/sql/connect/functions/builtin.py    |  5195 ++++
 .../pyspark/sql/connect/functions/partitioning.py  |   112 +
 python/pyspark/sql/connect/group.py                |   631 +
 python/pyspark/sql/connect/logging.py              |    67 +
 python/pyspark/sql/connect/merge.py                |   256 +
 python/pyspark/sql/connect/observation.py          |   117 +
 python/pyspark/sql/connect/plan.py                 |  2856 ++
 python/pyspark/sql/connect/profiler.py             |    41 +
 python/pyspark/sql/connect/proto/__init__.py       |    28 +
 python/pyspark/sql/connect/proto/base_pb2.py       |   275 +
 python/pyspark/sql/connect/proto/base_pb2.pyi      |  4313 +++
 python/pyspark/sql/connect/proto/base_pb2_grpc.py  |   593 +
 python/pyspark/sql/connect/proto/catalog_pb2.py    |   117 +
 python/pyspark/sql/connect/proto/catalog_pb2.pyi   |  1133 +
 python/pyspark/sql/connect/proto/commands_pb2.py   |   199 +
 python/pyspark/sql/connect/proto/commands_pb2.pyi  |  2419 ++
 python/pyspark/sql/connect/proto/common_pb2.py     |    91 +
 python/pyspark/sql/connect/proto/common_pb2.pyi    |   730 +
 .../sql/connect/proto/example_plugins_pb2.py       |    61 +
 .../sql/connect/proto/example_plugins_pb2.pyi      |   112 +
 .../pyspark/sql/connect/proto/expressions_pb2.py   |   156 +
 .../pyspark/sql/connect/proto/expressions_pb2.pyi  |  2167 ++
 python/pyspark/sql/connect/proto/ml_common_pb2.py  |    66 +
 python/pyspark/sql/connect/proto/ml_common_pb2.pyi |   177 +
 python/pyspark/sql/connect/proto/ml_pb2.py         |    82 +
 python/pyspark/sql/connect/proto/ml_pb2.pyi        |   678 +
 python/pyspark/sql/connect/proto/pipelines_pb2.py  |   130 +
 python/pyspark/sql/connect/proto/pipelines_pb2.pyi |  1574 +
 python/pyspark/sql/connect/proto/relations_pb2.py  |   251 +
 python/pyspark/sql/connect/proto/relations_pb2.pyi |  4585 +++
 python/pyspark/sql/connect/proto/types_pb2.py      |   109 +
 python/pyspark/sql/connect/proto/types_pb2.pyi     |  1126 +
 python/pyspark/sql/connect/protobuf/__init__.py    |    18 +
 python/pyspark/sql/connect/protobuf/functions.py   |   165 +
 python/pyspark/sql/connect/readwriter.py           |  1015 +
 python/pyspark/sql/connect/resource/__init__.py    |    16 +
 python/pyspark/sql/connect/resource/profile.py     |    69 +
 python/pyspark/sql/connect/session.py              |  1352 +
 python/pyspark/sql/connect/shell/__init__.py       |    26 +
 python/pyspark/sql/connect/shell/progress.py       |   212 +
 python/pyspark/sql/connect/sql_formatter.py        |    76 +
 python/pyspark/sql/connect/streaming/__init__.py   |    16 +
 python/pyspark/sql/connect/streaming/query.py      |   464 +
 python/pyspark/sql/connect/streaming/readwriter.py |   758 +
 .../sql/connect/streaming/worker/__init__.py       |    18 +
 .../streaming/worker/foreach_batch_worker.py       |   103 +
 .../connect/streaming/worker/listener_worker.py    |   117 +
 python/pyspark/sql/connect/table_arg.py            |   106 +
 python/pyspark/sql/connect/tvf.py                  |   168 +
 python/pyspark/sql/connect/types.py                |   435 +
 python/pyspark/sql/connect/udf.py                  |   340 +
 python/pyspark/sql/connect/udtf.py                 |   257 +
 python/pyspark/sql/connect/utils.py                |   117 +
 python/pyspark/sql/connect/window.py               |   190 +
 python/pyspark/sql/context.py                      |   829 +
 python/pyspark/sql/conversion.py                   |  1158 +
 python/pyspark/sql/dataframe.py                    |  7159 +++++
 python/pyspark/sql/datasource.py                   |  1236 +
 python/pyspark/sql/datasource_internal.py          |   146 +
 python/pyspark/sql/functions/__init__.py           |   565 +
 python/pyspark/sql/functions/builtin.py            | 29507 +++++++++++++++++++
 python/pyspark/sql/functions/partitioning.py       |   252 +
 python/pyspark/sql/geo_utils.py                    |   103 +
 python/pyspark/sql/group.py                        |   560 +
 python/pyspark/sql/interchange.py                  |    89 +
 python/pyspark/sql/internal.py                     |   173 +
 python/pyspark/sql/merge.py                        |   253 +
 python/pyspark/sql/metrics.py                      |   316 +
 python/pyspark/sql/observation.py                  |   174 +
 python/pyspark/sql/pandas/__init__.py              |    21 +
 python/pyspark/sql/pandas/_typing/__init__.pyi     |   452 +
 .../sql/pandas/_typing/protocols/__init__.pyi      |    17 +
 .../pyspark/sql/pandas/_typing/protocols/frame.pyi |    20 +
 .../sql/pandas/_typing/protocols/series.pyi        |    20 +
 python/pyspark/sql/pandas/conversion.py            |  1016 +
 python/pyspark/sql/pandas/functions.py             |  1066 +
 python/pyspark/sql/pandas/functions.pyi            |   189 +
 python/pyspark/sql/pandas/group_ops.py             |  1229 +
 python/pyspark/sql/pandas/map_ops.py               |   127 +
 python/pyspark/sql/pandas/serializers.py           |  2169 ++
 python/pyspark/sql/pandas/typehints.py             |   624 +
 python/pyspark/sql/pandas/types.py                 |  1671 ++
 python/pyspark/sql/pandas/utils.py                 |   126 +
 python/pyspark/sql/plot/__init__.py                |    21 +
 python/pyspark/sql/plot/core.py                    |   788 +
 python/pyspark/sql/plot/plotly.py                  |   282 +
 python/pyspark/sql/profiler.py                     |   616 +
 python/pyspark/sql/protobuf/__init__.py            |    18 +
 python/pyspark/sql/protobuf/functions.py           |   336 +
 python/pyspark/sql/readwriter.py                   |  2545 ++
 python/pyspark/sql/session.py                      |  2530 ++
 python/pyspark/sql/sql_formatter.py                |    86 +
 python/pyspark/sql/streaming/__init__.py           |    26 +
 python/pyspark/sql/streaming/benchmark/__init__.py |    16 +
 .../benchmark/benchmark_tws_state_server.py        |   375 +
 .../pyspark/sql/streaming/benchmark/tws_utils.py   |    73 +
 python/pyspark/sql/streaming/benchmark/utils.py    |    26 +
 python/pyspark/sql/streaming/list_state_client.py  |   211 +
 python/pyspark/sql/streaming/listener.py           |  1134 +
 python/pyspark/sql/streaming/map_state_client.py   |   338 +
 .../sql/streaming/proto/StateMessage_pb2.py        |   149 +
 .../sql/streaming/proto/StateMessage_pb2.pyi       |  1454 +
 python/pyspark/sql/streaming/proto/__init__.py     |    16 +
 .../streaming/python_streaming_source_runner.py    |   216 +
 python/pyspark/sql/streaming/query.py              |   810 +
 python/pyspark/sql/streaming/readwriter.py         |  1896 ++
 python/pyspark/sql/streaming/state.py              |   298 +
 python/pyspark/sql/streaming/stateful_processor.py |   475 +
 .../sql/streaming/stateful_processor_api_client.py |   635 +
 .../sql/streaming/stateful_processor_util.py       |   237 +
 .../transform_with_state_driver_worker.py          |   106 +
 python/pyspark/sql/streaming/value_state_client.py |   111 +
 python/pyspark/sql/table_arg.py                    |   224 +
 python/pyspark/sql/tests/__init__.py               |    16 +
 python/pyspark/sql/tests/arrow/__init__.py         |    16 +
 python/pyspark/sql/tests/arrow/test_arrow.py       |  1936 ++
 .../sql/tests/arrow/test_arrow_cogrouped_map.py    |   441 +
 .../sql/tests/arrow/test_arrow_grouped_map.py      |   501 +
 python/pyspark/sql/tests/arrow/test_arrow_map.py   |   356 +
 .../sql/tests/arrow/test_arrow_python_udf.py       |   540 +
 python/pyspark/sql/tests/arrow/test_arrow_udf.py   |   321 +
 .../sql/tests/arrow/test_arrow_udf_grouped_agg.py  |  1295 +
 .../sql/tests/arrow/test_arrow_udf_scalar.py       |  1294 +
 .../sql/tests/arrow/test_arrow_udf_typehints.py    |   466 +
 .../sql/tests/arrow/test_arrow_udf_window.py       |   860 +
 python/pyspark/sql/tests/arrow/test_arrow_udtf.py  |  1747 ++
 python/pyspark/sql/tests/coercion/__init__.py      |    16 +
 .../golden_pandas_udf_input_type_coercion_base.csv |    40 +
 .../golden_pandas_udf_input_type_coercion_base.md  |    41 +
 ...golden_pandas_udf_return_type_coercion_base.csv |    16 +
 .../golden_pandas_udf_return_type_coercion_base.md |    17 +
 ...lden_python_udf_input_type_coercion_vanilla.csv |    40 +
 ...olden_python_udf_input_type_coercion_vanilla.md |    41 +
 ...n_python_udf_input_type_coercion_with_arrow.csv |    40 +
 ...en_python_udf_input_type_coercion_with_arrow.md |    41 +
 ...f_input_type_coercion_with_arrow_and_pandas.csv |    40 +
 ...df_input_type_coercion_with_arrow_and_pandas.md |    41 +
 ...den_python_udf_return_type_coercion_vanilla.csv |    16 +
 ...lden_python_udf_return_type_coercion_vanilla.md |    17 +
 ..._python_udf_return_type_coercion_with_arrow.csv |    16 +
 ...n_python_udf_return_type_coercion_with_arrow.md |    17 +
 ..._return_type_coercion_with_arrow_and_pandas.csv |    16 +
 ...f_return_type_coercion_with_arrow_and_pandas.md |    17 +
 .../tests/coercion/test_pandas_udf_input_type.py   |   360 +
 .../tests/coercion/test_pandas_udf_return_type.py  |   290 +
 .../tests/coercion/test_python_udf_input_type.py   |   391 +
 .../tests/coercion/test_python_udf_return_type.py  |   287 +
 python/pyspark/sql/tests/connect/__init__.py       |    16 +
 python/pyspark/sql/tests/connect/arrow/__init__.py |    16 +
 .../sql/tests/connect/arrow/test_parity_arrow.py   |   177 +
 .../arrow/test_parity_arrow_cogrouped_map.py       |    30 +
 .../connect/arrow/test_parity_arrow_grouped_map.py |    30 +
 .../tests/connect/arrow/test_parity_arrow_map.py   |    30 +
 .../connect/arrow/test_parity_arrow_python_udf.py  |   115 +
 .../tests/connect/arrow/test_parity_arrow_udf.py   |    35 +
 .../arrow/test_parity_arrow_udf_grouped_agg.py     |    50 +
 .../connect/arrow/test_parity_arrow_udf_scalar.py  |    50 +
 .../connect/arrow/test_parity_arrow_udf_window.py  |    50 +
 .../tests/connect/arrow/test_parity_arrow_udtf.py  |    30 +
 .../pyspark/sql/tests/connect/client/__init__.py   |    16 +
 .../sql/tests/connect/client/test_artifact.py      |   443 +
 .../connect/client/test_artifact_localcluster.py   |    43 +
 .../sql/tests/connect/client/test_client.py        |   661 +
 .../connect/client/test_client_call_stack_trace.py |   318 +
 .../tests/connect/client/test_client_retries.py    |   242 +
 .../sql/tests/connect/client/test_reattach.py      |    94 +
 .../pyspark/sql/tests/connect/pandas/__init__.py   |    16 +
 .../sql/tests/connect/pandas/streaming/__init__.py |    16 +
 .../test_parity_pandas_grouped_map_with_state.py   |    33 +
 .../test_parity_pandas_transform_with_state.py     |    59 +
 ...y_pandas_transform_with_state_state_variable.py |    54 +
 .../streaming/test_parity_transform_with_state.py  |    59 +
 ...t_parity_transform_with_state_state_variable.py |    54 +
 .../pandas/test_parity_pandas_cogrouped_map.py     |    32 +
 .../pandas/test_parity_pandas_grouped_map.py       |    29 +
 .../tests/connect/pandas/test_parity_pandas_map.py |    32 +
 .../tests/connect/pandas/test_parity_pandas_udf.py |    29 +
 .../pandas/test_parity_pandas_udf_grouped_agg.py   |    32 +
 .../pandas/test_parity_pandas_udf_scalar.py        |    28 +
 .../pandas/test_parity_pandas_udf_window.py        |    32 +
 python/pyspark/sql/tests/connect/shell/__init__.py |    16 +
 .../sql/tests/connect/shell/test_progress.py       |   145 +
 .../sql/tests/connect/streaming/__init__.py        |    16 +
 .../tests/connect/streaming/test_parity_foreach.py |    29 +
 .../connect/streaming/test_parity_foreach_batch.py |   143 +
 .../connect/streaming/test_parity_listener.py      |   322 +
 .../connect/streaming/test_parity_streaming.py     |    30 +
 .../sql/tests/connect/test_connect_basic.py        |  1666 ++
 .../sql/tests/connect/test_connect_channel.py      |   156 +
 .../tests/connect/test_connect_clone_session.py    |   146 +
 .../sql/tests/connect/test_connect_collection.py   |   376 +
 .../sql/tests/connect/test_connect_column.py       |  1078 +
 .../sql/tests/connect/test_connect_creation.py     |   725 +
 .../connect/test_connect_dataframe_property.py     |   466 +
 .../sql/tests/connect/test_connect_error.py        |   248 +
 .../sql/tests/connect/test_connect_function.py     |  2596 ++
 .../pyspark/sql/tests/connect/test_connect_plan.py |  1071 +
 .../sql/tests/connect/test_connect_readwriter.py   |   533 +
 .../sql/tests/connect/test_connect_retry.py        |   171 +
 .../sql/tests/connect/test_connect_session.py      |   331 +
 .../pyspark/sql/tests/connect/test_connect_stat.py |   805 +
 python/pyspark/sql/tests/connect/test_df_debug.py  |    73 +
 .../sql/tests/connect/test_parity_catalog.py       |    29 +
 .../sql/tests/connect/test_parity_collection.py    |    52 +
 .../sql/tests/connect/test_parity_column.py        |    33 +
 .../pyspark/sql/tests/connect/test_parity_conf.py  |    29 +
 .../sql/tests/connect/test_parity_creation.py      |    32 +
 .../sql/tests/connect/test_parity_dataframe.py     |    45 +
 .../connect/test_parity_dataframe_query_context.py |    30 +
 .../sql/tests/connect/test_parity_datasources.py   |    41 +
 .../sql/tests/connect/test_parity_errors.py        |    29 +
 .../sql/tests/connect/test_parity_frame_plot.py    |    29 +
 .../tests/connect/test_parity_frame_plot_plotly.py |    29 +
 .../sql/tests/connect/test_parity_functions.py     |    45 +
 .../sql/tests/connect/test_parity_geographytype.py |    30 +
 .../sql/tests/connect/test_parity_geometrytype.py  |    30 +
 .../pyspark/sql/tests/connect/test_parity_group.py |    29 +
 .../tests/connect/test_parity_job_cancellation.py  |    39 +
 .../tests/connect/test_parity_memory_profiler.py   |    67 +
 .../sql/tests/connect/test_parity_observation.py   |    33 +
 .../tests/connect/test_parity_python_datasource.py |    28 +
 .../test_parity_python_streaming_datasource.py     |    32 +
 .../sql/tests/connect/test_parity_readwriter.py    |    41 +
 .../sql/tests/connect/test_parity_repartition.py   |    32 +
 .../pyspark/sql/tests/connect/test_parity_serde.py |    37 +
 .../pyspark/sql/tests/connect/test_parity_sql.py   |    30 +
 .../pyspark/sql/tests/connect/test_parity_stat.py  |    32 +
 .../sql/tests/connect/test_parity_subquery.py      |    51 +
 .../pyspark/sql/tests/connect/test_parity_tvf.py   |    29 +
 .../pyspark/sql/tests/connect/test_parity_types.py |   113 +
 .../pyspark/sql/tests/connect/test_parity_udf.py   |    75 +
 .../tests/connect/test_parity_udf_combinations.py  |    33 +
 .../sql/tests/connect/test_parity_udf_profiler.py  |    74 +
 .../pyspark/sql/tests/connect/test_parity_udtf.py  |   161 +
 .../sql/tests/connect/test_parity_unified_udf.py   |    30 +
 python/pyspark/sql/tests/connect/test_resources.py |    32 +
 python/pyspark/sql/tests/connect/test_session.py   |   100 +
 python/pyspark/sql/tests/connect/test_utils.py     |    29 +
 python/pyspark/sql/tests/pandas/__init__.py        |    16 +
 python/pyspark/sql/tests/pandas/helper/__init__.py |    16 +
 .../helper/helper_pandas_transform_with_state.py   |  2174 ++
 .../pyspark/sql/tests/pandas/streaming/__init__.py |    16 +
 .../streaming/test_pandas_transform_with_state.py  |  1661 ++
 ...st_pandas_transform_with_state_checkpoint_v2.py |    42 +
 ...t_pandas_transform_with_state_state_variable.py |  1017 +
 ...form_with_state_state_variable_checkpoint_v2.py |    44 +
 .../pandas/streaming/test_transform_with_state.py  |    62 +
 .../test_transform_with_state_checkpoint_v2.py     |    42 +
 .../test_transform_with_state_state_variable.py    |    65 +
 ...form_with_state_state_variable_checkpoint_v2.py |    44 +
 python/pyspark/sql/tests/pandas/test_converter.py  |   587 +
 .../sql/tests/pandas/test_pandas_cogrouped_map.py  |   769 +
 .../sql/tests/pandas/test_pandas_grouped_map.py    |  1479 +
 .../pandas/test_pandas_grouped_map_with_state.py   |   350 +
 python/pyspark/sql/tests/pandas/test_pandas_map.py |   568 +
 .../sql/tests/pandas/test_pandas_sqlmetrics.py     |    60 +
 python/pyspark/sql/tests/pandas/test_pandas_udf.py |   488 +
 .../tests/pandas/test_pandas_udf_grouped_agg.py    |  1158 +
 .../sql/tests/pandas/test_pandas_udf_scalar.py     |  2072 ++
 .../sql/tests/pandas/test_pandas_udf_typehints.py  |   539 +
 ...pandas_udf_typehints_with_future_annotations.py |   372 +
 .../sql/tests/pandas/test_pandas_udf_window.py     |   715 +
 python/pyspark/sql/tests/plot/__init__.py          |    16 +
 python/pyspark/sql/tests/plot/test_frame_plot.py   |    93 +
 .../sql/tests/plot/test_frame_plot_plotly.py       |   522 +
 python/pyspark/sql/tests/streaming/__init__.py     |    16 +
 .../pyspark/sql/tests/streaming/test_streaming.py  |   684 +
 .../sql/tests/streaming/test_streaming_foreach.py  |   288 +
 .../streaming/test_streaming_foreach_batch.py      |   236 +
 .../sql/tests/streaming/test_streaming_listener.py |   674 +
 .../test_streaming_offline_state_repartition.py    |   557 +
 python/pyspark/sql/tests/test_artifact.py          |    87 +
 python/pyspark/sql/tests/test_catalog.py           |   507 +
 python/pyspark/sql/tests/test_collection.py        |   426 +
 python/pyspark/sql/tests/test_column.py            |   512 +
 python/pyspark/sql/tests/test_conf.py              |   105 +
 .../sql/tests/test_connect_compatibility.py        |   527 +
 python/pyspark/sql/tests/test_context.py           |   192 +
 python/pyspark/sql/tests/test_conversion.py        |   382 +
 python/pyspark/sql/tests/test_creation.py          |   275 +
 python/pyspark/sql/tests/test_dataframe.py         |  1225 +
 .../sql/tests/test_dataframe_query_context.py      |   496 +
 python/pyspark/sql/tests/test_datasources.py       |   345 +
 python/pyspark/sql/tests/test_errors.py            |    62 +
 python/pyspark/sql/tests/test_functions.py         |  3110 ++
 python/pyspark/sql/tests/test_geographytype.py     |   130 +
 python/pyspark/sql/tests/test_geometrytype.py      |   121 +
 python/pyspark/sql/tests/test_group.py             |   207 +
 python/pyspark/sql/tests/test_job_cancellation.py  |   209 +
 python/pyspark/sql/tests/test_listener.py          |   110 +
 python/pyspark/sql/tests/test_observation.py       |   252 +
 python/pyspark/sql/tests/test_python_datasource.py |  1271 +
 .../sql/tests/test_python_streaming_datasource.py  |   457 +
 python/pyspark/sql/tests/test_readwriter.py        |   414 +
 python/pyspark/sql/tests/test_repartition.py       |   211 +
 python/pyspark/sql/tests/test_resources.py         |   100 +
 python/pyspark/sql/tests/test_serde.py             |   156 +
 python/pyspark/sql/tests/test_session.py           |   761 +
 python/pyspark/sql/tests/test_sql.py               |   184 +
 python/pyspark/sql/tests/test_stat.py              |   672 +
 python/pyspark/sql/tests/test_subquery.py          |  1206 +
 python/pyspark/sql/tests/test_tvf.py               |   655 +
 python/pyspark/sql/tests/test_types.py             |  3262 ++
 python/pyspark/sql/tests/test_udf.py               |  1784 ++
 python/pyspark/sql/tests/test_udf_combinations.py  |   202 +
 python/pyspark/sql/tests/test_udf_profiler.py      |   791 +
 python/pyspark/sql/tests/test_udtf.py              |  3999 +++
 python/pyspark/sql/tests/test_unified_udf.py       |   465 +
 python/pyspark/sql/tests/test_utils.py             |  1802 ++
 python/pyspark/sql/tests/typing/test_column.yml    |    37 +
 python/pyspark/sql/tests/typing/test_dataframe.yml |   140 +
 python/pyspark/sql/tests/typing/test_functions.yml |   101 +
 .../pyspark/sql/tests/typing/test_readwriter.yml   |    45 +
 python/pyspark/sql/tests/typing/test_session.yml   |   100 +
 python/pyspark/sql/tests/typing/test_udf.yml       |   170 +
 python/pyspark/sql/tvf.py                          |   751 +
 python/pyspark/sql/tvf_argument.py                 |    27 +
 python/pyspark/sql/types.py                        |  3876 +++
 python/pyspark/sql/udf.py                          |   821 +
 python/pyspark/sql/udtf.py                         |   573 +
 python/pyspark/sql/utils.py                        |   515 +
 python/pyspark/sql/variant_utils.py                |   821 +
 python/pyspark/sql/window.py                       |   425 +
 python/pyspark/sql/worker/__init__.py              |    16 +
 python/pyspark/sql/worker/analyze_udtf.py          |   250 +
 .../pyspark/sql/worker/commit_data_source_write.py |    89 +
 python/pyspark/sql/worker/create_data_source.py    |   157 +
 .../sql/worker/data_source_pushdown_filters.py     |   235 +
 python/pyspark/sql/worker/lookup_data_sources.py   |    71 +
 python/pyspark/sql/worker/plan_data_source_read.py |   387 +
 .../sql/worker/python_streaming_sink_runner.py     |   124 +
 python/pyspark/sql/worker/utils.py                 |    70 +
 .../pyspark/sql/worker/write_into_data_source.py   |   252 +
 python/pyspark/statcounter.py                      |   165 +
 python/pyspark/storagelevel.py                     |    97 +
 python/pyspark/streaming/__init__.py               |    22 +
 python/pyspark/streaming/context.py                |   471 +
 python/pyspark/streaming/dstream.py                |   931 +
 python/pyspark/streaming/kinesis.py                |   205 +
 python/pyspark/streaming/listener.py               |    83 +
 python/pyspark/streaming/tests/__init__.py         |    16 +
 python/pyspark/streaming/tests/test_context.py     |   176 +
 python/pyspark/streaming/tests/test_dstream.py     |   698 +
 python/pyspark/streaming/tests/test_kinesis.py     |   112 +
 python/pyspark/streaming/tests/test_listener.py    |   152 +
 python/pyspark/streaming/util.py                   |   168 +
 python/pyspark/taskcontext.py                      |   544 +
 python/pyspark/testing/__init__.py                 |    21 +
 python/pyspark/testing/connectutils.py             |   338 +
 python/pyspark/testing/mllibutils.py               |    30 +
 python/pyspark/testing/mlutils.py                  |   279 +
 python/pyspark/testing/objects.py                  |   121 +
 python/pyspark/testing/pandasutils.py              |   649 +
 python/pyspark/testing/sqlutils.py                 |   232 +
 python/pyspark/testing/streamingutils.py           |   177 +
 python/pyspark/testing/tests/__init__.py           |    16 +
 python/pyspark/testing/tests/test_fail.py          |    36 +
 .../testing/tests/test_fail_in_set_up_class.py     |    41 +
 python/pyspark/testing/tests/test_no_tests.py      |    23 +
 python/pyspark/testing/tests/test_pass_all.py      |    28 +
 python/pyspark/testing/tests/test_skip_all.py      |    29 +
 python/pyspark/testing/tests/test_skip_class.py    |    30 +
 .../testing/tests/test_skip_set_up_class.py        |    33 +
 python/pyspark/testing/unittestutils.py            |    55 +
 python/pyspark/testing/utils.py                    |  1171 +
 python/pyspark/tests/__init__.py                   |    16 +
 python/pyspark/tests/test_appsubmit.py             |   302 +
 python/pyspark/tests/test_broadcast.py             |   190 +
 python/pyspark/tests/test_conf.py                  |    38 +
 python/pyspark/tests/test_context.py               |   341 +
 python/pyspark/tests/test_daemon.py                |    83 +
 python/pyspark/tests/test_install_spark.py         |   118 +
 python/pyspark/tests/test_join.py                  |    62 +
 python/pyspark/tests/test_memory_profiler.py       |   621 +
 python/pyspark/tests/test_pin_thread.py            |   186 +
 python/pyspark/tests/test_profiler.py              |   167 +
 python/pyspark/tests/test_rdd.py                   |   932 +
 python/pyspark/tests/test_rddbarrier.py            |    45 +
 python/pyspark/tests/test_rddsampler.py            |    59 +
 python/pyspark/tests/test_readwrite.py             |   362 +
 python/pyspark/tests/test_serializers.py           |   251 +
 python/pyspark/tests/test_shuffle.py               |   261 +
 python/pyspark/tests/test_stage_sched.py           |   147 +
 python/pyspark/tests/test_statcounter.py           |   124 +
 python/pyspark/tests/test_taskcontext.py           |   360 +
 python/pyspark/tests/test_util.py                  |   206 +
 python/pyspark/tests/test_worker.py                |   278 +
 python/pyspark/tests/typing/test_context.yml       |    21 +
 python/pyspark/tests/typing/test_core.yml          |    20 +
 python/pyspark/tests/typing/test_rdd.yml           |   129 +
 .../pyspark/tests/typing/test_resultiterable.yml   |    22 +
 python/pyspark/tests/upstream/__init__.py          |    16 +
 python/pyspark/tests/upstream/pyarrow/__init__.py  |    16 +
 .../upstream/pyarrow/test_pyarrow_array_cast.py    |  8977 ++++++
 .../pyarrow/test_pyarrow_array_type_inference.py   |   590 +
 .../pyarrow/test_pyarrow_ignore_timezone.py        |   187 +
 .../pyarrow/test_pyarrow_scalar_type_coercion.py   |   391 +
 .../pyarrow/test_pyarrow_scalar_type_inference.py  |   492 +
 .../upstream/pyarrow/test_pyarrow_type_coercion.py |  1158 +
 python/pyspark/threaddump.py                       |    62 +
 python/pyspark/traceback_utils.py                  |    81 +
 python/pyspark/util.py                             |  1029 +
 python/pyspark/version.py                          |    19 +
 python/pyspark/worker.py                           |  3481 +++
 python/pyspark/worker_util.py                      |   196 +
 python/run-tests                                   |    30 +
 python/run-tests-with-coverage                     |    66 +
 python/run-tests.py                                |   549 +
 python/run-with-viztracer                          |    69 +
 python/run-with-vscode-breakpoints                 |    79 +
 python/test_coverage/sitecustomize.py              |    70 +
 python/test_support/SimpleHTTPServer.py            |    22 +
 python/test_support/hello/hello.txt                |     1 +
 python/test_support/hello/sub_hello/sub_hello.txt  |     1 +
 python/test_support/log4j2.properties              |    31 +
 python/test_support/sql/ages_newlines.csv          |     6 +
 python/test_support/sql/orc_partitioned/_SUCCESS   |     0
 ...00-829af031-b970-49d6-ad39-30460a0be2c8.orc.crc |   Bin 0 -> 12 bytes
 ...-00000-829af031-b970-49d6-ad39-30460a0be2c8.orc |   Bin 0 -> 168 bytes
 ...00-829af031-b970-49d6-ad39-30460a0be2c8.orc.crc |   Bin 0 -> 12 bytes
 ...-00000-829af031-b970-49d6-ad39-30460a0be2c8.orc |   Bin 0 -> 168 bytes
 python/test_support/sql/people.json                |     3 +
 python/test_support/sql/people1.json               |     2 +
 python/test_support/sql/people_array.json          |    13 +
 python/test_support/sql/people_array_utf16le.json  |   Bin 0 -> 182 bytes
 python/test_support/sql/streaming/text-test.txt    |     2 +
 .../sql/streaming/time/text-with-time-test.txt     |    10 +
 python/test_support/sql/text-test.txt              |     2 +
 python/test_support/test_pytorch_training_file.py  |   115 +
 python/test_support/userlib-0.1.zip                |   Bin 0 -> 668 bytes
 python/test_support/userlibrary.py                 |    26 +
 repl/pom.xml                                       |   145 +
 .../main/scala/org/apache/spark/repl/Main.scala    |   142 +
 .../scala/org/apache/spark/repl/Signaling.scala    |    44 +
 .../scala/org/apache/spark/repl/SparkILoop.scala   |   158 +
 repl/src/test/resources/log4j2.properties          |    31 +
 .../scala/org/apache/spark/repl/IntSumUdf.scala    |    24 +
 .../scala/org/apache/spark/repl/ReplSuite.scala    |   488 +
 .../org/apache/spark/repl/SingletonReplSuite.scala |   480 +
 .../org/apache/spark/repl/SparkShellSuite.scala    |   165 +
 resource-managers/kubernetes/core/pom.xml          |   186 +
 .../org.apache.spark.deploy.SparkDiagnosticsSetter |    18 +
 .../org.apache.spark.deploy.SparkSubmitOperation   |    18 +
 ...g.apache.spark.scheduler.ExternalClusterManager |    18 +
 .../scala/org/apache/spark/deploy/k8s/Config.scala |   853 +
 .../org/apache/spark/deploy/k8s/Constants.scala    |   138 +
 .../apache/spark/deploy/k8s/KubernetesConf.scala   |   335 +
 .../spark/deploy/k8s/KubernetesDriverSpec.scala    |    36 +
 .../spark/deploy/k8s/KubernetesExecutorSpec.scala  |    23 +
 .../apache/spark/deploy/k8s/KubernetesUtils.scala  |   417 +
 .../spark/deploy/k8s/KubernetesVolumeSpec.scala    |    48 +
 .../spark/deploy/k8s/KubernetesVolumeUtils.scala   |   164 +
 .../deploy/k8s/SparkKubernetesClientFactory.scala  |   144 +
 .../k8s/SparkKubernetesDiagnosticsSetter.scala     |    80 +
 .../org/apache/spark/deploy/k8s/SparkPod.scala     |    66 +
 .../k8s/features/BasicDriverFeatureStep.scala      |   196 +
 .../k8s/features/BasicExecutorFeatureStep.scala    |   318 +
 .../k8s/features/DriverCommandFeatureStep.scala    |   121 +
 .../DriverKubernetesCredentialsFeatureStep.scala   |   211 +
 .../k8s/features/DriverServiceFeatureStep.scala    |   107 +
 .../k8s/features/EnvSecretsFeatureStep.scala       |    52 +
 .../ExecutorKubernetesCredentialsFeatureStep.scala |    41 +
 .../k8s/features/HadoopConfDriverFeatureStep.scala |   133 +
 .../features/HadoopConfExecutorFeatureStep.scala   |    63 +
 .../features/KerberosConfDriverFeatureStep.scala   |   270 +
 .../KubernetesDriverCustomFeatureConfigStep.scala  |    78 +
 ...KubernetesExecutorCustomFeatureConfigStep.scala |    78 +
 .../k8s/features/KubernetesFeatureConfigStep.scala |    84 +
 .../deploy/k8s/features/LocalDirsFeatureStep.scala |    91 +
 .../k8s/features/MountSecretsFeatureStep.scala     |    57 +
 .../k8s/features/MountVolumesFeatureStep.scala     |   146 +
 .../k8s/features/PodTemplateConfigMapStep.scala    |    95 +
 .../spark/deploy/k8s/submit/K8sSubmitOps.scala     |   176 +
 .../k8s/submit/KubernetesClientApplication.scala   |   261 +
 .../deploy/k8s/submit/KubernetesClientUtils.scala  |   225 +
 .../k8s/submit/KubernetesDriverBuilder.scala       |   110 +
 .../k8s/submit/LoggingPodStatusWatcher.scala       |   120 +
 .../spark/deploy/k8s/submit/MainAppResource.scala  |    53 +
 .../cluster/k8s/AbstractPodsAllocator.scala        |    58 +
 .../cluster/k8s/DeploymentPodsAllocator.scala      |   200 +
 .../scheduler/cluster/k8s/ExecutorPodStates.scala  |    39 +
 .../cluster/k8s/ExecutorPodsAllocator.scala        |   555 +
 .../cluster/k8s/ExecutorPodsLifecycleManager.scala |   330 +
 .../k8s/ExecutorPodsPollingSnapshotSource.scala    |    97 +
 .../cluster/k8s/ExecutorPodsSnapshot.scala         |   135 +
 .../cluster/k8s/ExecutorPodsSnapshotsStore.scala   |    34 +
 .../k8s/ExecutorPodsSnapshotsStoreImpl.scala       |   172 +
 .../k8s/ExecutorPodsWatchSnapshotSource.scala      |   107 +
 .../scheduler/cluster/k8s/ExecutorRollPlugin.scala |   189 +
 .../cluster/k8s/KubernetesClusterManager.scala     |   199 +
 .../cluster/k8s/KubernetesClusterMessage.scala     |    21 +
 .../k8s/KubernetesClusterSchedulerBackend.scala    |   399 +
 .../cluster/k8s/KubernetesExecutorBackend.scala    |   235 +
 .../cluster/k8s/KubernetesExecutorBuilder.scala    |    95 +
 .../cluster/k8s/StatefulSetPodsAllocator.scala     |   208 +
 .../shuffle/KubernetesLocalDiskShuffleDataIO.scala |    37 +
 ...ernetesLocalDiskShuffleExecutorComponents.scala |   179 +
 .../test/resources/driver-podgroup-template.yml    |    25 +
 .../core/src/test/resources/log4j2.properties      |    39 +
 .../apache/spark/deploy/k8s/Fabric8Aliases.scala   |    50 +
 .../spark/deploy/k8s/KubernetesConfSuite.scala     |   289 +
 .../spark/deploy/k8s/KubernetesTestConf.scala      |   165 +
 .../spark/deploy/k8s/KubernetesUtilsSuite.scala    |   163 +
 .../deploy/k8s/KubernetesVolumeUtilsSuite.scala    |   319 +
 .../apache/spark/deploy/k8s/PodBuilderSuite.scala  |   370 +
 .../spark/deploy/k8s/SecretVolumeUtils.scala       |    36 +
 .../SparkKubernetesDiagnosticsSetterSuite.scala    |    89 +
 .../k8s/features/BasicDriverFeatureStepSuite.scala |   475 +
 .../features/BasicExecutorFeatureStepSuite.scala   |   694 +
 .../features/DriverCommandFeatureStepSuite.scala   |   181 +
 ...iverKubernetesCredentialsFeatureStepSuite.scala |   135 +
 .../features/DriverServiceFeatureStepSuite.scala   |   270 +
 .../k8s/features/EnvSecretsFeatureStepSuite.scala  |    47 +
 ...utorKubernetesCredentialsFeatureStepSuite.scala |    70 +
 .../HadoopConfDriverFeatureStepSuite.scala         |    69 +
 .../HadoopConfExecutorFeatureStepSuite.scala       |    80 +
 .../KerberosConfDriverFeatureStepSuite.scala       |   170 +
 .../k8s/features/KubernetesFeaturesTestUtils.scala |    72 +
 .../k8s/features/LocalDirsFeatureStepSuite.scala   |   161 +
 .../features/MountSecretsFeatureStepSuite.scala    |    48 +
 .../features/MountVolumesFeatureStepSuite.scala    |   576 +
 .../features/PodTemplateConfigMapStepSuite.scala   |    90 +
 .../spark/deploy/k8s/submit/ClientSuite.scala      |   379 +
 .../spark/deploy/k8s/submit/K8sSubmitOpSuite.scala |   170 +
 .../k8s/submit/KubernetesClientUtilsSuite.scala    |   131 +
 .../k8s/submit/KubernetesDriverBuilderSuite.scala  |   127 +
 .../cluster/k8s/DeploymentAllocatorSuite.scala     |   190 +
 .../DeterministicExecutorPodsSnapshotsStore.scala  |    69 +
 .../cluster/k8s/ExecutorLifecycleTestUtils.scala   |   269 +
 .../cluster/k8s/ExecutorPodsAllocatorSuite.scala   |  1030 +
 .../k8s/ExecutorPodsLifecycleManagerSuite.scala    |   289 +
 .../ExecutorPodsPollingSnapshotSourceSuite.scala   |   122 +
 .../cluster/k8s/ExecutorPodsSnapshotSuite.scala    |    87 +
 .../k8s/ExecutorPodsSnapshotsStoreSuite.scala      |   145 +
 .../k8s/ExecutorPodsWatchSnapshotSourceSuite.scala |    98 +
 .../cluster/k8s/ExecutorRollPluginSuite.scala      |   417 +
 .../k8s/KubernetesClusterManagerSuite.scala        |   120 +
 .../KubernetesClusterSchedulerBackendSuite.scala   |   309 +
 .../k8s/KubernetesExecutorBuilderSuite.scala       |    91 +
 .../cluster/k8s/StatefulSetAllocatorSuite.scala    |   166 +
 .../KubernetesLocalDiskShuffleDataIOSuite.scala    |   255 +
 .../spark/shuffle/ShuffleChecksumUtilsSuite.scala  |   134 +
 .../deploy/k8s/features/VolcanoFeatureStep.scala   |    98 +
 .../k8s/features/VolcanoFeatureStepSuite.scala     |   104 +
 .../docker/src/main/dockerfiles/spark/Dockerfile   |    71 +
 .../main/dockerfiles/spark/bindings/R/Dockerfile   |    47 +
 .../dockerfiles/spark/bindings/python/Dockerfile   |    46 +
 .../docker/src/main/dockerfiles/spark/decom.sh     |    40 +
 .../src/main/dockerfiles/spark/entrypoint.sh       |   118 +
 .../kubernetes/integration-tests/README.md         |   365 +
 .../dev/dev-run-integration-tests.sh               |   214 +
 .../integration-tests/dev/spark-rbac.yaml          |    52 +
 .../kubernetes/integration-tests/pom.xml           |   234 +
 .../scripts/setup-integration-test-env.sh          |   169 +
 .../deploy/k8s/integrationtest/YuniKornTag.java    |    27 +
 .../test/resources/driver-schedule-template.yml    |    27 +
 .../src/test/resources/driver-template.yml         |    26 +
 .../src/test/resources/executor-template.yml       |    25 +
 .../resources/log-config-test-log4j.properties     |    26 +
 .../src/test/resources/log4j2.properties           |    39 +
 .../src/test/resources/pagerank_data.txt           |     6 +
 .../volcano/driver-podgroup-template-memory-3g.yml |    23 +
 .../high-priority-driver-podgroup-template.yml     |    21 +
 .../volcano/high-priority-driver-template.yml      |    26 +
 .../low-priority-driver-podgroup-template.yml      |    21 +
 .../volcano/low-priority-driver-template.yml       |    26 +
 .../medium-priority-driver-podgroup-template.yml   |    21 +
 .../volcano/medium-priority-driver-template.yml    |    26 +
 .../src/test/resources/volcano/priorityClasses.yml |    33 +
 .../volcano/queue-driver-podgroup-template.yml     |    20 +
 .../volcano/queue0-driver-podgroup-template.yml    |    20 +
 .../volcano/queue1-driver-podgroup-template.yml    |    20 +
 .../k8s/integrationtest/BasicTestsSuite.scala      |   197 +
 .../k8s/integrationtest/ClientModeTestsSuite.scala |   140 +
 .../k8s/integrationtest/DecommissionSuite.scala    |   258 +
 .../k8s/integrationtest/DepsTestsSuite.scala       |   412 +
 .../k8s/integrationtest/KubernetesSuite.scala      |   658 +
 .../integrationtest/KubernetesTestComponents.scala |   140 +
 .../deploy/k8s/integrationtest/PVTestsSuite.scala  |   225 +
 .../k8s/integrationtest/PodTemplateSuite.scala     |    81 +
 .../deploy/k8s/integrationtest/ProcessUtils.scala  |    58 +
 .../k8s/integrationtest/PythonTestsSuite.scala     |    97 +
 .../deploy/k8s/integrationtest/RTestsSuite.scala   |    40 +
 .../k8s/integrationtest/SecretsTestsSuite.scala    |   130 +
 .../integrationtest/SparkConfPropagateSuite.scala  |    61 +
 .../integrationtest/SparkReadinessWatcher.scala    |    44 +
 .../deploy/k8s/integrationtest/TestConstants.scala |    41 +
 .../spark/deploy/k8s/integrationtest/Utils.scala   |   177 +
 .../deploy/k8s/integrationtest/VolumeSuite.scala   |   183 +
 .../deploy/k8s/integrationtest/YuniKornSuite.scala |    32 +
 .../backend/IntegrationTestBackend.scala           |    52 +
 .../backend/cloud/KubeConfigBackend.scala          |    71 +
 .../backend/docker/DockerForDesktopBackend.scala   |    25 +
 .../backend/docker/RancherDesktopBackend.scala     |    25 +
 .../backend/minikube/Minikube.scala                |   132 +
 .../backend/minikube/MinikubeTestBackend.scala     |    50 +
 .../integration-tests/tests/autoscale.py           |    49 +
 .../integration-tests/tests/decommissioning.py     |    55 +
 .../tests/decommissioning_cleanup.py               |    59 +
 .../integration-tests/tests/py_container_checks.py |    32 +
 .../kubernetes/integration-tests/tests/pyfiles.py  |    43 +
 .../integration-tests/tests/pyfiles_connect.py     |    43 +
 .../tests/python_executable_check.py               |    40 +
 .../integration-tests/tests/worker_memory_check.py |    45 +
 .../deploy/k8s/integrationtest/VolcanoSuite.scala  |    33 +
 .../k8s/integrationtest/VolcanoTestsSuite.scala    |   503 +
 resource-managers/yarn/pom.xml                     |   160 +
 .../org/apache/spark/deploy/yarn/AmIpFilter.java   |   239 +
 .../apache/spark/deploy/yarn/AmIpPrincipal.java    |    35 +
 .../deploy/yarn/AmIpServletRequestWrapper.java     |    54 +
 .../org/apache/spark/deploy/yarn/ProxyUtils.java   |   126 +
 ...g.apache.spark.scheduler.ExternalClusterManager |    18 +
 .../org/apache/spark/deploy/yarn/config.properties |     1 +
 .../spark/deploy/yarn/ApplicationMaster.scala      |   976 +
 .../deploy/yarn/ApplicationMasterArguments.scala   |   104 +
 .../deploy/yarn/ApplicationMasterSource.scala      |    50 +
 .../org/apache/spark/deploy/yarn/Client.scala      |  1835 ++
 .../apache/spark/deploy/yarn/ClientArguments.scala |    99 +
 .../yarn/ClientDistributedCacheManager.scala       |   201 +
 .../spark/deploy/yarn/ExecutorRunnable.scala       |   231 +
 ...calityPreferredContainerPlacementStrategy.scala |   230 +
 .../spark/deploy/yarn/ResourceRequestHelper.scala  |   189 +
 .../spark/deploy/yarn/SparkRackResolver.scala      |   113 +
 .../apache/spark/deploy/yarn/YarnAllocator.scala   |  1064 +
 .../yarn/YarnAllocatorNodeHealthTracker.scala      |   152 +
 .../deploy/yarn/YarnProxyRedirectFilter.scala      |    81 +
 .../apache/spark/deploy/yarn/YarnRMClient.scala    |   153 +
 .../spark/deploy/yarn/YarnSparkHadoopUtil.scala    |   212 +
 .../apache/spark/deploy/yarn/config/package.scala  |   507 +
 .../YarnCoarseGrainedExecutorBackend.scala         |    88 +
 .../spark/launcher/YarnCommandBuilderUtils.scala   |    39 +
 .../cluster/YarnClientSchedulerBackend.scala       |   189 +
 .../scheduler/cluster/YarnClusterManager.scala     |    56 +
 .../scheduler/cluster/YarnClusterScheduler.scala   |    37 +
 .../cluster/YarnClusterSchedulerBackend.scala      |    50 +
 .../spark/scheduler/cluster/YarnScheduler.scala    |    39 +
 .../scheduler/cluster/YarnSchedulerBackend.scala   |   411 +
 .../spark/util/YarnContainerInfoHelper.scala       |   132 +
 .../yarn/src/test/resources/certchain.pem          |    17 +
 resource-managers/yarn/src/test/resources/key.pem  |    29 +
 resource-managers/yarn/src/test/resources/keystore |   Bin 0 -> 2247 bytes
 .../yarn/src/test/resources/log4j2.properties      |    39 +
 .../yarn/src/test/resources/truststore             |   Bin 0 -> 957 bytes
 .../src/test/resources/unencrypted-certchain.pem   |    21 +
 .../yarn/src/test/resources/unencrypted-key.pem    |    28 +
 .../yarn/src/test/resources/untrusted-keystore     |   Bin 0 -> 2246 bytes
 .../apache/spark/deploy/yarn/AmIpFilterSuite.scala |   338 +
 .../spark/deploy/yarn/ApplicationMasterSuite.scala |    44 +
 .../spark/deploy/yarn/BaseYarnClusterSuite.scala   |   297 +
 .../yarn/ClientDistributedCacheManagerSuite.scala  |   258 +
 .../org/apache/spark/deploy/yarn/ClientSuite.scala |   834 +
 .../yarn/ContainerPlacementStrategySuite.scala     |   175 +
 .../spark/deploy/yarn/ExecutorRunnableSuite.scala  |   106 +
 .../yarn/LocalityPlacementStrategySuite.scala      |    97 +
 .../deploy/yarn/ResourceRequestHelperSuite.scala   |   185 +
 .../deploy/yarn/ResourceRequestTestHelper.scala    |    52 +
 .../yarn/YarnAllocatorHealthTrackerSuite.scala     |   169 +
 .../spark/deploy/yarn/YarnAllocatorSuite.scala     |   907 +
 .../spark/deploy/yarn/YarnClusterSuite.scala       |   905 +
 .../deploy/yarn/YarnProxyRedirectFilterSuite.scala |    55 +
 .../yarn/YarnShuffleAlternateNameConfigSuite.scala |    91 +
 .../deploy/yarn/YarnShuffleIntegrationSuite.scala  |   187 +
 .../deploy/yarn/YarnSparkHadoopUtilSuite.scala     |   181 +
 .../spark/launcher/TestClasspathBuilder.scala      |    36 +
 .../network/shuffle/ShuffleTestAccessor.scala      |   227 +
 .../network/yarn/SslYarnShuffleServiceSuite.scala  |    34 +
 .../yarn/YarnShuffleServiceMetricsSuite.scala      |   104 +
 .../network/yarn/YarnShuffleServiceSuite.scala     |  1267 +
 .../spark/network/yarn/YarnTestAccessor.scala      |    44 +
 .../cluster/YarnSchedulerBackendSuite.scala        |   137 +
 sbin/decommission-worker.sh                        |    57 +
 sbin/spark-config.sh                               |    33 +
 sbin/spark-daemon.sh                               |   268 +
 sbin/spark-daemons.sh                              |    36 +
 sbin/start-all.sh                                  |    35 +
 sbin/start-connect-server.sh                       |    46 +
 sbin/start-history-server.sh                       |    49 +
 sbin/start-master.sh                               |    68 +
 sbin/start-thriftserver.sh                         |    61 +
 sbin/start-worker.sh                               |    92 +
 sbin/start-workers.sh                              |    46 +
 sbin/stop-all.sh                                   |    49 +
 sbin/stop-connect-server.sh                        |    26 +
 sbin/stop-history-server.sh                        |    26 +
 sbin/stop-master.sh                                |    28 +
 sbin/stop-thriftserver.sh                          |    26 +
 sbin/stop-worker.sh                                |    44 +
 sbin/stop-workers.sh                               |    28 +
 sbin/workers.sh                                    |   120 +
 scalastyle-config.xml                              |   854 +
 sql/README.md                                      |    13 +
 sql/api/pom.xml                                    |   110 +
 .../spark/sql/catalyst/parser/SqlBaseLexer.g4      |   696 +
 .../spark/sql/catalyst/parser/SqlBaseParser.g4     |  2635 ++
 .../function/FlatMapGroupsWithStateFunction.java   |    37 +
 .../java/function/MapGroupsWithStateFunction.java  |    36 +
 .../main/java/org/apache/spark/sql/RowFactory.java |    40 +
 .../main/java/org/apache/spark/sql/SaveMode.java   |    58 +
 .../java/org/apache/spark/sql/api/java/UDF0.java   |    30 +
 .../java/org/apache/spark/sql/api/java/UDF1.java   |    30 +
 .../java/org/apache/spark/sql/api/java/UDF10.java  |    30 +
 .../java/org/apache/spark/sql/api/java/UDF11.java  |    30 +
 .../java/org/apache/spark/sql/api/java/UDF12.java  |    30 +
 .../java/org/apache/spark/sql/api/java/UDF13.java  |    30 +
 .../java/org/apache/spark/sql/api/java/UDF14.java  |    30 +
 .../java/org/apache/spark/sql/api/java/UDF15.java  |    30 +
 .../java/org/apache/spark/sql/api/java/UDF16.java  |    30 +
 .../java/org/apache/spark/sql/api/java/UDF17.java  |    30 +
 .../java/org/apache/spark/sql/api/java/UDF18.java  |    30 +
 .../java/org/apache/spark/sql/api/java/UDF19.java  |    30 +
 .../java/org/apache/spark/sql/api/java/UDF2.java   |    30 +
 .../java/org/apache/spark/sql/api/java/UDF20.java  |    30 +
 .../java/org/apache/spark/sql/api/java/UDF21.java  |    30 +
 .../java/org/apache/spark/sql/api/java/UDF22.java  |    30 +
 .../java/org/apache/spark/sql/api/java/UDF3.java   |    30 +
 .../java/org/apache/spark/sql/api/java/UDF4.java   |    30 +
 .../java/org/apache/spark/sql/api/java/UDF5.java   |    30 +
 .../java/org/apache/spark/sql/api/java/UDF6.java   |    30 +
 .../java/org/apache/spark/sql/api/java/UDF7.java   |    30 +
 .../java/org/apache/spark/sql/api/java/UDF8.java   |    30 +
 .../java/org/apache/spark/sql/api/java/UDF9.java   |    30 +
 .../apache/spark/sql/api/java/package-info.java    |    21 +
 .../spark/sql/connector/catalog/Identifier.java    |    43 +
 .../sql/connector/catalog/IdentifierImpl.java      |    74 +
 .../sql/connector/catalog/IdentityColumnSpec.java  |    88 +
 .../spark/sql/expressions/javalang/typed.java      |    74 +
 .../CartesianSpatialReferenceSystemMapper.java     |    38 +
 .../GeographicSpatialReferenceSystemMapper.java    |    38 +
 .../types/SpatialReferenceSystemCache.java         |    87 +
 .../types/SpatialReferenceSystemInformation.java   |    33 +
 .../types/SpatialReferenceSystemMapper.java        |    30 +
 .../spark/sql/streaming/GroupStateTimeout.java     |    68 +
 .../org/apache/spark/sql/streaming/OutputMode.java |    64 +
 .../org/apache/spark/sql/streaming/TimeMode.java   |    48 +
 .../org/apache/spark/sql/streaming/Trigger.java    |   233 +
 .../java/org/apache/spark/sql/types/DataTypes.java |   316 +
 .../java/org/apache/spark/sql/types/Geography.java |    80 +
 .../java/org/apache/spark/sql/types/Geometry.java  |    80 +
 .../apache/spark/sql/types/SQLUserDefinedType.java |    40 +
 .../resources/gregorian-julian-rebase-micros.json  |  2409 ++
 .../resources/julian-gregorian-rebase-micros.json  |  2409 ++
 .../org/apache/spark/sql/AnalysisException.scala   |   228 +
 .../main/scala/org/apache/spark/sql/Artifact.scala |   157 +
 .../main/scala/org/apache/spark/sql/Column.scala   |  1558 +
 .../apache/spark/sql/DataFrameNaFunctions.scala    |   361 +
 .../org/apache/spark/sql/DataFrameReader.scala     |   603 +
 .../apache/spark/sql/DataFrameStatFunctions.scala  |   585 +
 .../org/apache/spark/sql/DataFrameWriter.scala     |   518 +
 .../org/apache/spark/sql/DataFrameWriterV2.scala   |   251 +
 .../main/scala/org/apache/spark/sql/Dataset.scala  |  3301 +++
 .../scala/org/apache/spark/sql/DatasetHolder.scala |    45 +
 .../main/scala/org/apache/spark/sql/Encoder.scala  |    78 +
 .../main/scala/org/apache/spark/sql/Encoders.scala |   392 +
 .../scala/org/apache/spark/sql/ForeachWriter.scala |   130 +
 .../apache/spark/sql/KeyValueGroupedDataset.scala  |  1022 +
 .../org/apache/spark/sql/MergeIntoWriter.scala     |   325 +
 .../scala/org/apache/spark/sql/Observation.scala   |   162 +
 .../spark/sql/RelationalGroupedDataset.scala       |   359 +
 .../src/main/scala/org/apache/spark/sql/Row.scala  |   688 +
 .../scala/org/apache/spark/sql/RuntimeConfig.scala |   140 +
 .../scala/org/apache/spark/sql/SQLContext.scala    |  1030 +
 .../scala/org/apache/spark/sql/SQLImplicits.scala  |   318 +
 .../scala/org/apache/spark/sql/SparkSession.scala  |  1288 +
 .../org/apache/spark/sql/TableValuedFunction.scala |   192 +
 .../org/apache/spark/sql/UDFRegistration.scala     |  1364 +
 .../org/apache/spark/sql/avro/functions.scala      |   125 +
 .../org/apache/spark/sql/catalog/Catalog.scala     |   696 +
 .../org/apache/spark/sql/catalog/interface.scala   |   253 +
 .../sql/catalyst/DefinedByConstructorParams.scala  |    24 +
 .../spark/sql/catalyst/JavaTypeInference.scala     |   188 +
 .../spark/sql/catalyst/ScalaReflection.scala       |   536 +
 .../apache/spark/sql/catalyst/WalkedTypePath.scala |    58 +
 .../sql/catalyst/analysis/NonEmptyException.scala  |    36 +
 .../sql/catalyst/analysis/SqlApiAnalysis.scala     |    26 +
 .../catalyst/analysis/alreadyExistException.scala  |   199 +
 .../catalyst/analysis/noSuchItemsExceptions.scala  |   222 +
 .../sql/catalyst/encoders/AgnosticEncoder.scala    |   306 +
 .../spark/sql/catalyst/encoders/OuterScopes.scala  |   183 +
 .../spark/sql/catalyst/encoders/RowEncoder.scala   |   131 +
 .../spark/sql/catalyst/encoders/codecs.scala       |    73 +
 .../sql/catalyst/expressions/OrderUtils.scala      |    35 +
 .../spark/sql/catalyst/expressions/rows.scala      |    51 +
 .../sql/catalyst/parser/DataTypeAstBuilder.scala   |   595 +
 .../catalyst/parser/DataTypeParserInterface.scala  |    38 +
 .../catalyst/parser/LegacyTypeStringParser.scala   |    96 +
 .../spark/sql/catalyst/parser/PositionMapper.scala |   172 +
 .../catalyst/parser/SparkParserErrorStrategy.scala |   155 +
 .../parser/SubstituteParmsAstBuilder.scala         |   251 +
 .../apache/spark/sql/catalyst/parser/parsers.scala |   621 +
 .../catalyst/plans/logical/LogicalGroupState.scala |    20 +
 .../sql/catalyst/plans/logical/TimeMode.scala      |    52 +
 .../plans/logical/groupStateTimeouts.scala         |    24 +
 .../catalyst/streaming/InternalOutputModes.scala   |    65 +
 .../spark/sql/catalyst/trees/QueryContexts.scala   |   179 +
 .../apache/spark/sql/catalyst/trees/origin.scala   |   179 +
 .../sql/catalyst/util/AttributeNameParser.scala    |    68 +
 .../sql/catalyst/util/CaseInsensitiveMap.scala     |    64 +
 .../sql/catalyst/util/DataTypeJsonUtils.scala      |    56 +
 .../spark/sql/catalyst/util/DateFormatter.scala    |   232 +
 .../catalyst/util/DateTimeFormatterHelper.scala    |   371 +
 .../apache/spark/sql/catalyst/util/MathUtils.scala |   110 +
 .../spark/sql/catalyst/util/QuotingUtils.scala     |    86 +
 .../spark/sql/catalyst/util/RebaseDateTime.scala   |   556 +
 .../catalyst/util/ResolveDefaultColumnsUtils.scala |    46 +
 .../sql/catalyst/util/SparkCharVarcharUtils.scala  |    63 +
 .../sql/catalyst/util/SparkDateTimeUtils.scala     |   828 +
 .../sql/catalyst/util/SparkIntervalUtils.scala     |   572 +
 .../spark/sql/catalyst/util/SparkParserUtils.scala |   270 +
 .../spark/sql/catalyst/util/StringConcat.scala     |    63 +
 .../spark/sql/catalyst/util/TimeFormatter.scala    |   142 +
 .../sql/catalyst/util/TimestampFormatter.scala     |   660 +
 .../apache/spark/sql/catalyst/util/UDTUtils.scala  |    61 +
 .../spark/sql/errors/CompilationErrors.scala       |   148 +
 .../apache/spark/sql/errors/DataTypeErrors.scala   |   278 +
 .../spark/sql/errors/DataTypeErrorsBase.scala      |   103 +
 .../apache/spark/sql/errors/ExecutionErrors.scala  |   252 +
 .../spark/sql/errors/QueryParsingErrors.scala      |   864 +
 .../spark/sql/execution/streaming/Triggers.scala   |   156 +
 .../apache/spark/sql/expressions/Aggregator.scala  |   101 +
 .../spark/sql/expressions/ReduceAggregator.scala   |    84 +
 .../sql/expressions/UserDefinedFunction.scala      |   206 +
 .../org/apache/spark/sql/expressions/Window.scala  |   241 +
 .../apache/spark/sql/expressions/WindowSpec.scala  |   216 +
 .../spark/sql/expressions/scalalang/typed.scala    |    69 +
 .../org/apache/spark/sql/expressions/udaf.scala    |   162 +
 .../scala/org/apache/spark/sql/functions.scala     | 10346 +++++++
 .../spark/sql/internal/LegacyBehaviorPolicy.scala  |    21 +
 .../org/apache/spark/sql/internal/SqlApiConf.scala |   113 +
 .../spark/sql/internal/SqlApiConfHelper.scala      |    58 +
 .../sql/internal/TableValuedFunctionArgument.scala |    20 +
 .../org/apache/spark/sql/internal/ToScalaUDF.scala |   846 +
 .../apache/spark/sql/internal/columnNodes.scala    |   675 +
 .../spark/sql/internal/typedaggregators.scala      |    98 +
 .../sql/internal/types/AbstractArrayType.scala     |    36 +
 .../spark/sql/internal/types/AbstractMapType.scala |    40 +
 .../sql/internal/types/AbstractStringType.scala    |   121 +
 .../main/scala/org/apache/spark/sql/package.scala  |    63 +
 .../org/apache/spark/sql/protobuf/functions.scala  |   298 +
 .../spark/sql/streaming/DataStreamReader.scala     |   316 +
 .../spark/sql/streaming/DataStreamWriter.scala     |   229 +
 .../spark/sql/streaming/ExpiredTimerInfo.scala     |    36 +
 .../apache/spark/sql/streaming/GroupState.scala    |   340 +
 .../org/apache/spark/sql/streaming/ListState.scala |    78 +
 .../org/apache/spark/sql/streaming/MapState.scala  |   108 +
 .../org/apache/spark/sql/streaming/QueryInfo.scala |    58 +
 .../spark/sql/streaming/StatefulProcessor.scala    |   159 +
 .../sql/streaming/StatefulProcessorHandle.scala    |   233 +
 .../sql/streaming/StreamingCheckpointManager.scala |    58 +
 .../spark/sql/streaming/StreamingQuery.scala       |   176 +
 .../sql/streaming/StreamingQueryListener.scala     |   310 +
 .../sql/streaming/StreamingQueryManager.scala      |   129 +
 .../spark/sql/streaming/StreamingQueryStatus.scala |    71 +
 .../org/apache/spark/sql/streaming/TTLConfig.scala |    45 +
 .../apache/spark/sql/streaming/TimerValues.scala   |    54 +
 .../apache/spark/sql/streaming/ValueState.scala    |    61 +
 .../org/apache/spark/sql/streaming/progress.scala  |   399 +
 .../apache/spark/sql/types/AbstractDataType.scala  |   191 +
 .../org/apache/spark/sql/types/ArrayType.scala     |   122 +
 .../org/apache/spark/sql/types/BinaryType.scala    |    41 +
 .../org/apache/spark/sql/types/BooleanType.scala   |    42 +
 .../org/apache/spark/sql/types/ByteType.scala      |    44 +
 .../spark/sql/types/CalendarIntervalType.scala     |    49 +
 .../org/apache/spark/sql/types/CharType.scala      |    65 +
 .../org/apache/spark/sql/types/DataType.scala      |   606 +
 .../org/apache/spark/sql/types/DateType.scala      |    49 +
 .../spark/sql/types/DayTimeIntervalType.scala      |   105 +
 .../scala/org/apache/spark/sql/types/Decimal.scala |   734 +
 .../org/apache/spark/sql/types/DecimalType.scala   |   249 +
 .../org/apache/spark/sql/types/DoubleType.scala    |    71 +
 .../org/apache/spark/sql/types/FloatType.scala     |    74 +
 .../org/apache/spark/sql/types/GeographyType.scala |   269 +
 .../org/apache/spark/sql/types/GeometryType.scala  |   229 +
 .../org/apache/spark/sql/types/IntegerType.scala   |    44 +
 .../org/apache/spark/sql/types/LongType.scala      |    44 +
 .../scala/org/apache/spark/sql/types/MapType.scala |   126 +
 .../org/apache/spark/sql/types/Metadata.scala      |   357 +
 .../org/apache/spark/sql/types/NullType.scala      |    43 +
 .../org/apache/spark/sql/types/ObjectType.scala    |    48 +
 .../org/apache/spark/sql/types/ShortType.scala     |    44 +
 .../org/apache/spark/sql/types/SpatialType.scala   |    31 +
 .../org/apache/spark/sql/types/StringType.scala    |   190 +
 .../org/apache/spark/sql/types/StructField.scala   |   275 +
 .../org/apache/spark/sql/types/StructType.scala    |   737 +
 .../org/apache/spark/sql/types/TimeType.scala      |    58 +
 .../apache/spark/sql/types/TimestampNTZType.scala  |    53 +
 .../org/apache/spark/sql/types/TimestampType.scala |    50 +
 .../apache/spark/sql/types/UDTRegistration.scala   |    96 +
 .../org/apache/spark/sql/types/UpCastRule.scala    |    82 +
 .../apache/spark/sql/types/UserDefinedType.scala   |   162 +
 .../org/apache/spark/sql/types/VarcharType.scala   |    64 +
 .../org/apache/spark/sql/types/VariantType.scala   |    43 +
 .../spark/sql/types/YearMonthIntervalType.scala    |    97 +
 .../org/apache/spark/sql/util/ArrowUtils.scala     |   376 +
 .../org/apache/spark/sql/util/ArtifactUtils.scala  |    59 +
 .../apache/spark/sql/util/CloseableIterator.scala  |    66 +
 .../sql/util/ConcatenatingArrowStreamReader.scala  |   187 +
 .../org/apache/spark/sql/util/ProtobufUtils.scala  |    39 +
 .../spark/sql/types/GeographyClientSuite.java      |    86 +
 .../spark/sql/types/GeometryClientSuite.java       |    86 +
 ...kSessionBuilderImplementationBindingSuite.scala |    87 +
 .../spark/sql/StaticProcedureSuiteBase.scala       |    33 +
 .../CalendarIntervalBenchmark-jdk21-results.txt    |    11 +
 .../CalendarIntervalBenchmark-results.txt          |    11 +
 .../EnumTypeSetBenchmark-jdk21-results.txt         |   105 +
 .../benchmarks/EnumTypeSetBenchmark-results.txt    |   105 +
 .../EscapePathBenchmark-jdk21-results.txt          |    24 +
 .../benchmarks/EscapePathBenchmark-results.txt     |    24 +
 .../GenericArrayDataBenchmark-jdk21-results.txt    |    10 +
 .../GenericArrayDataBenchmark-results.txt          |    10 +
 .../benchmarks/HashBenchmark-jdk21-results.txt     |    70 +
 sql/catalyst/benchmarks/HashBenchmark-results.txt  |    70 +
 .../HashByteArrayBenchmark-jdk21-results.txt       |    77 +
 .../benchmarks/HashByteArrayBenchmark-results.txt  |    77 +
 .../benchmarks/HexBenchmark-jdk21-results.txt      |    14 +
 sql/catalyst/benchmarks/HexBenchmark-results.txt   |    14 +
 ...RowComparableWrapperBenchmark-jdk21-results.txt |     7 +
 ...ternalRowComparableWrapperBenchmark-results.txt |     7 +
 .../SparkBloomFilterBenchmark-jdk21-results.txt    |   300 +
 .../SparkBloomFilterBenchmark-results.txt          |   300 +
 .../UnsafeProjectionBenchmark-jdk21-results.txt    |    14 +
 .../UnsafeProjectionBenchmark-results.txt          |    14 +
 sql/catalyst/pom.xml                               |   181 +
 .../catalyst/expressions/ArrayExpressionUtils.java |   208 +
 .../catalyst/expressions/ArrayOfDecimalsSerDe.java |   259 +
 .../expressions/BitmapExpressionUtils.java         |    72 +
 .../expressions/ExpressionDescription.java         |   120 +
 .../catalyst/expressions/ExpressionImplUtils.java  |   345 +
 .../sql/catalyst/expressions/ExpressionInfo.java   |   214 +
 .../FixedLengthRowBasedKeyValueBatch.java          |   176 +
 .../expressions/RowBasedKeyValueBatch.java         |   185 +
 .../catalyst/expressions/SpecializedGetters.java   |    70 +
 .../expressions/SpecializedGettersReader.java      |    97 +
 .../sql/catalyst/expressions/ToJavaArrayUtils.java |   112 +
 .../sql/catalyst/expressions/UnsafeArrayData.java  |   553 +
 .../sql/catalyst/expressions/UnsafeDataUtils.java  |    40 +
 .../sql/catalyst/expressions/UnsafeMapData.java    |   169 +
 .../spark/sql/catalyst/expressions/UnsafeRow.java  |   682 +
 .../catalyst/expressions/UnsafeRowChecksum.scala   |    53 +
 .../VariableLengthRowBasedKeyValueBatch.java       |   189 +
 .../expressions/VectorFunctionImplUtils.java       |   465 +
 .../spark/sql/catalyst/expressions/XXH64.java      |   206 +
 .../catalyst/expressions/codegen/BufferHolder.java |   116 +
 .../expressions/codegen/UnsafeArrayWriter.java     |   209 +
 .../expressions/codegen/UnsafeRowWriter.java       |   221 +
 .../catalyst/expressions/codegen/UnsafeWriter.java |   246 +
 .../expressions/json/JsonExpressionUtils.java      |    80 +
 .../sql/catalyst/expressions/xml/UDFXPathUtil.java |   222 +
 .../sql/catalyst/util/CharVarcharCodegenUtils.java |    67 +
 .../org/apache/spark/sql/catalyst/util/Geo.java    |    84 +
 .../apache/spark/sql/catalyst/util/Geography.java  |   181 +
 .../apache/spark/sql/catalyst/util/Geometry.java   |   188 +
 .../sql/catalyst/util/HadoopCompressionCodec.java  |    67 +
 .../apache/spark/sql/catalyst/util/STUtils.java    |   152 +
 .../spark/sql/catalyst/util/geo/GeoTypeId.java     |    54 +
 .../sql/catalyst/util/geo/GeometryCollection.java  |    49 +
 .../spark/sql/catalyst/util/geo/GeometryModel.java |   179 +
 .../spark/sql/catalyst/util/geo/LineString.java    |    64 +
 .../sql/catalyst/util/geo/MultiLineString.java     |    49 +
 .../spark/sql/catalyst/util/geo/MultiPoint.java    |    49 +
 .../spark/sql/catalyst/util/geo/MultiPolygon.java  |    49 +
 .../apache/spark/sql/catalyst/util/geo/Point.java  |   101 +
 .../spark/sql/catalyst/util/geo/Polygon.java       |    57 +
 .../apache/spark/sql/catalyst/util/geo/Ring.java   |    69 +
 .../sql/catalyst/util/geo/WkbParseException.java   |    58 +
 .../spark/sql/catalyst/util/geo/WkbReader.java     |   433 +
 .../spark/sql/catalyst/util/geo/WkbUtil.java       |   132 +
 .../spark/sql/catalyst/util/geo/WkbWriter.java     |   225 +
 .../spark/sql/connector/ExternalCommandRunner.java |    45 +
 .../sql/connector/catalog/CatalogExtension.java    |    40 +
 .../spark/sql/connector/catalog/CatalogPlugin.java |    77 +
 .../apache/spark/sql/connector/catalog/Column.java |   139 +
 .../sql/connector/catalog/ColumnDefaultValue.java  |    86 +
 .../spark/sql/connector/catalog/DefaultValue.java  |    95 +
 .../catalog/DelegatingCatalogExtension.java        |   206 +
 .../sql/connector/catalog/FunctionCatalog.java     |    66 +
 .../sql/connector/catalog/MetadataColumn.java      |   125 +
 .../sql/connector/catalog/NamespaceChange.java     |   102 +
 .../sql/connector/catalog/ProcedureCatalog.java    |    43 +
 .../connector/catalog/SessionConfigSupport.java    |    39 +
 .../spark/sql/connector/catalog/StagedTable.java   |    69 +
 .../sql/connector/catalog/StagingTableCatalog.java |   250 +
 .../catalog/SupportsAtomicPartitionManagement.java |   132 +
 .../connector/catalog/SupportsCatalogOptions.java  |    71 +
 .../sql/connector/catalog/SupportsDelete.java      |    94 +
 .../sql/connector/catalog/SupportsDeleteV2.java    |    84 +
 .../connector/catalog/SupportsMetadataColumns.java |    68 +
 .../sql/connector/catalog/SupportsNamespaces.java  |   162 +
 .../catalog/SupportsPartitionManagement.java       |   177 +
 .../spark/sql/connector/catalog/SupportsRead.java  |    43 +
 .../catalog/SupportsRowLevelOperations.java        |    42 +
 .../SupportsV1OverwriteWithSaveAsTable.java        |    71 +
 .../spark/sql/connector/catalog/SupportsWrite.java |    40 +
 .../apache/spark/sql/connector/catalog/Table.java  |   113 +
 .../sql/connector/catalog/TableCapability.java     |   110 +
 .../spark/sql/connector/catalog/TableCatalog.java  |   359 +
 .../connector/catalog/TableCatalogCapability.java  |    96 +
 .../spark/sql/connector/catalog/TableChange.java   |   915 +
 .../spark/sql/connector/catalog/TableInfo.java     |    94 +
 .../spark/sql/connector/catalog/TableProvider.java |    96 +
 .../spark/sql/connector/catalog/TableSummary.java  |    44 +
 .../sql/connector/catalog/TableWritePrivilege.java |    40 +
 .../sql/connector/catalog/TruncatableTable.java    |    37 +
 .../apache/spark/sql/connector/catalog/View.java   |    74 +
 .../spark/sql/connector/catalog/ViewCatalog.java   |   199 +
 .../spark/sql/connector/catalog/ViewChange.java    |    79 +
 .../spark/sql/connector/catalog/ViewInfo.java      |   181 +
 .../catalog/constraints/BaseConstraint.java        |   148 +
 .../sql/connector/catalog/constraints/Check.java   |   134 +
 .../connector/catalog/constraints/Constraint.java  |   127 +
 .../connector/catalog/constraints/ForeignKey.java  |   151 +
 .../connector/catalog/constraints/PrimaryKey.java  |   107 +
 .../sql/connector/catalog/constraints/Unique.java  |   104 +
 .../catalog/functions/AggregateFunction.java       |    95 +
 .../connector/catalog/functions/BoundFunction.java |   103 +
 .../sql/connector/catalog/functions/Function.java  |    38 +
 .../sql/connector/catalog/functions/Reducer.java   |    42 +
 .../catalog/functions/ReducibleFunction.java       |   106 +
 .../catalog/functions/ScalarFunction.java          |   161 +
 .../catalog/functions/SimpleFunction.java          |    42 +
 .../catalog/functions/UnboundFunction.java         |    54 +
 .../sql/connector/catalog/index/SupportsIndex.java |    76 +
 .../sql/connector/catalog/index/TableIndex.java    |    77 +
 .../catalog/procedures/BoundProcedure.java         |    57 +
 .../connector/catalog/procedures/Procedure.java    |    40 +
 .../catalog/procedures/ProcedureParameter.java     |   142 +
 .../catalog/procedures/SimpleProcedure.java        |    42 +
 .../catalog/procedures/UnboundProcedure.java       |    49 +
 .../distributions/ClusteredDistribution.java       |    35 +
 .../sql/connector/distributions/Distribution.java  |    28 +
 .../sql/connector/distributions/Distributions.java |    56 +
 .../distributions/OrderedDistribution.java         |    35 +
 .../distributions/UnspecifiedDistribution.java     |    28 +
 .../spark/sql/connector/expressions/Cast.java      |    77 +
 .../sql/connector/expressions/Expression.java      |    64 +
 .../sql/connector/expressions/Expressions.java     |   193 +
 .../spark/sql/connector/expressions/Extract.java   |    61 +
 .../expressions/GeneralScalarExpression.java       |   468 +
 .../sql/connector/expressions/GetArrayItem.java    |    55 +
 .../spark/sql/connector/expressions/Literal.java   |    46 +
 .../sql/connector/expressions/NamedReference.java  |    41 +
 .../sql/connector/expressions/NullOrdering.java    |    38 +
 .../sql/connector/expressions/SortDirection.java   |    57 +
 .../spark/sql/connector/expressions/SortOrder.java |    46 +
 .../spark/sql/connector/expressions/Transform.java |    44 +
 .../expressions/UserDefinedScalarFunc.java         |    68 +
 .../expressions/aggregate/AggregateFunc.java       |    32 +
 .../expressions/aggregate/Aggregation.java         |    34 +
 .../sql/connector/expressions/aggregate/Avg.java   |    44 +
 .../sql/connector/expressions/aggregate/Count.java |    44 +
 .../connector/expressions/aggregate/CountStar.java |    37 +
 .../aggregate/GeneralAggregateFunc.java            |   103 +
 .../sql/connector/expressions/aggregate/Max.java   |    39 +
 .../sql/connector/expressions/aggregate/Min.java   |    39 +
 .../sql/connector/expressions/aggregate/Sum.java   |    44 +
 .../aggregate/UserDefinedAggregateFunc.java        |    77 +
 .../connector/expressions/filter/AlwaysFalse.java  |    49 +
 .../connector/expressions/filter/AlwaysTrue.java   |    49 +
 .../sql/connector/expressions/filter/And.java      |    37 +
 .../sql/connector/expressions/filter/Not.java      |    35 +
 .../spark/sql/connector/expressions/filter/Or.java |    37 +
 .../connector/expressions/filter/Predicate.java    |   158 +
 .../apache/spark/sql/connector/join/JoinType.java  |    32 +
 .../sql/connector/metric/CustomAvgMetric.java      |    45 +
 .../spark/sql/connector/metric/CustomMetric.java   |    55 +
 .../sql/connector/metric/CustomSumMetric.java      |    38 +
 .../sql/connector/metric/CustomTaskMetric.java     |    50 +
 .../org/apache/spark/sql/connector/read/Batch.java |    48 +
 .../spark/sql/connector/read/HasPartitionKey.java  |    53 +
 .../sql/connector/read/HasPartitionStatistics.java |    47 +
 .../spark/sql/connector/read/InputPartition.java   |    56 +
 .../apache/spark/sql/connector/read/LocalScan.java |    31 +
 .../spark/sql/connector/read/PartitionReader.java  |    61 +
 .../sql/connector/read/PartitionReaderFactory.java |    69 +
 .../org/apache/spark/sql/connector/read/Scan.java  |   160 +
 .../spark/sql/connector/read/ScanBuilder.java      |    34 +
 .../spark/sql/connector/read/Statistics.java       |    41 +
 .../connector/read/SupportsPushDownAggregates.java |    68 +
 .../connector/read/SupportsPushDownFilters.java    |    57 +
 .../sql/connector/read/SupportsPushDownJoin.java   |    78 +
 .../sql/connector/read/SupportsPushDownLimit.java  |    42 +
 .../sql/connector/read/SupportsPushDownOffset.java |    36 +
 .../read/SupportsPushDownRequiredColumns.java      |    44 +
 .../read/SupportsPushDownTableSample.java          |    39 +
 .../sql/connector/read/SupportsPushDownTopN.java   |    43 +
 .../connector/read/SupportsPushDownV2Filters.java  |    63 +
 .../read/SupportsPushDownVariantExtractions.java   |    63 +
 .../sql/connector/read/SupportsReportOrdering.java |    40 +
 .../connector/read/SupportsReportPartitioning.java |    39 +
 .../connector/read/SupportsReportStatistics.java   |    39 +
 .../connector/read/SupportsRuntimeFiltering.java   |    68 +
 .../connector/read/SupportsRuntimeV2Filtering.java |    68 +
 .../sql/connector/read/VariantExtraction.java      |    64 +
 .../connector/read/colstats/ColumnStatistics.java  |    83 +
 .../sql/connector/read/colstats/Histogram.java     |    40 +
 .../sql/connector/read/colstats/HistogramBin.java  |    43 +
 .../read/partitioning/KeyGroupedPartitioning.java  |    55 +
 .../connector/read/partitioning/Partitioning.java  |    42 +
 .../read/partitioning/UnknownPartitioning.java     |    39 +
 .../read/streaming/AcceptsLatestSeenOffset.java    |    39 +
 .../read/streaming/CompositeReadLimit.java         |    59 +
 .../read/streaming/ContinuousPartitionReader.java  |    39 +
 .../ContinuousPartitionReaderFactory.java          |    43 +
 .../connector/read/streaming/ContinuousStream.java |    72 +
 .../connector/read/streaming/MicroBatchStream.java |    59 +
 .../spark/sql/connector/read/streaming/Offset.java |    68 +
 .../connector/read/streaming/PartitionOffset.java  |    34 +
 .../connector/read/streaming/ReadAllAvailable.java |    41 +
 .../sql/connector/read/streaming/ReadLimit.java    |    49 +
 .../sql/connector/read/streaming/ReadMaxBytes.java |    56 +
 .../sql/connector/read/streaming/ReadMaxFiles.java |    55 +
 .../sql/connector/read/streaming/ReadMaxRows.java  |    55 +
 .../sql/connector/read/streaming/ReadMinRows.java  |    65 +
 .../read/streaming/ReportsSinkMetrics.java         |    36 +
 .../read/streaming/ReportsSourceMetrics.java       |    40 +
 .../connector/read/streaming/SparkDataStream.java  |    59 +
 .../read/streaming/SupportsAdmissionControl.java   |    64 +
 .../read/streaming/SupportsRealTimeMode.java       |    47 +
 .../read/streaming/SupportsRealTimeRead.java       |    86 +
 .../streaming/SupportsTriggerAvailableNow.java     |    48 +
 .../sql/connector/util/V2ExpressionSQLBuilder.java |   409 +
 .../spark/sql/connector/write/BatchWrite.java      |   130 +
 .../spark/sql/connector/write/DataWriter.java      |   151 +
 .../sql/connector/write/DataWriterFactory.java     |    58 +
 .../spark/sql/connector/write/DeltaBatchWrite.java |    31 +
 .../spark/sql/connector/write/DeltaWrite.java      |    37 +
 .../sql/connector/write/DeltaWriteBuilder.java     |    37 +
 .../spark/sql/connector/write/DeltaWriter.java     |    78 +
 .../sql/connector/write/DeltaWriterFactory.java    |    34 +
 .../sql/connector/write/LogicalWriteInfo.java      |    68 +
 .../spark/sql/connector/write/MergeSummary.java    |    72 +
 .../sql/connector/write/PhysicalWriteInfo.java     |    35 +
 .../write/RequiresDistributionAndOrdering.java     |   113 +
 .../sql/connector/write/RowLevelOperation.java     |   108 +
 .../connector/write/RowLevelOperationBuilder.java  |    34 +
 .../sql/connector/write/RowLevelOperationInfo.java |    40 +
 .../spark/sql/connector/write/SupportsDelta.java   |    48 +
 .../connector/write/SupportsDynamicOverwrite.java  |    42 +
 .../sql/connector/write/SupportsOverwrite.java     |    78 +
 .../sql/connector/write/SupportsOverwriteV2.java   |    66 +
 .../sql/connector/write/SupportsTruncate.java      |    37 +
 .../apache/spark/sql/connector/write/Write.java    |    90 +
 .../spark/sql/connector/write/WriteBuilder.java    |    78 +
 .../spark/sql/connector/write/WriteSummary.java    |    29 +
 .../sql/connector/write/WriterCommitMessage.java   |    36 +
 .../streaming/StreamingDataWriterFactory.java      |    62 +
 .../connector/write/streaming/StreamingWrite.java  |   101 +
 .../spark/sql/util/CaseInsensitiveStringMap.java   |   204 +
 .../apache/spark/sql/util/NumericHistogram.java    |   292 +
 .../spark/sql/vectorized/ArrowColumnVector.java    |   684 +
 .../apache/spark/sql/vectorized/ColumnVector.java  |   370 +
 .../apache/spark/sql/vectorized/ColumnarArray.java |   228 +
 .../apache/spark/sql/vectorized/ColumnarBatch.java |   128 +
 .../spark/sql/vectorized/ColumnarBatchRow.java     |   231 +
 .../apache/spark/sql/vectorized/ColumnarMap.java   |    54 +
 .../apache/spark/sql/vectorized/ColumnarRow.java   |   234 +
 .../apache/spark/sql/catalyst/CapturesConfig.scala |   100 +
 .../sql/catalyst/CatalystTypeConverters.scala      |   669 +
 .../spark/sql/catalyst/CurrentUserContext.scala    |    30 +
 .../spark/sql/catalyst/DataSourceOptions.scala     |   114 +
 .../sql/catalyst/DeserializerBuildHelper.scala     |   529 +
 .../sql/catalyst/ExtendedAnalysisException.scala   |    93 +
 .../spark/sql/catalyst/FileSourceOptions.scala     |    53 +
 .../apache/spark/sql/catalyst/InternalRow.scala    |   194 +
 .../apache/spark/sql/catalyst/OrderedFilters.scala |    99 +
 .../spark/sql/catalyst/ProjectingInternalRow.scala |   133 +
 .../spark/sql/catalyst/QueryPlanningTracker.scala  |   234 +
 .../apache/spark/sql/catalyst/SQLConfHelper.scala  |    61 +
 .../spark/sql/catalyst/SerializerBuildHelper.scala |   540 +
 .../sql/catalyst/SqlScriptingContextManager.scala  |    26 +
 .../apache/spark/sql/catalyst/StructFilters.scala  |   169 +
 .../sql/catalyst/analysis/AliasResolution.scala    |    76 +
 .../catalyst/analysis/AlreadyExistException.scala  |   101 +
 .../spark/sql/catalyst/analysis/Analyzer.scala     |  4167 +++
 .../AnsiGetDateFieldOperationsTypeCoercion.scala   |    34 +
 .../analysis/AnsiStringPromotionTypeCoercion.scala |   118 +
 .../sql/catalyst/analysis/AnsiTypeCoercion.scala   |   259 +
 .../analysis/ApplyCharTypePaddingHelper.scala      |   206 +
 .../catalyst/analysis/ApplyDefaultCollation.scala  |   378 +
 .../sql/catalyst/analysis/ApplyLimitAll.scala      |    52 +
 .../sql/catalyst/analysis/AssignmentUtils.scala    |   394 +
 .../BinaryArithmeticWithDatetimeResolver.scala     |   165 +
 .../analysis/BooleanEqualityTypeCoercion.scala     |    87 +
 .../sql/catalyst/analysis/CTESubstitution.scala    |   465 +
 .../sql/catalyst/analysis/CheckAnalysis.scala      |  1143 +
 .../catalyst/analysis/CollationRulesRunner.scala   |    29 +
 .../sql/catalyst/analysis/CollationTypeCasts.scala |    27 +
 .../catalyst/analysis/CollationTypeCoercion.scala  |   518 +
 .../catalyst/analysis/ColumnResolutionHelper.scala |   665 +
 .../sql/catalyst/analysis/DecimalPrecision.scala   |    56 +
 .../analysis/DecimalPrecisionTypeCoercion.scala    |   208 +
 .../catalyst/analysis/DeduplicateRelations.scala   |   529 +
 .../analysis/DeduplicateUnionChildOutput.scala     |    82 +
 .../catalyst/analysis/DivisionTypeCoercion.scala   |    41 +
 .../analysis/EliminateLazyExpression.scala         |    34 +
 .../analysis/ExtractDistributedSequenceID.scala    |    43 +
 .../sql/catalyst/analysis/FunctionRegistry.scala   |  1311 +
 .../sql/catalyst/analysis/FunctionResolution.scala |   353 +
 .../catalyst/analysis/GeneratorResolution.scala    |    51 +
 .../analysis/GroupingAnalyticsTransformer.scala    |   278 +
 .../sql/catalyst/analysis/HintErrorLogger.scala    |    66 +
 .../catalyst/analysis/IdentifierResolution.scala   |    51 +
 .../analysis/IntegralDivisionTypeCoercion.scala    |    38 +
 .../sql/catalyst/analysis/KeepLegacyOutputs.scala  |    48 +
 .../spark/sql/catalyst/analysis/LambdaBinder.scala |    90 +
 .../analysis/LiteralFunctionResolution.scala       |    52 +
 .../catalyst/analysis/MultiInstanceRelation.scala  |    33 +
 .../catalyst/analysis/NameStreamingSources.scala   |   142 +
 .../sql/catalyst/analysis/NamedRelation.scala      |    27 +
 .../catalyst/analysis/NamedStreamingRelation.scala |   103 +
 .../analysis/NaturalAndUsingJoinResolution.scala   |   169 +
 .../catalyst/analysis/NoSuchItemException.scala    |    94 +
 .../NondeterministicExpressionCollection.scala     |    48 +
 .../catalyst/analysis/NormalizeableRelation.scala  |    32 +
 .../ParameterizedQueryArgumentsValidator.scala     |    59 +
 .../sql/catalyst/analysis/PivotTransformer.scala   |   223 +
 .../spark/sql/catalyst/analysis/PlanToString.scala |    36 +
 .../analysis/PullOutNondeterministic.scala         |    77 +
 .../sql/catalyst/analysis/RelationCache.scala      |    28 +
 .../sql/catalyst/analysis/RelationResolution.scala |   322 +
 .../sql/catalyst/analysis/RelationTimeTravel.scala |    34 +
 .../sql/catalyst/analysis/ResolveCatalogs.scala    |   208 +
 .../catalyst/analysis/ResolveCollationName.scala   |    36 +
 .../ResolveColumnDefaultInCommandInputQuery.scala  |   178 +
 .../analysis/ResolveCommandsWithIfExists.scala     |    35 +
 .../sql/catalyst/analysis/ResolveCursors.scala     |   109 +
 .../analysis/ResolveDataFrameDropColumns.scala     |    56 +
 .../analysis/ResolveEventTimeWatermark.scala       |    55 +
 .../sql/catalyst/analysis/ResolveFetchCursor.scala |    91 +
 .../spark/sql/catalyst/analysis/ResolveHints.scala |   318 +
 .../analysis/ResolveIdentifierClause.scala         |   107 +
 .../catalyst/analysis/ResolveInlineTables.scala    |    36 +
 .../catalyst/analysis/ResolveInsertionBase.scala   |    77 +
 .../ResolveLateralColumnAliasReference.scala       |   311 +
 .../analysis/ResolveMergeIntoSchemaEvolution.scala |    85 +
 .../catalyst/analysis/ResolvePartitionSpec.scala   |    92 +
 .../analysis/ResolveReferencesInAggregate.scala    |   203 +
 .../analysis/ResolveReferencesInSort.scala         |   116 +
 .../analysis/ResolveReferencesInUpdate.scala       |    74 +
 .../ResolveRowLevelCommandAssignments.scala        |   135 +
 .../sql/catalyst/analysis/ResolveSetVariable.scala |    91 +
 .../analysis/ResolveTableConstraints.scala         |    89 +
 .../sql/catalyst/analysis/ResolveTableSpec.scala   |   115 +
 .../sql/catalyst/analysis/ResolveTimeWindows.scala |   363 +
 .../spark/sql/catalyst/analysis/ResolveUnion.scala |   214 +
 .../ResolveUpdateEventTimeWatermarkColumn.scala    |    52 +
 .../sql/catalyst/analysis/ResolveWithCTE.scala     |   368 +
 .../catalyst/analysis/RewriteCollationJoin.scala   |    99 +
 .../catalyst/analysis/RewriteDeleteFromTable.scala |   119 +
 .../catalyst/analysis/RewriteMergeIntoTable.scala  |   517 +
 .../catalyst/analysis/RewriteRowLevelCommand.scala |   318 +
 .../sql/catalyst/analysis/RewriteUpdateTable.scala |   226 +
 .../catalyst/analysis/SQLFunctionExpression.scala  |    90 +
 .../sql/catalyst/analysis/SQLFunctionNode.scala    |    64 +
 .../sql/catalyst/analysis/StackTypeCoercion.scala  |    40 +
 .../catalyst/analysis/StreamingJoinHelper.scala    |   300 +
 .../analysis/StringLiteralTypeCoercion.scala       |    48 +
 .../analysis/StringPromotionTypeCoercion.scala     |    83 +
 ...ionInLambdaOrHigherOrderFunctionValidator.scala |    43 +
 .../catalyst/analysis/TableOutputResolver.scala    |   719 +
 .../sql/catalyst/analysis/TimeTravelSpec.scala     |   112 +
 .../sql/catalyst/analysis/TypeCheckResult.scala    |    71 +
 .../spark/sql/catalyst/analysis/TypeCoercion.scala |   412 +
 .../sql/catalyst/analysis/TypeCoercionBase.scala   |   469 +
 .../sql/catalyst/analysis/TypeCoercionHelper.scala |   734 +
 .../catalyst/analysis/TypeCoercionValidation.scala |   127 +
 .../sql/catalyst/analysis/UnpivotTransformer.scala |    76 +
 .../catalyst/analysis/UnpivotTypeCoercion.scala    |    52 +
 .../analysis/UnsupportedOperationChecker.scala     |   651 +
 .../sql/catalyst/analysis/UpCastResolution.scala   |    69 +
 .../analysis/UpdateAttributeNullability.scala      |    56 +
 .../sql/catalyst/analysis/V2TableReference.scala   |   143 +
 .../analysis/ValidateSubqueryExpression.scala      |   557 +
 .../sql/catalyst/analysis/VariableResolution.scala |   174 +
 .../sql/catalyst/analysis/ViewResolution.scala     |    63 +
 .../sql/catalyst/analysis/WindowResolution.scala   |   174 +
 .../catalyst/analysis/higherOrderFunctions.scala   |    83 +
 .../spark/sql/catalyst/analysis/package.scala      |   109 +
 .../spark/sql/catalyst/analysis/parameters.scala   |   279 +
 .../resolver/AggregateExpressionResolver.scala     |   240 +
 .../resolver/AggregateResolutionResult.scala       |    34 +
 .../analysis/resolver/AggregateResolver.scala      |   251 +
 .../AggregateWithLcaResolutionResult.scala         |    36 +
 .../catalyst/analysis/resolver/AliasResolver.scala |   135 +
 .../analysis/resolver/AnalyzerBridgeState.scala    |    72 +
 .../analysis/resolver/AttributeScopeStack.scala    |   120 +
 .../resolver/AutoGeneratedAliasProvider.scala      |    77 +
 .../resolver/BinaryArithmeticResolver.scala        |   170 +
 .../analysis/resolver/BridgedRelationId.scala      |    31 +
 .../resolver/BridgedRelationMetadataProvider.scala |   104 +
 .../resolver/CandidatesForResolution.scala         |    26 +
 .../analysis/resolver/CoercesExpressionTypes.scala |   185 +
 .../sql/catalyst/analysis/resolver/CteScope.scala  |   381 +
 .../resolver/DefaultCollationTypeCoercion.scala    |    98 +
 .../resolver/DelegatesResolutionToExtensions.scala |    72 +
 .../ExplicitlyUnsupportedResolverFeature.scala     |    56 +
 .../analysis/resolver/ExpressionIdAssigner.scala   |   712 +
 .../resolver/ExpressionResolutionContext.scala     |    92 +
 .../resolver/ExpressionResolutionValidator.scala   |   213 +
 .../analysis/resolver/ExpressionResolver.scala     |  1101 +
 .../resolver/ExpressionTreeTraversal.scala         |   128 +
 .../analysis/resolver/FilterResolver.scala         |    71 +
 .../analysis/resolver/FunctionResolver.scala       |   205 +
 .../GroupingAndAggregateExpressionsExtractor.scala |   134 +
 .../analysis/resolver/HavingResolver.scala         |   224 +
 .../analysis/resolver/HybridAnalyzer.scala         |   318 +
 .../resolver/IdentifierAndCteSubstituor.scala      |   204 +
 .../catalyst/analysis/resolver/IdentifierMap.scala |    28 +
 .../catalyst/analysis/resolver/JoinResolver.scala  |   351 +
 .../analysis/resolver/KeyTransformingMap.scala     |    56 +
 .../LateralColumnAliasProhibitedRegistry.scala     |    44 +
 .../resolver/LateralColumnAliasRegistry.scala      |    46 +
 .../resolver/LateralColumnAliasRegistryImpl.scala  |   184 +
 .../resolver/LateralColumnAliasResolver.scala      |   347 +
 .../resolver/LimitLikeExpressionValidator.scala    |   151 +
 .../analysis/resolver/MetadataResolver.scala       |   144 +
 .../sql/catalyst/analysis/resolver/NameScope.scala |  1297 +
 .../catalyst/analysis/resolver/NameTarget.scala    |   105 +
 .../OperatorWithUncomparableTypeValidator.scala    |    80 +
 .../resolver/OrdinalReplacementExpressions.scala   |    48 +
 .../analysis/resolver/OrdinalResolver.scala        |   124 +
 .../catalyst/analysis/resolver/OutputType.scala    |    28 +
 .../catalyst/analysis/resolver/PlanLogger.scala    |    94 +
 .../catalyst/analysis/resolver/PlanRewriter.scala  |    74 +
 .../resolver/ProducesUnresolvedSubtree.scala       |    79 +
 .../analysis/resolver/ProhibitedResolver.scala     |    35 +
 .../analysis/resolver/ProjectResolver.scala        |   259 +
 .../analysis/resolver/PruneMetadataColumns.scala   |   153 +
 .../catalyst/analysis/resolver/RelationId.scala    |    31 +
 .../resolver/RelationMetadataProvider.scala        |    91 +
 .../analysis/resolver/ResolutionCheckRunner.scala  |    63 +
 .../analysis/resolver/ResolutionValidator.scala    |   326 +
 .../resolver/ResolvedAggregateExpressions.scala    |    46 +
 .../analysis/resolver/ResolvedProjectList.scala    |    40 +
 .../resolver/ResolvedSubqueryExpressionPlan.scala  |    37 +
 .../sql/catalyst/analysis/resolver/Resolver.scala  |   799 +
 .../analysis/resolver/ResolverExtension.scala      |    41 +
 .../catalyst/analysis/resolver/ResolverGuard.scala |   559 +
 .../analysis/resolver/ResolverMetricTracker.scala  |    77 +
 .../analysis/resolver/ResolverRunner.scala         |    95 +
 .../catalyst/analysis/resolver/ResolverTag.scala   |    50 +
 .../resolver/ResolvesExpressionChildren.scala      |   109 +
 .../resolver/ResolvesNameByHiddenOutput.scala      |   380 +
 .../resolver/RewritesAliasesInTopLcaProject.scala  |   256 +
 .../analysis/resolver/SemanticComparator.scala     |    71 +
 .../resolver/SemiStructuredExtractResolver.scala   |    77 +
 .../resolver/SetOperationLikeResolver.scala        |   399 +
 .../catalyst/analysis/resolver/SortResolver.scala  |   318 +
 .../resolver/SubqueryExpressionResolver.scala      |   201 +
 .../catalyst/analysis/resolver/SubqueryScope.scala |   142 +
 .../resolver/TimezoneAwareExpressionResolver.scala |   164 +
 .../analysis/resolver/TreeNodeResolver.scala       |    32 +
 .../resolver/UnresolvedCteRelationRef.scala        |    29 +
 ...UnsupportedExpressionInOperatorValidation.scala |    77 +
 .../analysis/resolver/ValidatesFilter.scala        |    78 +
 .../catalyst/analysis/resolver/ViewResolver.scala  |   186 +
 .../sql/catalyst/analysis/resolver/package.scala   |    26 +
 .../sql/catalyst/analysis/timeZoneAnalysis.scala   |    61 +
 .../spark/sql/catalyst/analysis/unresolved.scala   |  1243 +
 .../sql/catalyst/analysis/v2ResolutionPlans.scala  |   274 +
 .../apache/spark/sql/catalyst/analysis/view.scala  |   119 +
 .../sql/catalyst/catalog/ExternalCatalog.scala     |   320 +
 .../catalyst/catalog/ExternalCatalogUtils.scala    |   323 +
 .../catalog/ExternalCatalogWithListener.scala      |   322 +
 .../catalog/FunctionExpressionBuilder.scala        |    32 +
 .../catalyst/catalog/GlobalTempViewManager.scala   |   119 +
 .../sql/catalyst/catalog/InMemoryCatalog.scala     |   684 +
 .../catalog/InvalidUDFClassException.scala         |    39 +
 .../sql/catalyst/catalog/RoutineLanguage.scala     |    29 +
 .../spark/sql/catalyst/catalog/SQLFunction.scala   |   312 +
 .../sql/catalyst/catalog/SessionCatalog.scala      |  2474 ++
 .../catalog/SqlScriptingContextManager.scala       |    33 +
 .../SqlScriptingExecutionContextExtension.scala    |    49 +
 .../sql/catalyst/catalog/UserDefinedFunction.scala |   171 +
 .../catalog/UserDefinedFunctionErrors.scala        |   124 +
 .../sql/catalyst/catalog/VariableManager.scala     |   163 +
 .../apache/spark/sql/catalyst/catalog/events.scala |   290 +
 .../sql/catalyst/catalog/functionResources.scala   |    64 +
 .../spark/sql/catalyst/catalog/interface.scala     |  1193 +
 .../spark/sql/catalyst/csv/CSVExprUtils.scala      |   143 +
 .../spark/sql/catalyst/csv/CSVHeaderChecker.scala  |   154 +
 .../spark/sql/catalyst/csv/CSVInferSchema.scala    |   322 +
 .../apache/spark/sql/catalyst/csv/CSVOptions.scala |   441 +
 .../sql/catalyst/csv/UnivocityGenerator.scala      |   230 +
 .../spark/sql/catalyst/csv/UnivocityParser.scala   |   665 +
 .../apache/spark/sql/catalyst/dsl/package.scala    |   584 +
 .../spark/sql/catalyst/encoders/EncoderUtils.scala |   176 +
 .../sql/catalyst/encoders/ExpressionEncoder.scala  |   360 +
 .../encoders/KryoSerializationCodecImpl.scala      |    33 +
 .../spark/sql/catalyst/encoders/package.scala      |    37 +
 .../sql/catalyst/expressions/AliasHelper.scala     |   173 +
 .../expressions/AnalysisAwareExpression.scala      |    23 +
 .../expressions/ApplyFunctionExpression.scala      |    55 +
 .../expressions/ApproxTopKExpressions.scala        |   118 +
 .../sql/catalyst/expressions/AttributeMap.scala    |    61 +
 .../sql/catalyst/expressions/AttributeSet.scala    |   157 +
 .../spark/sql/catalyst/expressions/Between.scala   |    63 +
 .../expressions/BloomFilterMightContain.scala      |   123 +
 .../sql/catalyst/expressions/BoundAttribute.scala  |   110 +
 .../expressions/CallMethodViaReflection.scala      |   245 +
 .../spark/sql/catalyst/expressions/Cast.scala      |  2482 ++
 .../CodeGeneratorWithInterpretedFallback.scala     |    61 +
 .../sql/catalyst/expressions/CollationKey.scala    |   107 +
 .../sql/catalyst/expressions/CursorReference.scala |    77 +
 .../expressions/DirectShufflePartitionID.scala     |    46 +
 .../expressions/DistributedSequenceID.scala        |    42 +
 .../sql/catalyst/expressions/DynamicPruning.scala  |   119 +
 .../expressions/EquivalentExpressions.scala        |   258 +
 .../sql/catalyst/expressions/EvalHelper.scala      |    32 +
 .../spark/sql/catalyst/expressions/EvalMode.scala  |    43 +
 .../catalyst/expressions/ExpectsInputTypes.scala   |    76 +
 .../spark/sql/catalyst/expressions/ExprUtils.scala |   219 +
 .../sql/catalyst/expressions/Expression.scala      |  1499 +
 .../sql/catalyst/expressions/ExpressionSet.scala   |   191 +
 .../expressions/ExpressionsEvaluator.scala         |    52 +
 .../FunctionTableSubqueryArgumentExpression.scala  |   187 +
 .../expressions/GetStructFieldObject.scala         |    32 +
 .../expressions/GroupingAnalyticsExtractor.scala   |    48 +
 .../expressions/InterpretedMutableProjection.scala |   113 +
 .../expressions/InterpretedSafeProjection.scala    |   133 +
 .../expressions/InterpretedUnsafeProjection.scala  |   330 +
 .../spark/sql/catalyst/expressions/JoinedRow.scala |   179 +
 .../expressions/MonotonicallyIncreasingID.scala    |    97 +
 .../expressions/NamedArgumentExpression.scala      |    58 +
 .../catalyst/expressions/NumericEvalContext.scala  |    57 +
 .../catalyst/expressions/PartitionTransforms.scala |   108 +
 .../sql/catalyst/expressions/Projection.scala      |   205 +
 .../expressions/ProjectionOverSchema.scala         |    85 +
 .../spark/sql/catalyst/expressions/PythonUDF.scala |   387 +
 .../spark/sql/catalyst/expressions/ScalaUDF.scala  |  1213 +
 .../sql/catalyst/expressions/SchemaPruning.scala   |   173 +
 .../sql/catalyst/expressions/SelectedField.scala   |   157 +
 .../expressions/SemiStructuredExtract.scala        |    60 +
 .../sql/catalyst/expressions/SessionWindow.scala   |   106 +
 .../spark/sql/catalyst/expressions/SortOrder.scala |   238 +
 .../catalyst/expressions/SparkPartitionID.scala    |    60 +
 .../catalyst/expressions/SpecificInternalRow.scala |   326 +
 .../expressions/SubExprEvaluationRuntime.scala     |   151 +
 .../sql/catalyst/expressions/TimeWindow.scala      |   255 +
 .../sql/catalyst/expressions/ToJavaArray.scala     |   105 +
 .../sql/catalyst/expressions/ToPrettyString.scala  |    78 +
 .../sql/catalyst/expressions/ToStringBase.scala    |   495 +
 .../catalyst/expressions/TransformExpression.scala |   138 +
 .../spark/sql/catalyst/expressions/TryEval.scala   |   308 +
 .../spark/sql/catalyst/expressions/UnwrapUDT.scala |    55 +
 .../catalyst/expressions/V2ExpressionUtils.scala   |   377 +
 .../catalyst/expressions/VariableReference.scala   |    63 +
 .../sql/catalyst/expressions/WindowTime.scala      |    65 +
 .../spark/sql/catalyst/expressions/With.scala      |   168 +
 .../catalyst/expressions/aggregate/AnyValue.scala  |    68 +
 .../ApproxCountDistinctForIntervals.scala          |   278 +
 .../aggregate/ApproxTopKAggregates.scala           |   879 +
 .../aggregate/ApproximatePercentile.scala          |   383 +
 .../catalyst/expressions/aggregate/Average.scala   |   169 +
 .../aggregate/BloomFilterAggregate.scala           |   259 +
 .../expressions/aggregate/CentralMomentAgg.scala   |   455 +
 .../sql/catalyst/expressions/aggregate/Corr.scala  |   134 +
 .../sql/catalyst/expressions/aggregate/Count.scala |   108 +
 .../catalyst/expressions/aggregate/CountIf.scala   |    47 +
 .../expressions/aggregate/CountMinSketchAgg.scala  |   229 +
 .../expressions/aggregate/Covariance.scala         |   171 +
 .../sql/catalyst/expressions/aggregate/First.scala |   128 +
 .../expressions/aggregate/HistogramNumeric.scala   |   268 +
 .../aggregate/HyperLogLogPlusPlus.scala            |   154 +
 .../sql/catalyst/expressions/aggregate/Last.scala  |   117 +
 .../sql/catalyst/expressions/aggregate/Max.scala   |    67 +
 .../expressions/aggregate/MaxByAndMinBy.scala      |   147 +
 .../catalyst/expressions/aggregate/Measure.scala   |    60 +
 .../sql/catalyst/expressions/aggregate/Min.scala   |    67 +
 .../sql/catalyst/expressions/aggregate/Mode.scala  |   353 +
 .../expressions/aggregate/PandasAggregate.scala    |    35 +
 .../expressions/aggregate/PivotFirst.scala         |   162 +
 .../catalyst/expressions/aggregate/Product.scala   |   179 +
 .../sql/catalyst/expressions/aggregate/Sum.scala   |   227 +
 .../aggregate/SupportsOrderingWithinGroup.scala    |    45 +
 .../expressions/aggregate/V2Aggregator.scala       |    68 +
 .../expressions/aggregate/bitwiseAggregates.scala  |   117 +
 .../expressions/aggregate/boolAggregates.scala     |    66 +
 .../catalyst/expressions/aggregate/collect.scala   |   581 +
 .../aggregate/datasketchesAggregates.scala         |   407 +
 .../expressions/aggregate/interfaces.scala         |   704 +
 .../expressions/aggregate/kllAggregates.scala      |   856 +
 .../expressions/aggregate/linearRegression.scala   |   382 +
 .../expressions/aggregate/percentiles.scala        |   530 +
 .../aggregate/thetasketchesAggregates.scala        |   650 +
 .../aggregate/tupleIntersectionAgg.scala           |   353 +
 .../expressions/aggregate/tupleSketchAgg.scala     |   534 +
 .../expressions/aggregate/tupleSketchState.scala   |    55 +
 .../expressions/aggregate/tupleUnionAgg.scala      |   409 +
 .../sql/catalyst/expressions/arithmetic.scala      |  1422 +
 .../catalyst/expressions/avroSqlFunctions.scala    |   295 +
 .../catalyst/expressions/bitmapExpressions.scala   |   417 +
 .../catalyst/expressions/bitwiseExpressions.scala  |   314 +
 .../expressions/codegen/CodeFormatter.scala        |   165 +
 .../expressions/codegen/CodeGenerator.scala        |  2098 ++
 .../expressions/codegen/CodegenFallback.scala      |    66 +
 .../codegen/GenerateMutableProjection.scala        |   151 +
 .../expressions/codegen/GenerateOrdering.scala     |   222 +
 .../expressions/codegen/GeneratePredicate.scala    |    78 +
 .../codegen/GenerateSafeProjection.scala           |   209 +
 .../codegen/GenerateUnsafeProjection.scala         |   381 +
 .../codegen/GenerateUnsafeRowJoiner.scala          |   257 +
 .../catalyst/expressions/codegen/javaCode.scala    |   367 +
 .../sql/catalyst/expressions/codegen/package.scala |    75 +
 .../expressions/collationExpressions.scala         |   175 +
 .../expressions/collectionOperations.scala         |  5356 ++++
 .../catalyst/expressions/complexTypeCreator.scala  |   834 +
 .../expressions/complexTypeExtractors.scala        |   570 +
 .../expressions/conditionalExpressions.scala       |   442 +
 .../expressions/constraintExpressions.scala        |    69 +
 .../sql/catalyst/expressions/constraints.scala     |   380 +
 .../expressions/csv/CsvExpressionEvalUtils.scala   |   123 +
 .../sql/catalyst/expressions/csvExpressions.scala  |   296 +
 .../expressions/datasketchesExpressions.scala      |   133 +
 .../catalyst/expressions/datetimeExpressions.scala |  3899 +++
 .../catalyst/expressions/decimalExpressions.scala  |   322 +
 .../sql/catalyst/expressions/generators.scala      |   768 +
 .../spark/sql/catalyst/expressions/grouping.scala  |   289 +
 .../spark/sql/catalyst/expressions/hash.scala      |  1256 +
 .../expressions/higherOrderFunctions.scala         |  1333 +
 .../sql/catalyst/expressions/inputFileBlock.scala  |   119 +
 .../catalyst/expressions/intervalExpressions.scala |   889 +
 .../expressions/json/JsonExpressionEvalUtils.scala |   572 +
 .../sql/catalyst/expressions/jsonExpressions.scala |   582 +
 .../sql/catalyst/expressions/kllExpressions.scala  |   701 +
 .../spark/sql/catalyst/expressions/literals.scala  |   615 +
 .../sql/catalyst/expressions/maskExpressions.scala |   353 +
 .../sql/catalyst/expressions/mathExpressions.scala |  1913 ++
 .../spark/sql/catalyst/expressions/misc.scala      |   565 +
 .../catalyst/expressions/namedExpressions.scala    |   812 +
 .../sql/catalyst/expressions/nullExpressions.scala |   566 +
 .../expressions/numberFormatExpressions.scala      |   341 +
 .../sql/catalyst/expressions/objects/objects.scala |  2145 ++
 .../spark/sql/catalyst/expressions/ordering.scala  |   131 +
 .../spark/sql/catalyst/expressions/package.scala   |   417 +
 .../sql/catalyst/expressions/pipeOperators.scala   |   138 +
 .../sql/catalyst/expressions/predicates.scala      |  1396 +
 .../catalyst/expressions/randomExpressions.scala   |   442 +
 .../catalyst/expressions/regexpExpressions.scala   |  1258 +
 .../spark/sql/catalyst/expressions/rows.scala      |   184 +
 .../expressions/st/STExpressionUtils.scala         |    85 +
 .../catalyst/expressions/st/stExpressions.scala    |   296 +
 .../catalyst/expressions/stringExpressions.scala   |  3793 +++
 .../spark/sql/catalyst/expressions/subquery.scala  |   643 +
 .../expressions/thetasketchesExpressions.scala     |   216 +
 .../sql/catalyst/expressions/timeExpressions.scala |   980 +
 .../expressions/toFromProtobufSqlFunctions.scala   |   320 +
 .../sql/catalyst/expressions/tupleDifference.scala |   128 +
 .../catalyst/expressions/tupleIntersection.scala   |   170 +
 .../catalyst/expressions/tupleSketchEstimate.scala |   170 +
 .../catalyst/expressions/tupleSketchSummary.scala  |   157 +
 .../sql/catalyst/expressions/tupleUnion.scala      |   290 +
 .../expressions/url/UrlExpressionEvalUtils.scala   |   148 +
 .../sql/catalyst/expressions/urlExpressions.scala  |   273 +
 .../variant/VariantExpressionEvalUtils.scala       |   153 +
 .../expressions/variant/variantExpressions.scala   |   954 +
 .../catalyst/expressions/vectorExpressions.scala   |   799 +
 .../catalyst/expressions/windowExpressions.scala   |  1228 +
 .../expressions/xml/XmlExpressionEvalUtils.scala   |   211 +
 .../spark/sql/catalyst/expressions/xml/xpath.scala |   257 +
 .../sql/catalyst/expressions/xmlExpressions.scala  |   285 +
 .../apache/spark/sql/catalyst/identifiers.scala    |   158 +
 .../sql/catalyst/json/CreateJacksonParser.scala    |    93 +
 .../spark/sql/catalyst/json/JSONOptions.scala      |   316 +
 .../spark/sql/catalyst/json/JacksonGenerator.scala |   336 +
 .../spark/sql/catalyst/json/JacksonParser.scala    |   736 +
 .../spark/sql/catalyst/json/JacksonUtils.scala     |    63 +
 .../spark/sql/catalyst/json/JsonFilters.scala      |   157 +
 .../spark/sql/catalyst/json/JsonInferSchema.scala  |   464 +
 .../sql/catalyst/normalizer/NormalizeCTEIds.scala  |    67 +
 .../sql/catalyst/optimizer/ComplexTypes.scala      |    69 +
 .../catalyst/optimizer/CostBasedJoinReorder.scala  |   496 +
 .../CrossJoinArrayContainsToInnerJoin.scala        |   129 +
 .../catalyst/optimizer/DecorrelateInnerQuery.scala |  1117 +
 .../catalyst/optimizer/EliminateResolvedHint.scala |   100 +
 .../optimizer/EliminateSQLFunctionNode.scala       |    47 +
 .../optimizer/EliminateWindowPartitions.scala      |    41 +
 .../catalyst/optimizer/InferWindowGroupLimit.scala |   138 +
 .../catalyst/optimizer/InjectRuntimeFilter.scala   |   339 +
 .../spark/sql/catalyst/optimizer/InlineCTE.scala   |   278 +
 .../optimizer/InsertMapSortExpression.scala        |   160 +
 .../optimizer/LimitPushDownThroughWindow.scala     |    59 +
 .../sql/catalyst/optimizer/MergeSubplans.scala     |   380 +
 .../catalyst/optimizer/NestedColumnAliasing.scala  |   504 +
 .../optimizer/NormalizeFloatingNumbers.scala       |   236 +
 .../catalyst/optimizer/OptimizeCsvJsonExprs.scala  |   131 +
 .../catalyst/optimizer/OptimizeJoinCondition.scala |    45 +
 .../catalyst/optimizer/OptimizeOneRowPlan.scala    |    68 +
 .../sql/catalyst/optimizer/OptimizeRand.scala      |    63 +
 .../spark/sql/catalyst/optimizer/Optimizer.scala   |  2855 ++
 .../spark/sql/catalyst/optimizer/PlanMerger.scala  |   304 +
 .../optimizer/PropagateEmptyRelation.scala         |   224 +
 .../optimizer/PullOutGroupingExpressions.scala     |    80 +
 .../PullOutNestedDataOuterRefExpressions.scala     |   136 +
 .../optimizer/PushDownLeftSemiAntiJoin.scala       |   277 +
 .../optimizer/PushExtraPredicateThroughJoin.scala  |    81 +
 .../PushProjectionThroughLimitAndOffset.scala      |    43 +
 ...ushdownPredicatesAndPruneColumnsForCTEDef.scala |   176 +
 .../optimizer/RemoveRedundantAggregates.scala      |    69 +
 .../catalyst/optimizer/RemoveRedundantSorts.scala  |    62 +
 .../optimizer/ReplaceCTERefWithRepartition.scala   |    91 +
 .../optimizer/ReplaceExceptWithFilter.scala        |   118 +
 .../ReplaceNullWithFalseInPredicate.scala          |   166 +
 .../sql/catalyst/optimizer/RewriteAsOfJoin.scala   |    90 +
 .../optimizer/RewriteDistinctAggregates.scala      |   437 +
 .../catalyst/optimizer/RewriteWithExpression.scala |   201 +
 .../catalyst/optimizer/StarSchemaDetection.scala   |   352 +
 .../optimizer/UnwrapCastInBinaryComparison.scala   |   513 +
 .../sql/catalyst/optimizer/UpdateFields.scala      |    87 +
 .../spark/sql/catalyst/optimizer/expressions.scala |  1237 +
 .../sql/catalyst/optimizer/finishAnalysis.scala    |   198 +
 .../spark/sql/catalyst/optimizer/joins.scala       |   563 +
 .../spark/sql/catalyst/optimizer/objects.scala     |   294 +
 .../spark/sql/catalyst/optimizer/subquery.scala    |  1162 +
 .../org/apache/spark/sql/catalyst/package.scala    |    25 +
 .../sql/catalyst/parser/AbstractSqlParser.scala    |   123 +
 .../spark/sql/catalyst/parser/AstBuilder.scala     |  7088 +++++
 .../sql/catalyst/parser/CatalystSqlParser.scala    |    27 +
 .../sql/catalyst/parser/ParameterContext.scala     |    52 +
 .../sql/catalyst/parser/ParameterHandler.scala     |   229 +
 .../sql/catalyst/parser/ParserInterface.scala      |    83 +
 .../spark/sql/catalyst/parser/ParserUtils.scala    |   497 +
 .../catalyst/parser/SubstituteParamsParser.scala   |   224 +
 .../apache/spark/sql/catalyst/parser/package.scala |    48 +
 .../spark/sql/catalyst/planning/QueryPlanner.scala |   105 +
 .../spark/sql/catalyst/planning/package.scala      |    23 +
 .../spark/sql/catalyst/planning/patterns.scala     |   486 +
 .../plans/AliasAwareOutputExpression.scala         |   133 +
 .../sql/catalyst/plans/DescribeCommandSchema.scala |    43 +
 .../spark/sql/catalyst/plans/NormalizePlan.scala   |   291 +
 .../spark/sql/catalyst/plans/QueryPlan.scala       |   865 +
 .../sql/catalyst/plans/ReferenceAllColumns.scala   |    34 +
 .../spark/sql/catalyst/plans/joinTypes.scala       |   183 +
 .../catalyst/plans/logical/AnalysisHelper.scala    |   427 +
 .../catalyst/plans/logical/ColumnDefinition.scala  |   275 +
 .../spark/sql/catalyst/plans/logical/Command.scala |    76 +
 .../plans/logical/DistinctKeyVisitor.scala         |   160 +
 .../sql/catalyst/plans/logical/EmptyRelation.scala |    35 +
 .../plans/logical/EventTimeWatermark.scala         |   119 +
 .../plans/logical/ExecutableDuringAnalysis.scala   |    28 +
 .../plans/logical/FunctionBuilderBase.scala        |   234 +
 .../sql/catalyst/plans/logical/LocalRelation.scala |   125 +
 .../sql/catalyst/plans/logical/LogicalPlan.scala   |   505 +
 .../plans/logical/LogicalPlanDistinctKeys.scala    |    34 +
 .../plans/logical/LogicalPlanVisitor.scala         |    99 +
 .../sql/catalyst/plans/logical/MergeRows.scala     |   135 +
 .../sql/catalyst/plans/logical/MultiResult.scala   |    30 +
 .../sql/catalyst/plans/logical/PlanHelper.scala    |    65 +
 .../plans/logical/QueryPlanConstraints.scala       |   143 +
 .../plans/logical/ScriptTransformation.scala       |    52 +
 .../plans/logical/SqlScriptingLogicalPlans.scala   |   456 +
 .../sql/catalyst/plans/logical/Statistics.scala    |   245 +
 .../logical/StreamSourceAwareLogicalPlan.scala     |    35 +
 .../plans/logical/basicLogicalOperators.scala      |  2298 ++
 .../sql/catalyst/plans/logical/cteOperators.scala  |   296 +
 .../spark/sql/catalyst/plans/logical/hints.scala   |   245 +
 .../spark/sql/catalyst/plans/logical/object.scala  |   905 +
 .../plans/logical/pythonLogicalOperators.scala     |   449 +
 .../sql/catalyst/plans/logical/statements.scala    |   198 +
 .../statsEstimation/AggregateEstimation.scala      |    76 +
 .../statsEstimation/BasicStatsPlanVisitor.scala    |   112 +
 .../logical/statsEstimation/EstimationUtils.scala  |   424 +
 .../logical/statsEstimation/FilterEstimation.scala |   927 +
 .../logical/statsEstimation/JoinEstimation.scala   |   359 +
 .../logical/statsEstimation/LogicalPlanStats.scala |    50 +
 .../statsEstimation/ProjectEstimation.scala        |    41 +
 .../SizeInBytesOnlyStatsPlanVisitor.scala          |   188 +
 .../logical/statsEstimation/UnionEstimation.scala  |   128 +
 .../logical/statsEstimation/ValueInterval.scala    |    96 +
 .../plans/logical/v2AlterTableCommands.scala       |   375 +
 .../sql/catalyst/plans/logical/v2Commands.scala    |  2061 ++
 .../apache/spark/sql/catalyst/plans/package.scala  |    24 +
 .../catalyst/plans/physical/broadcastMode.scala    |    46 +
 .../sql/catalyst/plans/physical/partitioning.scala |  1049 +
 .../catalyst/rules/QueryExecutionMetering.scala    |   128 +
 .../org/apache/spark/sql/catalyst/rules/Rule.scala |    36 +
 .../spark/sql/catalyst/rules/RuleExecutor.scala    |   326 +
 .../sql/catalyst/rules/RuleIdCollection.scala      |   239 +
 .../apache/spark/sql/catalyst/rules/package.scala  |    23 +
 .../catalyst/streaming/StreamingRelationV2.scala   |    79 +
 .../streaming/StreamingSourceIdentifyingName.scala |    67 +
 .../sql/catalyst/streaming/WriteToStream.scala     |    48 +
 .../streaming/WriteToStreamStatement.scala         |    70 +
 .../apache/spark/sql/catalyst/trees/TreeNode.scala |  1403 +
 .../spark/sql/catalyst/trees/TreePatternBits.scala |    77 +
 .../spark/sql/catalyst/trees/TreePatterns.scala    |   200 +
 .../apache/spark/sql/catalyst/trees/package.scala  |    51 +
 .../spark/sql/catalyst/types/DataTypeUtils.scala   |   331 +
 .../sql/catalyst/types/PhysicalDataType.scala      |   431 +
 .../sql/catalyst/util/ArrayBasedMapBuilder.scala   |   156 +
 .../sql/catalyst/util/ArrayBasedMapData.scala      |   149 +
 .../apache/spark/sql/catalyst/util/ArrayData.scala |   209 +
 .../sql/catalyst/util/BadRecordException.scala     |   130 +
 .../spark/sql/catalyst/util/CharVarcharUtils.scala |   335 +
 .../spark/sql/catalyst/util/CharsetProvider.scala  |    82 +
 .../sql/catalyst/util/CompressionCodecs.scala      |    69 +
 .../spark/sql/catalyst/util/DateTimeUtils.scala    |  1062 +
 .../util/EvaluateUnresolvedInlineTable.scala       |   132 +
 .../sql/catalyst/util/FailureSafeParser.scala      |    94 +
 .../spark/sql/catalyst/util/GeneratedColumn.scala  |   209 +
 .../spark/sql/catalyst/util/GenericArrayData.scala |   164 +
 .../catalyst/util/HyperLogLogPlusPlusHelper.scala  |   385 +
 .../spark/sql/catalyst/util/IdentityColumn.scala   |    78 +
 .../util/InternalRowComparableWrapper.scala        |   136 +
 .../sql/catalyst/util/IntervalMathUtils.scala      |    49 +
 .../spark/sql/catalyst/util/IntervalUtils.scala    |   991 +
 .../apache/spark/sql/catalyst/util/MapData.scala   |    47 +
 .../spark/sql/catalyst/util/NumberConverter.scala  |   229 +
 .../apache/spark/sql/catalyst/util/ParseMode.scala |    66 +
 .../sql/catalyst/util/QuantileSummaries.scala      |   391 +
 .../sql/catalyst/util/RandomIndicesGenerator.scala |    45 +
 .../sql/catalyst/util/RandomUUIDGenerator.scala    |    43 +
 .../sql/catalyst/util/ReplaceDataProjections.scala |    24 +
 .../catalyst/util/ResolveDefaultColumnsUtil.scala  |   631 +
 .../catalyst/util/ResolveTableConstraints.scala    |    55 +
 .../spark/sql/catalyst/util/RowDeltaUtils.scala    |    32 +
 .../spark/sql/catalyst/util/SQLKeywordUtils.scala  |    53 +
 .../spark/sql/catalyst/util/SQLOrderingUtil.scala  |    41 +
 .../spark/sql/catalyst/util/StringKeyHashMap.scala |    49 +
 .../spark/sql/catalyst/util/StringUtils.scala      |   417 +
 .../spark/sql/catalyst/util/ThetaSketchUtils.scala |    86 +
 .../spark/sql/catalyst/util/ToNumberParser.scala   |   884 +
 .../spark/sql/catalyst/util/TupleSketchUtils.scala |   330 +
 .../apache/spark/sql/catalyst/util/TypeUtils.scala |   152 +
 .../spark/sql/catalyst/util/UDTUtilsImpl.scala     |    26 +
 .../spark/sql/catalyst/util/UTF8StringUtils.scala  |    54 +
 .../spark/sql/catalyst/util/UnsafeRowUtils.scala   |   212 +
 .../sql/catalyst/util/V2ExpressionBuilder.scala    |   486 +
 .../sql/catalyst/util/WriteDeltaProjections.scala  |    25 +
 .../apache/spark/sql/catalyst/util/package.scala   |   253 +
 .../catalyst/xml/IndentingXMLStreamWriter.scala    |   194 +
 .../sql/catalyst/xml/StaxXMLRecordReader.scala     |   132 +
 .../spark/sql/catalyst/xml/StaxXmlGenerator.scala  |   430 +
 .../spark/sql/catalyst/xml/StaxXmlParser.scala     |  1303 +
 .../sql/catalyst/xml/StaxXmlParserUtils.scala      |   283 +
 .../spark/sql/catalyst/xml/ValidatorUtil.scala     |    78 +
 .../spark/sql/catalyst/xml/XmlInferSchema.scala    |   716 +
 .../apache/spark/sql/catalyst/xml/XmlOptions.scala |   257 +
 .../sql/connector/catalog/CatalogManager.scala     |   166 +
 .../catalog/CatalogNotFoundException.scala         |    32 +
 .../sql/connector/catalog/CatalogV2Implicits.scala |   255 +
 .../sql/connector/catalog/CatalogV2Util.scala      |   759 +
 .../spark/sql/connector/catalog/Catalogs.scala     |   109 +
 .../sql/connector/catalog/LookupCatalog.scala      |   162 +
 .../spark/sql/connector/catalog/V1Table.scala      |   120 +
 .../spark/sql/connector/catalog/V2TableUtil.scala  |   139 +
 .../connector/distributions/distributions.scala    |    60 +
 .../sql/connector/expressions/expressions.scala    |   432 +
 .../sql/connector/write/LogicalWriteInfoImpl.scala |    49 +
 .../sql/connector/write/MergeSummaryImpl.scala     |    33 +
 .../connector/write/PhysicalWriteInfoImpl.scala    |    20 +
 .../write/RowLevelOperationInfoImpl.scala          |    25 +
 .../connector/write/RowLevelOperationTable.scala   |    52 +
 .../spark/sql/errors/QueryCompilationErrors.scala  |  4542 +++
 .../apache/spark/sql/errors/QueryErrorsBase.scala  |    77 +
 .../spark/sql/errors/QueryExecutionErrors.scala    |  3261 ++
 .../spark/sql/errors/SqlScriptingErrors.scala      |   272 +
 .../sql/exceptions/SqlScriptingException.scala     |    53 +
 .../sql/execution/QueryExecutionException.scala    |    21 +
 .../apache/spark/sql/execution/RowIterator.scala   |    92 +
 .../spark/sql/execution/arrow/ArrowWriter.scala    |   574 +
 .../sql/execution/arrow/ArrowWriterWrapper.scala   |    88 +
 .../datasources/v2/DataSourceV2Implicits.scala     |   134 +
 .../datasources/v2/DataSourceV2Relation.scala      |   356 +
 .../spark/sql/internal/ReadOnlySQLConf.scala       |    67 +
 .../org/apache/spark/sql/internal/SQLConf.scala    |  8300 ++++++
 .../apache/spark/sql/internal/StaticSQLConf.scala  |   317 +
 .../spark/sql/internal/connector/ColumnImpl.scala  |    32 +
 .../connector/ExpressionWithToString.scala         |    25 +
 .../connector/PartitionOffsetWithIndex.scala       |    24 +
 .../sql/internal/connector/PredicateUtils.scala    |   141 +
 .../connector/ProcedureParameterImpl.scala         |    30 +
 .../internal/connector/SimpleTableProvider.scala   |    51 +
 .../sql/internal/connector/SupportsMetadata.scala  |    26 +
 .../SupportsPushDownCatalystFilters.scala          |    42 +
 .../SupportsStreamingUpdateAsAppend.scala          |    33 +
 .../internal/connector/ToStringSQLBuilder.scala    |    43 +
 .../spark/sql/internal/connector/V1Function.scala  |    95 +
 .../internal/connector/VariantExtractionImpl.scala |    45 +
 .../sql/metricview/logical/metricViewNodes.scala   |    53 +
 .../spark/sql/metricview/serde/JsonUtils.scala     |    34 +
 .../sql/metricview/serde/MetricViewCanonical.scala |   170 +
 .../sql/metricview/serde/MetricViewFactory.scala   |    69 +
 .../sql/metricview/serde/MetricViewSerDeBase.scala |   189 +
 .../sql/metricview/serde/MetricViewSerDeV01.scala  |    83 +
 .../sql/metricview/util/MetricViewPlanner.scala    |    83 +
 .../org/apache/spark/sql/sources/filters.scala     |   467 +
 .../spark/sql/types/DataTypeExpression.scala       |    90 +
 .../org/apache/spark/sql/types/numerics.scala      |   183 +
 .../scala/org/apache/spark/sql/types/package.scala |    24 +
 .../apache/spark/sql/util/PartitioningUtils.scala  |   139 +
 .../org/apache/spark/sql/util/SQLOpenHashSet.scala |   164 +
 .../org/apache/spark/sql/util/SchemaUtils.scala    |   533 +
 .../spark/sql/catalyst/JavaTypeInferenceBeans.java |   138 +
 .../sql/catalyst/expressions/HiveHasherSuite.java  |   127 +
 .../expressions/RowBasedKeyValueBatchSuite.java    |   359 +
 .../expressions/TestThrowExceptionMethod.java      |    39 +
 .../spark/sql/catalyst/expressions/XXH64Suite.java |   199 +
 .../catalyst/util/geo/WkbErrorHandlingTest.java    |   399 +
 .../sql/catalyst/util/geo/WkbLineStringTest.java   |   142 +
 .../catalyst/util/geo/WkbMultiGeometryTest.java    |   208 +
 .../spark/sql/catalyst/util/geo/WkbPointTest.java  |   190 +
 .../sql/catalyst/util/geo/WkbPolygonTest.java      |   175 +
 .../util/geo/WkbReaderWriterAdvancedTest.java      |  1135 +
 .../util/geo/WkbReaderWriterGeometryModelTest.java |   400 +
 .../spark/sql/catalyst/util/geo/WkbTestBase.java   |    54 +
 .../sql/connector/catalog/CatalogLoadingSuite.java |   237 +
 .../sql/streaming/JavaGroupStateTimeoutSuite.java  |    36 +
 .../spark/sql/streaming/JavaOutputModeSuite.java   |    34 +
 .../resources/ansi-sql-2016-reserved-keywords.txt  |   403 +
 sql/catalyst/src/test/resources/log4j2.properties  |    31 +
 .../spark/sql/CalendarIntervalBenchmark.scala      |    75 +
 .../scala/org/apache/spark/sql/HashBenchmark.scala |   142 +
 .../apache/spark/sql/HashByteArrayBenchmark.scala  |    99 +
 .../org/apache/spark/sql/RandomDataGenerator.scala |   440 +
 .../spark/sql/RandomDataGeneratorSuite.scala       |   161 +
 .../scala/org/apache/spark/sql/RowJsonSuite.scala  |   148 +
 .../test/scala/org/apache/spark/sql/RowTest.scala  |   150 +
 .../spark/sql/SparkBloomFilterBenchmark.scala      |   318 +
 .../spark/sql/UnsafeProjectionBenchmark.scala      |   134 +
 .../sql/catalyst/CatalystTypeConvertersSuite.scala |   457 +
 .../spark/sql/catalyst/DistributionSuite.scala     |   394 +
 .../spark/sql/catalyst/EscapePathBenchmark.scala   |   124 +
 .../sql/catalyst/JavaTypeInferenceSuite.scala      |   304 +
 .../spark/sql/catalyst/OrderedFiltersSuite.scala   |    27 +
 .../sql/catalyst/QueryPlanningTrackerSuite.scala   |   114 +
 .../spark/sql/catalyst/SQLKeywordSuite.scala       |   211 +
 .../spark/sql/catalyst/ScalaReflectionSuite.scala  |   624 +
 .../spark/sql/catalyst/SchemaPruningTest.scala     |    43 +
 .../spark/sql/catalyst/ShuffleSpecSuite.scala      |   545 +
 .../spark/sql/catalyst/StructFiltersSuite.scala    |   136 +
 .../sql/catalyst/analysis/AnalysisErrorSuite.scala |  1332 +
 .../analysis/AnalysisExceptionPositionSuite.scala  |   103 +
 .../analysis/AnalysisExternalCatalogSuite.scala    |    77 +
 .../sql/catalyst/analysis/AnalysisSuite.scala      |  1952 ++
 .../spark/sql/catalyst/analysis/AnalysisTest.scala |   241 +
 .../catalyst/analysis/AnsiTypeCoercionSuite.scala  |  1106 +
 .../spark/sql/catalyst/analysis/CatalogSuite.scala |    54 +
 .../CreateTablePartitioningValidationSuite.scala   |   156 +
 .../spark/sql/catalyst/analysis/DSLHintSuite.scala |    46 +
 .../catalyst/analysis/DecimalPrecisionSuite.scala  |   292 +
 .../analysis/ExpressionTypeCheckingSuite.scala     |   819 +
 .../catalyst/analysis/ExtractGeneratorSuite.scala  |    42 +
 .../GroupByOrdinalsRepeatedAnalysisSuite.scala     |    86 +
 .../catalyst/analysis/LookupFunctionsSuite.scala   |   159 +
 .../analysis/NameStreamingSourcesSuite.scala       |   394 +
 .../analysis/NamedParameterFunctionSuite.scala     |   275 +
 .../analysis/NamedStreamingRelationSuite.scala     |   120 +
 .../analysis/PullOutNondeterministicSuite.scala    |    56 +
 .../catalyst/analysis/ResolveAliasesSuite.scala    |   139 +
 .../analysis/ResolveEncodersInUDFSuite.scala       |    96 +
 .../analysis/ResolveEventTimeWatermarkSuite.scala  |   120 +
 .../analysis/ResolveGroupingAnalyticsSuite.scala   |   328 +
 .../sql/catalyst/analysis/ResolveHintsSuite.scala  |   376 +
 .../analysis/ResolveInlineTablesSuite.scala        |   164 +
 .../analysis/ResolveLambdaVariablesSuite.scala     |   103 +
 .../analysis/ResolveNaturalJoinSuite.scala         |   157 +
 .../analysis/ResolveRecursiveCTESuite.scala        |   119 +
 .../catalyst/analysis/ResolveSubquerySuite.scala   |   302 +
 .../sql/catalyst/analysis/ResolveUnionSuite.scala  |    75 +
 .../analysis/ResolvedUuidExpressionsSuite.scala    |    75 +
 .../analysis/StreamingJoinHelperSuite.scala        |   154 +
 .../catalyst/analysis/TableLookupCacheSuite.scala  |   110 +
 .../sql/catalyst/analysis/TestRelations.scala      |    85 +
 .../sql/catalyst/analysis/TypeCoercionSuite.scala  |  1865 ++
 .../analysis/UnsupportedOperationsSuite.scala      |  1248 +
 .../catalyst/analysis/V2WriteAnalysisSuite.scala   |  1519 +
 .../resolver/CoercesExpressionTypesSuite.scala     |    83 +
 .../LimitLikeExpressionValidatorSuite.scala        |   109 +
 .../resolver/ResolutionValidatorSuite.scala        |   260 +
 .../TimezoneAwareExpressionResolverSuite.scala     |    94 +
 .../catalog/ExternalCatalogEventSuite.scala        |   320 +
 .../catalyst/catalog/ExternalCatalogSuite.scala    |  1185 +
 .../catalog/ExternalCatalogUtilsSuite.scala        |    68 +
 .../catalyst/catalog/InMemoryCatalogSuite.scala    |    31 +
 .../sql/catalyst/catalog/SessionCatalogSuite.scala |  2262 ++
 .../spark/sql/catalyst/csv/CSVExprUtilsSuite.scala |   113 +
 .../sql/catalyst/csv/CSVInferSchemaSuite.scala     |   276 +
 .../sql/catalyst/csv/UnivocityParserSuite.scala    |   417 +
 .../encoders/EncoderErrorMessageSuite.scala        |    99 +
 .../catalyst/encoders/EncoderResolutionSuite.scala |   322 +
 .../catalyst/encoders/ExpressionEncoderSuite.scala |  1015 +
 .../sql/catalyst/encoders/RowEncoderSuite.scala    |   559 +
 .../expressions/ArithmeticExpressionSuite.scala    |  1129 +
 .../catalyst/expressions/AttributeMapSuite.scala   |   278 +
 .../expressions/AttributeResolutionSuite.scala     |   149 +
 .../catalyst/expressions/AttributeSetSuite.scala   |   121 +
 .../catalyst/expressions/BindReferencesSuite.scala |    43 +
 .../expressions/BitmapExpressionUtilsSuite.scala   |    95 +
 .../expressions/BitwiseExpressionsSuite.scala      |   237 +
 .../expressions/CallMethodViaReflectionSuite.scala |   174 +
 .../catalyst/expressions/CanonicalizeSuite.scala   |   495 +
 .../sql/catalyst/expressions/CastSuiteBase.scala   |  1895 ++
 .../expressions/CastWithAnsiOffSuite.scala         |   933 +
 .../catalyst/expressions/CastWithAnsiOnSuite.scala |   903 +
 .../catalyst/expressions/CodeGenerationSuite.scala |   656 +
 ...CodeGeneratorWithInterpretedFallbackSuite.scala |   124 +
 .../expressions/CollationExpressionSuite.scala     |   267 +
 .../CollationRegexpExpressionsSuite.scala          |   216 +
 .../expressions/CollectionExpressionsSuite.scala   |  3114 ++
 .../catalyst/expressions/ComplexTypeSuite.scala    |   665 +
 .../expressions/ConditionalExpressionSuite.scala   |   304 +
 .../expressions/ConstraintExpressionSuite.scala    |    45 +
 .../catalyst/expressions/CsvExpressionsSuite.scala |   314 +
 .../expressions/DateExpressionsSuite.scala         |  2316 ++
 .../expressions/DecimalExpressionSuite.scala       |    97 +
 .../expressions/DynamicPruningSubquerySuite.scala  |    89 +
 .../sql/catalyst/expressions/ExprIdSuite.scala     |    50 +
 .../expressions/ExpressionEvalHelper.scala         |   576 +
 .../expressions/ExpressionEvalHelperSuite.scala    |    99 +
 .../expressions/ExpressionImplUtilsSuite.scala     |   424 +
 .../expressions/ExpressionSQLBuilderSuite.scala    |   186 +
 .../catalyst/expressions/ExpressionSetSuite.scala  |   257 +
 .../ExtractPredicatesWithinOutputSetSuite.scala    |    79 +
 .../expressions/GeneratorExpressionSuite.scala     |   117 +
 .../expressions/HashExpressionsSuite.scala         |   916 +
 .../sql/catalyst/expressions/HexBenchmark.scala    |    90 +
 .../spark/sql/catalyst/expressions/HexSuite.scala  |    40 +
 .../expressions/HigherOrderFunctionsSuite.scala    |   888 +
 .../expressions/IntegralLiteralTestUtils.scala     |    42 +
 .../expressions/IntervalExpressionsSuite.scala     |   676 +
 .../expressions/JsonExpressionsSuite.scala         |   976 +
 .../expressions/LiteralExpressionSuite.scala       |   658 +
 .../catalyst/expressions/LiteralGenerator.scala    |   221 +
 .../expressions/MathExpressionsSuite.scala         |  1013 +
 .../expressions/MiscExpressionsSuite.scala         |   105 +
 .../expressions/MutableProjectionSuite.scala       |   231 +
 .../expressions/NamedExpressionSuite.scala         |    67 +
 .../catalyst/expressions/NonFoldableLiteral.scala  |    53 +
 .../expressions/NondeterministicSuite.scala        |    34 +
 .../expressions/NullExpressionsSuite.scala         |   194 +
 .../expressions/ObjectExpressionsSuite.scala       |   848 +
 .../sql/catalyst/expressions/OrderingSuite.scala   |   191 +
 .../sql/catalyst/expressions/PredicateSuite.scala  |   769 +
 .../sql/catalyst/expressions/RandomSuite.scala     |    67 +
 .../expressions/RegexpExpressionsSuite.scala       |   653 +
 .../sql/catalyst/expressions/ScalaUDFSuite.scala   |   124 +
 .../catalyst/expressions/SchemaPruningSuite.scala  |   146 +
 .../catalyst/expressions/SelectedFieldSuite.scala  |   584 +
 .../expressions/SortOrderExpressionsSuite.scala    |   104 +
 .../expressions/StringExpressionsSuite.scala       |  2139 ++
 .../SubExprEvaluationRuntimeSuite.scala            |   132 +
 .../SubexpressionEliminationSuite.scala            |   526 +
 .../expressions/TimeExpressionsSuite.scala         |   698 +
 .../sql/catalyst/expressions/TimeWindowSuite.scala |   194 +
 .../catalyst/expressions/ToPrettyStringSuite.scala |   144 +
 .../sql/catalyst/expressions/TryCastSuite.scala    |   129 +
 .../sql/catalyst/expressions/TryEvalSuite.scala    |   103 +
 .../expressions/UnsafeRowConverterSuite.scala      |   706 +
 .../expressions/UnwrapUDTExpressionSuite.scala     |    40 +
 .../expressions/V2ExpressionUtilsSuite.scala       |    40 +
 .../catalyst/expressions/XmlExpressionsSuite.scala |   522 +
 .../aggregate/AggregateExpressionSuite.scala       |   149 +
 .../ApproxCountDistinctForIntervalsSuite.scala     |   270 +
 .../expressions/aggregate/ApproxTopKSuite.scala    |   408 +
 .../aggregate/ApproximatePercentileSuite.scala     |   440 +
 .../aggregate/CentralMomentAggSuite.scala          |    77 +
 .../aggregate/CountMinSketchAggSuite.scala         |   239 +
 .../expressions/aggregate/CovarianceAggSuite.scala |    39 +
 .../aggregate/DatasketchesHllSketchSuite.scala     |   156 +
 .../aggregate/DeclarativeAggregateEvaluator.scala  |    60 +
 .../expressions/aggregate/FirstLastTestSuite.scala |   125 +
 .../aggregate/HistogramNumericSuite.scala          |   292 +
 .../aggregate/HyperLogLogPlusPlusSuite.scala       |   178 +
 .../expressions/aggregate/PercentileSuite.scala    |   427 +
 .../expressions/aggregate/ProductAggSuite.scala    |   156 +
 .../aggregate/TestWithAndWithoutCodegen.scala      |    35 +
 .../aggregate/ThetasketchesAggSuite.scala          |   175 +
 .../expressions/aggregate/VectorAggSuite.scala     |   497 +
 .../codegen/BufferHolderSparkSubmitSuite.scala     |    91 +
 .../expressions/codegen/BufferHolderSuite.scala    |    45 +
 .../expressions/codegen/CodeBlockSuite.scala       |   212 +
 .../expressions/codegen/CodeFormatterSuite.scala   |   298 +
 .../codegen/CodegenExpressionCachingSuite.scala    |   111 +
 .../CodegenSubexpressionEliminationSuite.scala     |    73 +
 .../expressions/codegen/ExprValueSuite.scala       |    38 +
 .../codegen/GenerateUnsafeProjectionSuite.scala    |   134 +
 .../GenerateUnsafeRowJoinerBitsetSuite.scala       |   152 +
 .../codegen/GenerateUnsafeRowJoinerSuite.scala     |   245 +
 .../codegen/GeneratedProjectionSuite.scala         |   297 +
 .../codegen/UnsafeArrayWriterSuite.scala           |    40 +
 .../expressions/codegen/UnsafeRowWriterSuite.scala |    97 +
 .../variant/VariantExpressionEvalUtilsSuite.scala  |   180 +
 .../variant/VariantExpressionSuite.scala           |  1056 +
 .../xml/ReusableStringReaderSuite.scala            |   103 +
 .../expressions/xml/UDFXPathUtilSuite.scala        |   122 +
 .../expressions/xml/XPathExpressionSuite.scala     |   221 +
 .../sql/catalyst/json/JacksonGeneratorSuite.scala  |   140 +
 .../sql/catalyst/json/JacksonParserSuite.scala     |    70 +
 .../spark/sql/catalyst/json/JsonFiltersSuite.scala |    28 +
 .../sql/catalyst/json/JsonInferSchemaSuite.scala   |   123 +
 .../optimizer/AggregateOptimizeSuite.scala         |   221 +
 .../BinaryComparisonSimplificationSuite.scala      |   265 +
 .../optimizer/BooleanSimplificationSuite.scala     |   366 +
 .../optimizer/CheckCartesianProductsSuite.scala    |    91 +
 .../catalyst/optimizer/CollapseProjectSuite.scala  |   367 +
 .../optimizer/CollapseRepartitionSuite.scala       |   225 +
 .../catalyst/optimizer/CollapseWindowSuite.scala   |   171 +
 .../catalyst/optimizer/ColumnPruningSuite.scala    |   485 +
 .../catalyst/optimizer/CombineConcatsSuite.scala   |    87 +
 .../catalyst/optimizer/CombiningLimitsSuite.scala  |   307 +
 .../optimizer/ComputeCurrentTimeSuite.scala        |   282 +
 .../catalyst/optimizer/ConstantFoldingSuite.scala  |   462 +
 .../optimizer/ConstantPropagationSuite.scala       |   217 +
 .../optimizer/ConvertToLocalRelationSuite.scala    |   110 +
 .../CrossJoinArrayContainsToInnerJoinSuite.scala   |   270 +
 .../optimizer/DecimalAggregatesSuite.scala         |   123 +
 .../optimizer/DecorrelateInnerQuerySuite.scala     |   739 +
 .../optimizer/EliminateAggregateFilterSuite.scala  |    89 +
 .../optimizer/EliminateDistinctSuite.scala         |    92 +
 .../optimizer/EliminateMapObjectsSuite.scala       |    64 +
 .../catalyst/optimizer/EliminateOffsetsSuite.scala |   100 +
 .../optimizer/EliminateSerializationSuite.scala    |    81 +
 .../EliminateSortsBeforeRepartitionSuite.scala     |   214 +
 .../catalyst/optimizer/EliminateSortsSuite.scala   |   494 +
 .../optimizer/EliminateSubqueryAliasesSuite.scala  |    61 +
 .../optimizer/EliminateWindowPartitionsSuite.scala |    80 +
 .../ExtractPythonUDFFromJoinConditionSuite.scala   |   203 +
 .../optimizer/FilterPushdownOnePassSuite.scala     |   183 +
 .../catalyst/optimizer/FilterPushdownSuite.scala   |  1542 +
 .../optimizer/FoldablePropagationSuite.scala       |   228 +
 .../optimizer/GenerateOptimizationSuite.scala      |   133 +
 .../InferFiltersFromConstraintsSuite.scala         |   402 +
 .../optimizer/InferFiltersFromGenerateSuite.scala  |   124 +
 .../optimizer/InferWindowGroupLimitSuite.scala     |   357 +
 .../sql/catalyst/optimizer/InlineCTESuite.scala    |    42 +
 .../catalyst/optimizer/JoinOptimizationSuite.scala |   129 +
 .../optimizer/JoinSelectionHelperSuite.scala       |   159 +
 .../optimizer/LeftSemiAntiJoinPushDownSuite.scala  |   494 +
 .../optimizer/LikeSimplificationSuite.scala        |   312 +
 .../catalyst/optimizer/LimitPushdownSuite.scala    |   359 +
 .../LimitPushdownThroughWindowSuite.scala          |   212 +
 .../catalyst/optimizer/MergeSubplansSuite.scala    |   724 +
 .../optimizer/NestedColumnAliasingSuite.scala      |   899 +
 .../NormalizeFloatingPointNumbersSuite.scala       |   136 +
 .../optimizer/NullDownPropagationSuite.scala       |    58 +
 .../optimizer/ObjectSerializerPruningSuite.scala   |   145 +
 .../catalyst/optimizer/OptimizeCsvExprsSuite.scala |    83 +
 .../sql/catalyst/optimizer/OptimizeInSuite.scala   |   350 +
 .../optimizer/OptimizeJoinConditionSuite.scala     |    49 +
 .../optimizer/OptimizeJsonExprsSuite.scala         |   327 +
 .../optimizer/OptimizeLimitZeroSuite.scala         |   110 +
 .../optimizer/OptimizeOneRowPlanSuite.scala        |   104 +
 .../OptimizeOneRowRelationSubquerySuite.scala      |   204 +
 .../sql/catalyst/optimizer/OptimizeRandSuite.scala |   176 +
 .../optimizer/OptimizeWindowFunctionsSuite.scala   |   103 +
 .../optimizer/OptimizeWithFieldsSuite.scala        |   150 +
 .../optimizer/OptimizerExtendableSuite.scala       |    57 +
 .../catalyst/optimizer/OptimizerLoggingSuite.scala |   172 +
 .../optimizer/OptimizerRuleExclusionSuite.scala    |   136 +
 .../OptimizerStructuralIntegrityCheckerSuite.scala |   140 +
 .../sql/catalyst/optimizer/OptimizerSuite.scala    |   337 +
 .../optimizer/OuterJoinEliminationSuite.scala      |   318 +
 .../optimizer/PropagateEmptyRelationSuite.scala    |   423 +
 .../sql/catalyst/optimizer/PruneFiltersSuite.scala |   211 +
 .../PullupCorrelatedPredicatesSuite.scala          |   177 +
 .../optimizer/PushFoldableIntoBranchesSuite.scala  |   383 +
 .../optimizer/PushProjectThroughUnionSuite.scala   |    54 +
 .../PushProjectionThroughLimitAndOffsetSuite.scala |   162 +
 .../optimizer/ReassignLambdaVariableIDSuite.scala  |    65 +
 .../optimizer/RemoveNoopOperatorsSuite.scala       |    69 +
 .../catalyst/optimizer/RemoveNoopUnionSuite.scala  |    84 +
 .../optimizer/RemoveRedundantAggregatesSuite.scala |   311 +
 .../RemoveRedundantAliasAndProjectSuite.scala      |   272 +
 .../ReorderAssociativeOperatorSuite.scala          |   126 +
 .../ReplaceNullWithFalseInPredicateSuite.scala     |   564 +
 .../catalyst/optimizer/ReplaceOperatorSuite.scala  |   299 +
 .../catalyst/optimizer/RewriteAsOfJoinSuite.scala  |   289 +
 .../optimizer/RewriteDistinctAggregatesSuite.scala |   128 +
 .../catalyst/optimizer/RewriteSubquerySuite.scala  |    99 +
 .../optimizer/RewriteWithExpressionSuite.scala     |   503 +
 .../sql/catalyst/optimizer/SetOperationSuite.scala |   416 +
 .../catalyst/optimizer/SimpleTestOptimizer.scala   |    34 +
 .../catalyst/optimizer/SimplifyCastsSuite.scala    |   141 +
 .../optimizer/SimplifyConditionalSuite.scala       |   296 +
 .../SimplifyDateTimeConversionsSuite.scala         |    69 +
 .../SimplifyStringCaseConversionSuite.scala        |    90 +
 .../optimizer/SpecialDatetimeValuesSuite.scala     |   101 +
 .../catalyst/optimizer/TransposeWindowSuite.scala  |   164 +
 .../optimizer/TypedFilterOptimizationSuite.scala   |   190 +
 .../UnwrapCastInBinaryComparisonSuite.scala        |   559 +
 .../sql/catalyst/optimizer/complexTypesSuite.scala |   819 +
 .../joinReorder/JoinReorderPlanTestBase.scala      |    82 +
 .../optimizer/joinReorder/JoinReorderSuite.scala   |   387 +
 .../StarJoinCostBasedReorderSuite.scala            |   416 +
 .../joinReorder/StarJoinReorderSuite.scala         |   584 +
 .../sql/catalyst/parser/CastingSyntaxSuite.scala   |   103 +
 .../spark/sql/catalyst/parser/DDLParserSuite.scala |  3413 +++
 .../sql/catalyst/parser/DataTypeParserSuite.scala  |   240 +
 .../sql/catalyst/parser/ErrorParserSuite.scala     |   340 +
 .../catalyst/parser/ExpressionParserSuite.scala    |  1258 +
 .../parser/ParameterSubstitutionSuite.scala        |   136 +
 .../sql/catalyst/parser/ParserUtilsSuite.scala     |   299 +
 .../sql/catalyst/parser/PlanParserSuite.scala      |  2166 ++
 .../catalyst/parser/SqlScriptingParserSuite.scala  |  3488 +++
 .../parser/StreamRelationParserSuite.scala         |   635 +
 .../parser/TableIdentifierParserSuite.scala        |   383 +
 .../catalyst/parser/TableSchemaParserSuite.scala   |   100 +
 .../sql/catalyst/parser/UnpivotParserSuite.scala   |   340 +
 .../catalyst/planning/PhysicalOperationSuite.scala |   113 +
 .../plans/ConstraintPropagationSuite.scala         |   431 +
 .../spark/sql/catalyst/plans/JoinTypesTest.scala   |    79 +
 .../sql/catalyst/plans/LogicalPlanSuite.scala      |   183 +
 .../sql/catalyst/plans/NormalizePlanSuite.scala    |   303 +
 .../apache/spark/sql/catalyst/plans/PlanTest.scala |   107 +
 .../spark/sql/catalyst/plans/QueryPlanSuite.scala  |   188 +
 .../spark/sql/catalyst/plans/SQLHelper.scala       |    67 +
 .../spark/sql/catalyst/plans/SameResultSuite.scala |    90 +
 .../plans/logical/AnalysisHelperSuite.scala        |   159 +
 .../plans/logical/DistinctKeyVisitorSuite.scala    |   206 +
 .../plans/logical/LogicalPlanIntegritySuite.scala  |    53 +
 .../statsEstimation/AggregateEstimationSuite.scala |   187 +
 .../BasicStatsEstimationSuite.scala                |   451 +
 .../statsEstimation/FilterEstimationSuite.scala    |   949 +
 .../statsEstimation/JoinEstimationSuite.scala      |   576 +
 .../statsEstimation/ProjectEstimationSuite.scala   |   178 +
 .../statsEstimation/StatsEstimationTestBase.scala  |    84 +
 .../statsEstimation/UnionEstimationSuite.scala     |   263 +
 .../streaming/InternalOutputModesSuite.scala       |    47 +
 .../sql/catalyst/trees/RuleExecutorSuite.scala     |   193 +
 .../sql/catalyst/trees/SQLQueryContextSuite.scala  |    49 +
 .../spark/sql/catalyst/trees/TreeNodeSuite.scala   |  1147 +
 .../catalyst/util/ArrayBasedMapBuilderSuite.scala  |   200 +
 .../catalyst/util/ArrayDataIndexedSeqSuite.scala   |   102 +
 .../catalyst/util/CaseInsensitiveMapSuite.scala    |    55 +
 .../spark/sql/catalyst/util/ComplexDataSuite.scala |   141 +
 .../sql/catalyst/util/DateFormatterSuite.scala     |   212 +
 .../util/DateTimeFormatterHelperSuite.scala        |    96 +
 .../sql/catalyst/util/DateTimeTestUtils.scala      |   126 +
 .../sql/catalyst/util/DateTimeUtilsSuite.scala     |  1528 +
 .../sql/catalyst/util/DatetimeFormatterSuite.scala |   135 +
 .../catalyst/util/GenericArrayDataBenchmark.scala  |    94 +
 .../sql/catalyst/util/GeographyExecutionSuite.java |   240 +
 .../sql/catalyst/util/GeometryExecutionSuite.java  |   249 +
 .../InternalRowComparableWrapperBenchmark.scala    |    77 +
 .../sql/catalyst/util/IntervalUtilsSuite.scala     |   902 +
 .../spark/sql/catalyst/util/MetadataSuite.scala    |    97 +
 .../sql/catalyst/util/NumberConverterSuite.scala   |   107 +
 .../catalyst/util/PhysicalAggregationSuite.scala   |    55 +
 .../sql/catalyst/util/QuantileSummariesSuite.scala |   231 +
 .../sql/catalyst/util/QuotingUtilsSuite.scala      |    43 +
 .../catalyst/util/RandomUUIDGeneratorSuite.scala   |    57 +
 .../sql/catalyst/util/RebaseDateTimeSuite.scala    |   445 +
 .../sql/catalyst/util/SQLOrderingUtilSuite.scala   |    75 +
 .../spark/sql/catalyst/util/StUtilsSuite.java      |   169 +
 .../spark/sql/catalyst/util/StringUtilsSuite.scala |   285 +
 .../sql/catalyst/util/ThetaSketchUtilsSuite.scala  |   112 +
 .../sql/catalyst/util/TimeFormatterSuite.scala     |   182 +
 .../catalyst/util/TimestampFormatterSuite.scala    |   562 +
 .../sql/catalyst/util/TupleSketchUtilsSuite.scala  |   513 +
 .../spark/sql/catalyst/util/TypeUtilsSuite.scala   |    47 +
 .../spark/sql/catalyst/util/UnsafeArraySuite.scala |   212 +
 .../spark/sql/catalyst/util/UnsafeMapSuite.scala   |    64 +
 .../sql/catalyst/util/UnsafeRowUtilsSuite.scala    |   160 +
 .../connector/catalog/CatalogManagerSuite.scala    |   139 +
 .../spark/sql/connector/catalog/CatalogSuite.scala |  1380 +
 .../sql/connector/catalog/CatalogV2UtilSuite.scala |    40 +
 .../sql/connector/catalog/ConstraintSuite.scala    |   146 +
 .../connector/catalog/EnumTypeSetBenchmark.scala   |   179 +
 .../catalog/InMemoryAtomicPartitionTable.scala     |    95 +
 .../sql/connector/catalog/InMemoryBaseTable.scala  |  1054 +
 .../sql/connector/catalog/InMemoryCatalog.scala    |    72 +
 .../connector/catalog/InMemoryPartitionTable.scala |   140 +
 .../catalog/InMemoryPartitionTableCatalog.scala    |    72 +
 .../catalog/InMemoryRowLevelOperationTable.scala   |   240 +
 .../InMemoryRowLevelOperationTableCatalog.scala    |    69 +
 .../sql/connector/catalog/InMemoryTable.scala      |   263 +
 .../connector/catalog/InMemoryTableCatalog.scala   |   382 +
 .../catalog/InMemoryTableWithV2Filter.scala        |   188 +
 .../catalog/InMemoryTableWithV2FilterCatalog.scala |    49 +
 .../sql/connector/catalog/LookupCatalogSuite.scala |   179 +
 .../catalog/StagingInMemoryTableCatalog.scala      |   131 +
 .../SupportsAtomicPartitionManagementSuite.scala   |   178 +
 .../catalog/SupportsPartitionManagementSuite.scala |   269 +
 .../sql/connector/catalog/V2TableUtilSuite.scala   |   656 +
 .../catalog/functions/AggregateFunctionSuite.scala |   148 +
 .../expressions/TransformExtractorSuite.scala      |   255 +
 .../metricview/serde/MetricViewFactorySuite.scala  |   373 +
 .../spark/sql/sources/FakeCommandRunner.scala      |    30 +
 .../org/apache/spark/sql/types/DataTypeSuite.scala |  1450 +
 .../apache/spark/sql/types/DataTypeTestUtils.scala |   126 +
 .../types/DataTypeWriteCompatibilitySuite.scala    |   733 +
 .../org/apache/spark/sql/types/DecimalSuite.scala  |   467 +
 .../org/apache/spark/sql/types/MetadataSuite.scala |   168 +
 .../apache/spark/sql/types/StructTypeSuite.scala   |   879 +
 .../scala/org/apache/spark/sql/types/TestUDT.scala |   154 +
 .../apache/spark/sql/util/ArrowUtilsSuite.scala    |   177 +
 .../sql/util/CaseInsensitiveStringMapSuite.scala   |   111 +
 .../apache/spark/sql/util/SchemaUtilsSuite.scala   |   113 +
 sql/connect/README.md                              |    68 +
 sql/connect/bin/spark-connect                      |    44 +
 sql/connect/bin/spark-connect-build                |    33 +
 sql/connect/bin/spark-connect-scala-client         |    77 +
 .../bin/spark-connect-scala-client-classpath       |    34 +
 sql/connect/bin/spark-connect-shell                |    39 +
 sql/connect/client/jdbc/pom.xml                    |   140 +
 .../connect/client/jdbc/SparkConnectDriver.java    |    31 +
 .../resources/META-INF/services/java.sql.Driver    |    18 +
 .../jdbc/NonRegisteringSparkConnectDriver.scala    |    49 +
 .../client/jdbc/SparkConnectConnection.scala       |   280 +
 .../client/jdbc/SparkConnectDatabaseMetaData.scala |   819 +
 .../client/jdbc/SparkConnectResultSet.scala        |   789 +
 .../jdbc/SparkConnectResultSetMetaData.scala       |    84 +
 .../client/jdbc/SparkConnectStatement.scala        |   267 +
 .../connect/client/jdbc/util/JdbcErrorUtils.scala  |    56 +
 .../connect/client/jdbc/util/JdbcTypeUtils.scala   |   152 +
 .../jdbc/src/test/resources/log4j2.properties      |    42 +
 .../jdbc/SparkConnectDatabaseMetaDataSuite.scala   |   804 +
 .../client/jdbc/SparkConnectDriverSuite.scala      |    43 +
 .../jdbc/SparkConnectJdbcDataTypeSuite.scala       |   792 +
 .../client/jdbc/SparkConnectResultSetSuite.scala   |   189 +
 .../client/jdbc/SparkConnectStatementSuite.scala   |   119 +
 .../sql/connect/client/jdbc/test/JdbcHelper.scala  |    48 +
 sql/connect/client/jvm/pom.xml                     |   289 +
 .../apache/spark/sql/application/ConnectRepl.scala |   193 +
 .../sql/connect/client/AmmoniteClassFinder.scala   |    47 +
 .../org/apache/spark/sql/JavaEncoderSuite.java     |   114 +
 .../jvm/src/test/resources/StubClassDummyUdf.scala |    56 +
 .../jvm/src/test/resources/TestHelloV2_2.13.jar    |   Bin 0 -> 4118 bytes
 .../jvm/src/test/resources/log4j2.properties       |    42 +
 sql/connect/client/jvm/src/test/resources/udf2.13  |   Bin 0 -> 1630 bytes
 .../client/jvm/src/test/resources/udf2.13.jar      |   Bin 0 -> 5674 bytes
 .../apache/spark/sql/DataFrameSubquerySuite.scala  |   862 +
 .../sql/DataFrameTableValuedFunctionsSuite.scala   |   544 +
 .../spark/sql/GeographyConnectDataFrameSuite.scala |   127 +
 .../spark/sql/GeometryConnectDataFrameSuite.scala  |   133 +
 .../apache/spark/sql/PlanGenerationTestSuite.scala |  3616 +++
 .../org/apache/spark/sql/SQLExpressionsSuite.scala |    47 +
 .../spark/sql/application/ReplE2ESuite.scala       |   566 +
 .../apache/spark/sql/connect/CatalogSuite.scala    |   368 +
 .../apache/spark/sql/connect/CheckpointSuite.scala |   126 +
 .../sql/connect/ClientDataFrameStatSuite.scala     |   276 +
 .../spark/sql/connect/ClientDatasetSuite.scala     |   187 +
 .../spark/sql/connect/ClientE2ETestSuite.scala     |  2044 ++
 .../connect/ColumnNodeToProtoConverterSuite.scala  |   482 +
 .../apache/spark/sql/connect/ColumnTestSuite.scala |   266 +
 .../sql/connect/DataFrameNaFunctionSuite.scala     |   408 +
 .../apache/spark/sql/connect/DataFrameSuite.scala  |    81 +
 .../spark/sql/connect/FunctionTestSuite.scala      |   285 +
 .../KeyValueGroupedDatasetE2ETestSuite.scala       |   821 +
 .../spark/sql/connect/MergeIntoE2ETestSuite.scala  |   188 +
 .../spark/sql/connect/SQLImplicitsTestSuite.scala  |   190 +
 ...kSessionBuilderImplementationBindingSuite.scala |    36 +
 .../spark/sql/connect/SparkSessionCloneSuite.scala |    64 +
 .../spark/sql/connect/SparkSessionE2ESuite.scala   |   503 +
 .../spark/sql/connect/SparkSessionSuite.scala      |   278 +
 .../spark/sql/connect/StaticProcedureSuite.scala   |    31 +
 .../spark/sql/connect/StubbingTestSuite.scala      |    37 +
 .../sql/connect/UDFClassLoadingE2ESuite.scala      |    89 +
 .../sql/connect/UnsupportedFeaturesSuite.scala     |   112 +
 .../connect/UserDefinedFunctionE2ETestSuite.scala  |   593 +
 .../sql/connect/UserDefinedFunctionSuite.scala     |    91 +
 .../spark/sql/connect/client/ArtifactSuite.scala   |   356 +
 .../CheckConnectJvmClientCompatibility.scala       |   427 +
 .../sql/connect/client/ClassFinderSuite.scala      |    55 +
 .../SparkConnectClientBuilderParseTestSuite.scala  |   135 +
 .../client/SparkConnectClientRetriesSuite.scala    |   278 +
 .../connect/client/SparkConnectClientSuite.scala   |   932 +
 .../connect/client/arrow/ArrowEncoderSuite.scala   |  1313 +
 .../streaming/ClientStreamingQuerySuite.scala      |   874 +
 .../FlatMapGroupsWithStateStreamingSuite.scala     |   233 +
 .../streaming/StreamingQueryProgressSuite.scala    |   229 +
 .../streaming/TransformWithStateConnectSuite.scala |   740 +
 .../spark/sql/connect/test/ConnectFunSuite.scala   |    61 +
 .../sql/connect/test/IntegrationTestUtils.scala    |   165 +
 .../apache/spark/sql/connect/test/QueryTest.scala  |   407 +
 .../sql/connect/test/RemoteSparkSession.scala      |   265 +
 .../apache/spark/sql/connect/test/SQLHelper.scala  |   157 +
 sql/connect/common/README.md                       |     5 +
 sql/connect/common/pom.xml                         |   219 +
 sql/connect/common/src/main/buf.gen.yaml           |    36 +
 sql/connect/common/src/main/buf.work.yaml          |    19 +
 sql/connect/common/src/main/protobuf/buf.yaml      |    25 +
 .../src/main/protobuf/spark/connect/base.proto     |  1275 +
 .../src/main/protobuf/spark/connect/catalog.proto  |   243 +
 .../src/main/protobuf/spark/connect/commands.proto |   554 +
 .../src/main/protobuf/spark/connect/common.proto   |   179 +
 .../protobuf/spark/connect/example_plugins.proto   |    42 +
 .../main/protobuf/spark/connect/expressions.proto  |   557 +
 .../src/main/protobuf/spark/connect/ml.proto       |   147 +
 .../main/protobuf/spark/connect/ml_common.proto    |    64 +
 .../main/protobuf/spark/connect/pipelines.proto    |   307 +
 .../main/protobuf/spark/connect/relations.proto    |  1257 +
 .../src/main/protobuf/spark/connect/types.proto    |   227 +
 .../org/apache/spark/sql/connect/Catalog.scala     |   729 +
 .../connect/ConnectClientUnsupportedErrors.scala   |    53 +
 .../spark/sql/connect/ConnectConversions.scala     |    88 +
 .../spark/sql/connect/ConnectProtoUtils.scala      |    56 +
 .../spark/sql/connect/DataFrameNaFunctions.scala   |   214 +
 .../apache/spark/sql/connect/DataFrameReader.scala |   225 +
 .../spark/sql/connect/DataFrameStatFunctions.scala |   145 +
 .../apache/spark/sql/connect/DataFrameWriter.scala |   158 +
 .../spark/sql/connect/DataFrameWriterV2.scala      |   125 +
 .../spark/sql/connect/DataStreamReader.scala       |   161 +
 .../spark/sql/connect/DataStreamWriter.scala       |   201 +
 .../org/apache/spark/sql/connect/Dataset.scala     |  1571 +
 .../spark/sql/connect/KeyValueGroupedDataset.scala |   800 +
 .../apache/spark/sql/connect/MergeIntoWriter.scala |   136 +
 .../sql/connect/RelationalGroupedDataset.scala     |   182 +
 .../apache/spark/sql/connect/RuntimeConfig.scala   |   153 +
 .../org/apache/spark/sql/connect/SQLContext.scala  |   317 +
 .../apache/spark/sql/connect/SQLImplicits.scala    |    40 +
 .../apache/spark/sql/connect/SessionCleaner.scala  |    48 +
 .../apache/spark/sql/connect/SparkSession.scala    |  1131 +
 .../apache/spark/sql/connect/StreamingQuery.scala  |   173 +
 .../sql/connect/StreamingQueryListenerBus.scala    |   155 +
 .../spark/sql/connect/StreamingQueryManager.scala  |   120 +
 .../spark/sql/connect/TableValuedFunction.scala    |   109 +
 .../apache/spark/sql/connect/UDFRegistration.scala |    68 +
 .../apache/spark/sql/connect/UdfToProtoUtils.scala |   109 +
 .../UserDefinedAggregateFunctionWrapper.scala      |    94 +
 .../spark/sql/connect/client/ArtifactManager.scala |   482 +
 .../spark/sql/connect/client/ClassFinder.scala     |    55 +
 .../client/CustomSparkConnectBlockingStub.scala    |   148 +
 .../connect/client/CustomSparkConnectStub.scala    |    35 +
 .../ExecutePlanResponseReattachableIterator.scala  |   345 +
 .../connect/client/GrpcExceptionConverter.scala    |   522 +
 .../sql/connect/client/GrpcRetryHandler.scala      |   238 +
 .../sql/connect/client/ResponseValidator.scala     |   142 +
 .../spark/sql/connect/client/RetryPolicy.scala     |   195 +
 .../sql/connect/client/SparkConnectClient.scala    |  1122 +
 .../connect/client/SparkConnectClientParser.scala  |   104 +
 .../sql/connect/client/SparkConnectStubState.scala |    41 +
 .../spark/sql/connect/client/SparkResult.scala     |   452 +
 .../apache/spark/sql/connect/client/ToStub.scala   |    23 +
 .../connect/client/arrow/ArrowDeserializer.scala   |   611 +
 .../connect/client/arrow/ArrowEncoderUtils.scala   |    68 +
 .../sql/connect/client/arrow/ArrowSerializer.scala |   621 +
 .../connect/client/arrow/ArrowVectorReader.scala   |   288 +
 .../client/arrow/GeospatialArrowSerDe.scala        |   101 +
 .../client/arrow/ScalaCollectionUtils.scala        |    37 +
 .../apache/spark/sql/connect/client/package.scala  |    28 +
 .../spark/sql/connect/columnNodeSupport.scala      |   319 +
 .../spark/sql/connect/common/Abbreviator.scala     |   112 +
 .../connect/common/DataTypeProtoConverter.scala    |   393 +
 .../sql/connect/common/ForeachWriterPacket.scala   |    34 +
 .../sql/connect/common/InvalidPlanInput.scala      |    56 +
 .../common/LiteralValueProtoConverter.scala        |   663 +
 .../spark/sql/connect/common/ProtoDataTypes.scala  |   136 +
 .../sql/connect/common/ProtoSpecializedArray.scala |   172 +
 .../spark/sql/connect/common/ProtoUtils.scala      |    79 +
 .../common/StorageLevelProtoConverter.scala        |    46 +
 .../connect/common/StreamingListenerPacket.scala   |    32 +
 .../spark/sql/connect/common/UdfPacket.scala       |    71 +
 .../apache/spark/sql/connect/common/UdfUtils.scala |   589 +
 .../sql/connect/common/config/ConnectCommon.scala  |    24 +
 .../org/apache/spark/sql/connect/package.scala     |    22 +
 .../src/test/resources/artifact-tests/Hello.class  |   Bin 0 -> 5671 bytes
 .../test/resources/artifact-tests/crc/Hello.txt    |     1 +
 .../test/resources/artifact-tests/crc/README.md    |     5 +
 .../resources/artifact-tests/crc/junitLargeJar.txt |    12 +
 .../artifact-tests/crc/smallClassFile.txt          |     1 +
 .../artifact-tests/crc/smallClassFileDup.txt       |     1 +
 .../test/resources/artifact-tests/crc/smallJar.txt |     1 +
 .../resources/artifact-tests/junitLargeJar.jar     |   Bin 0 -> 384581 bytes
 .../resources/artifact-tests/smallClassFile.class  |   Bin 0 -> 424 bytes
 .../artifact-tests/smallClassFileDup.class         |   Bin 0 -> 424 bytes
 .../src/test/resources/artifact-tests/smallJar.jar |   Bin 0 -> 787 bytes
 .../src/test/resources/protobuf-tests/common.desc  |    12 +
 .../explain-results/alias_string.explain           |     2 +
 .../explain-results/alias_symbol.explain           |     2 +
 .../query-tests/explain-results/apply.explain      |     2 +
 .../query-tests/explain-results/as_string.explain  |     2 +
 .../query-tests/explain-results/as_symbol.explain  |     2 +
 .../explain-results/between_expr.explain           |     3 +
 .../query-tests/explain-results/coalesce.explain   |     2 +
 .../query-tests/explain-results/col.explain        |     2 +
 .../query-tests/explain-results/colRegex.explain   |     2 +
 .../query-tests/explain-results/column_add.explain |     2 +
 .../explain-results/column_alias.explain           |     2 +
 .../query-tests/explain-results/column_and.explain |     2 +
 .../explain-results/column_apply.explain           |     2 +
 .../explain-results/column_as_multi.explain        |     3 +
 .../column_as_with_metadata.explain                |     2 +
 .../query-tests/explain-results/column_asc.explain |     2 +
 .../explain-results/column_asc_nulls_first.explain |     2 +
 .../explain-results/column_asc_nulls_last.explain  |     2 +
 .../explain-results/column_between.explain         |     2 +
 .../explain-results/column_bitwiseAND.explain      |     2 +
 .../explain-results/column_bitwiseOR.explain       |     2 +
 .../explain-results/column_bitwiseXOR.explain      |     2 +
 .../explain-results/column_cast.explain            |     2 +
 .../explain-results/column_contains.explain        |     2 +
 .../explain-results/column_desc.explain            |     2 +
 .../column_desc_nulls_first.explain                |     2 +
 .../explain-results/column_desc_nulls_last.explain |     2 +
 .../explain-results/column_divide.explain          |     2 +
 .../explain-results/column_dropFields.explain      |     2 +
 .../explain-results/column_endsWith.explain        |     2 +
 .../explain-results/column_eqNullSafe.explain      |     2 +
 .../explain-results/column_equals.explain          |     2 +
 .../query-tests/explain-results/column_geq.explain |     2 +
 .../explain-results/column_getField.explain        |     2 +
 .../explain-results/column_getItem.explain         |     2 +
 .../query-tests/explain-results/column_gt.explain  |     2 +
 .../explain-results/column_ilike.explain           |     2 +
 .../explain-results/column_isNaN.explain           |     2 +
 .../explain-results/column_isNotNull.explain       |     2 +
 .../explain-results/column_isNull.explain          |     2 +
 .../explain-results/column_isin.explain            |     2 +
 .../query-tests/explain-results/column_leq.explain |     2 +
 .../explain-results/column_like.explain            |     2 +
 .../query-tests/explain-results/column_lt.explain  |     2 +
 .../explain-results/column_modulo.explain          |     2 +
 .../explain-results/column_multiply.explain        |     2 +
 .../query-tests/explain-results/column_not.explain |     2 +
 .../explain-results/column_not_equals.explain      |     2 +
 .../query-tests/explain-results/column_or.explain  |     2 +
 .../explain-results/column_rlike.explain           |     2 +
 .../explain-results/column_star.explain            |     2 +
 .../column_star_with_target.explain                |     2 +
 .../explain-results/column_startsWith.explain      |     2 +
 .../explain-results/column_substr.explain          |     2 +
 .../explain-results/column_subtract.explain        |     2 +
 .../explain-results/column_try_cast.explain        |     2 +
 .../explain-results/column_unary_minus.explain     |     2 +
 .../explain-results/column_when_otherwise.explain  |     2 +
 .../explain-results/column_withField.explain       |     2 +
 .../query-tests/explain-results/crossJoin.explain  |     3 +
 .../query-tests/explain-results/crosstab.explain   |     5 +
 .../explain-results/csv_from_dataset.explain       |     1 +
 .../explain-results/cube_column.explain            |     4 +
 .../explain-results/cube_string.explain            |     4 +
 .../query-tests/explain-results/describe.explain   |     6 +
 .../query-tests/explain-results/distinct.explain   |     2 +
 .../query-tests/explain-results/drop.explain       |     2 +
 .../explain-results/dropDuplicates.explain         |     2 +
 .../dropDuplicates_names_array.explain             |     2 +
 .../dropDuplicates_names_seq.explain               |     2 +
 .../explain-results/dropDuplicates_varargs.explain |     2 +
 .../explain-results/drop_multiple_column.explain   |     2 +
 .../explain-results/drop_multiple_strings.explain  |     2 +
 .../explain-results/drop_single_column.explain     |     2 +
 .../explain-results/drop_single_string.explain     |     2 +
 .../query-tests/explain-results/except.explain     |     3 +
 .../query-tests/explain-results/exceptAll.explain  |     3 +
 .../explain-results/expression_extension.explain   |     2 +
 .../expression_extension_deprecated.explain        |     2 +
 .../query-tests/explain-results/fill.explain       |     2 +
 .../query-tests/explain-results/filter.explain     |     2 +
 .../explain-results/filter_expr.explain            |     2 +
 .../query-tests/explain-results/freqItems.explain  |     2 +
 .../explain-results/from_avro_with_options.explain |     2 +
 .../from_avro_without_options.explain              |     2 +
 ..._protobuf_messageClassName_descFilePath.explain |     2 +
 ...f_messageClassName_descFilePath_options.explain |     2 +
 .../explain-results/function_abs.explain           |     2 +
 .../explain-results/function_acos.explain          |     2 +
 .../explain-results/function_acosh.explain         |     2 +
 .../explain-results/function_add_months.explain    |     2 +
 .../explain-results/function_aes_decrypt.explain   |     2 +
 .../function_aes_decrypt_with_mode.explain         |     2 +
 .../function_aes_decrypt_with_mode_padding.explain |     2 +
 ...ction_aes_decrypt_with_mode_padding_aad.explain |     2 +
 .../explain-results/function_aes_encrypt.explain   |     2 +
 .../function_aes_encrypt_with_mode.explain         |     2 +
 .../function_aes_encrypt_with_mode_padding.explain |     2 +
 ...nction_aes_encrypt_with_mode_padding_iv.explain |     2 +
 ...on_aes_encrypt_with_mode_padding_iv_aad.explain |     2 +
 .../explain-results/function_aggregate.explain     |     2 +
 .../function_aggregate_with_finish_lambda.explain  |     2 +
 .../explain-results/function_any.explain           |     2 +
 .../function_any_value_with_ignore_nulls.explain   |     2 +
 .../function_any_value_with_respect_nulls.explain  |     2 +
 .../function_approx_count_distinct.explain         |     2 +
 .../function_approx_count_distinct_rsd.explain     |     2 +
 .../function_approx_percentile.explain             |     2 +
 .../explain-results/function_array.explain         |     2 +
 .../explain-results/function_array_agg.explain     |     2 +
 .../explain-results/function_array_append.explain  |     2 +
 .../explain-results/function_array_compact.explain |     2 +
 .../function_array_contains.explain                |     2 +
 .../function_array_distinct.explain                |     2 +
 .../explain-results/function_array_except.explain  |     2 +
 .../explain-results/function_array_insert.explain  |     2 +
 .../function_array_intersect.explain               |     2 +
 .../explain-results/function_array_join.explain    |     2 +
 ...nction_array_join_with_null_replacement.explain |     2 +
 .../explain-results/function_array_max.explain     |     2 +
 .../explain-results/function_array_min.explain     |     2 +
 .../function_array_position.explain                |     2 +
 .../explain-results/function_array_prepend.explain |     2 +
 .../explain-results/function_array_remove.explain  |     2 +
 .../explain-results/function_array_repeat.explain  |     2 +
 .../explain-results/function_array_size.explain    |     2 +
 .../explain-results/function_array_sort.explain    |     2 +
 .../function_array_sort_with_comparator.explain    |     2 +
 .../explain-results/function_array_union.explain   |     2 +
 .../function_arrays_overlap.explain                |     2 +
 .../explain-results/function_arrays_zip.explain    |     2 +
 .../explain-results/function_asc.explain           |     2 +
 .../function_asc_nulls_first.explain               |     2 +
 .../function_asc_nulls_last.explain                |     2 +
 .../explain-results/function_ascii.explain         |     2 +
 .../explain-results/function_asin.explain          |     2 +
 .../explain-results/function_asinh.explain         |     2 +
 .../function_assert_true_with_message.explain      |     2 +
 .../explain-results/function_atan.explain          |     2 +
 .../explain-results/function_atan2.explain         |     2 +
 .../explain-results/function_atanh.explain         |     2 +
 .../explain-results/function_avg.explain           |     2 +
 .../explain-results/function_base64.explain        |     2 +
 .../explain-results/function_bin.explain           |     2 +
 .../explain-results/function_bit_and.explain       |     2 +
 .../explain-results/function_bit_count.explain     |     2 +
 .../explain-results/function_bit_get.explain       |     2 +
 .../explain-results/function_bit_length.explain    |     2 +
 .../explain-results/function_bit_or.explain        |     2 +
 .../explain-results/function_bit_xor.explain       |     2 +
 .../function_bitmap_bit_position.explain           |     2 +
 .../function_bitmap_bucket_number.explain          |     2 +
 .../function_bitmap_construct_agg.explain          |     2 +
 .../explain-results/function_bitmap_count.explain  |     2 +
 .../explain-results/function_bitmap_or_agg.explain |     2 +
 .../explain-results/function_bitwise_not.explain   |     2 +
 .../explain-results/function_bool_and.explain      |     2 +
 .../explain-results/function_bool_or.explain       |     2 +
 .../explain-results/function_bround.explain        |     2 +
 .../explain-results/function_btrim.explain         |     2 +
 ...nction_btrim_with_specified_trim_string.explain |     2 +
 .../explain-results/function_bucket.explain        |     2 +
 .../explain-results/function_call_function.explain |     2 +
 .../function_call_function_orphaned.explain        |     2 +
 .../explain-results/function_cardinality.explain   |     2 +
 .../explain-results/function_ceil.explain          |     2 +
 .../explain-results/function_ceil_scale.explain    |     2 +
 .../explain-results/function_ceiling.explain       |     2 +
 .../explain-results/function_ceiling_scale.explain |     2 +
 .../explain-results/function_char.explain          |     2 +
 .../explain-results/function_char_length.explain   |     2 +
 .../function_character_length.explain              |     2 +
 .../explain-results/function_chr.explain           |     2 +
 .../explain-results/function_coalesce.explain      |     2 +
 .../explain-results/function_col.explain           |     2 +
 .../explain-results/function_collate.explain       |     2 +
 .../explain-results/function_collation.explain     |     2 +
 .../explain-results/function_collect_list.explain  |     2 +
 .../explain-results/function_collect_set.explain   |     2 +
 .../explain-results/function_concat.explain        |     2 +
 .../explain-results/function_concat_ws.explain     |     2 +
 .../explain-results/function_contains.explain      |     2 +
 .../explain-results/function_conv.explain          |     2 +
 ..._convert_timezone_with_source_time_zone.explain |     2 +
 ...nvert_timezone_without_source_time_zone.explain |     2 +
 .../explain-results/function_corr.explain          |     2 +
 .../explain-results/function_cos.explain           |     2 +
 .../explain-results/function_cosh.explain          |     2 +
 .../explain-results/function_cot.explain           |     2 +
 .../explain-results/function_count.explain         |     2 +
 .../explain-results/function_countDistinct.explain |     2 +
 .../explain-results/function_count_if.explain      |     4 +
 .../function_count_min_sketch.explain              |     2 +
 .../explain-results/function_count_typed.explain   |     2 +
 .../explain-results/function_covar_pop.explain     |     2 +
 .../explain-results/function_covar_samp.explain    |     2 +
 .../explain-results/function_crc32.explain         |     2 +
 .../explain-results/function_csc.explain           |     2 +
 .../explain-results/function_cume_dist.explain     |     5 +
 .../explain-results/function_curdate.explain       |     2 +
 .../function_current_catalog.explain               |     2 +
 .../function_current_database.explain              |     2 +
 .../explain-results/function_current_date.explain  |     2 +
 .../function_current_schema.explain                |     2 +
 .../function_current_timestamp.explain             |     2 +
 .../function_current_timezone.explain              |     2 +
 .../explain-results/function_current_user.explain  |     2 +
 .../explain-results/function_date_add.explain      |     2 +
 .../explain-results/function_date_diff.explain     |     2 +
 .../explain-results/function_date_format.explain   |     2 +
 .../function_date_from_unix_date.explain           |     2 +
 .../explain-results/function_date_part.explain     |     2 +
 .../explain-results/function_date_sub.explain      |     2 +
 .../explain-results/function_date_trunc.explain    |     2 +
 .../explain-results/function_dateadd.explain       |     2 +
 .../explain-results/function_datediff.explain      |     2 +
 .../explain-results/function_datepart.explain      |     2 +
 .../explain-results/function_day.explain           |     2 +
 .../explain-results/function_dayname.explain       |     2 +
 .../explain-results/function_dayofmonth.explain    |     2 +
 .../explain-results/function_dayofweek.explain     |     2 +
 .../explain-results/function_dayofyear.explain     |     2 +
 .../explain-results/function_days.explain          |     2 +
 .../explain-results/function_decode.explain        |     2 +
 .../explain-results/function_degrees.explain       |     2 +
 .../explain-results/function_dense_rank.explain    |     5 +
 .../explain-results/function_desc.explain          |     2 +
 .../function_desc_nulls_first.explain              |     2 +
 .../function_desc_nulls_last.explain               |     2 +
 .../query-tests/explain-results/function_e.explain |     2 +
 .../explain-results/function_element_at.explain    |     2 +
 .../explain-results/function_elt.explain           |     2 +
 .../explain-results/function_encode.explain        |     2 +
 .../explain-results/function_endswith.explain      |     2 +
 .../explain-results/function_equal_null.explain    |     2 +
 .../explain-results/function_every.explain         |     2 +
 .../explain-results/function_exists.explain        |     2 +
 .../explain-results/function_exp.explain           |     2 +
 .../explain-results/function_explode.explain       |     3 +
 .../explain-results/function_explode_outer.explain |     3 +
 .../explain-results/function_expm1.explain         |     2 +
 .../explain-results/function_expr.explain          |     2 +
 .../explain-results/function_extract.explain       |     2 +
 .../explain-results/function_factorial.explain     |     2 +
 .../explain-results/function_filter.explain        |     2 +
 .../function_filter_with_pair_input.explain        |     2 +
 .../explain-results/function_find_in_set.explain   |     2 +
 .../function_first_value_with_ignore_nulls.explain |     2 +
 ...function_first_value_with_respect_nulls.explain |     2 +
 .../function_first_with_ignore_nulls.explain       |     2 +
 .../function_first_with_respect_nulls.explain      |     2 +
 .../explain-results/function_flatten.explain       |     2 +
 .../explain-results/function_floor.explain         |     2 +
 .../explain-results/function_floor_scale.explain   |     2 +
 .../explain-results/function_forall.explain        |     2 +
 .../explain-results/function_format_number.explain |     2 +
 .../explain-results/function_from_csv.explain      |     2 +
 .../explain-results/function_from_json.explain     |     2 +
 .../function_from_json_orphaned.explain            |     2 +
 .../function_from_json_with_json_schema.explain    |     2 +
 .../explain-results/function_from_unixtime.explain |     2 +
 .../function_from_utc_timestamp.explain            |     2 +
 .../explain-results/function_from_xml.explain      |     2 +
 .../function_from_xml_with_json_schema.explain     |     2 +
 .../explain-results/function_get.explain           |     2 +
 .../function_get_json_object.explain               |     2 +
 .../explain-results/function_getbit.explain        |     2 +
 .../explain-results/function_greatest.explain      |     2 +
 .../explain-results/function_hash.explain          |     2 +
 .../explain-results/function_hex.explain           |     2 +
 .../function_histogram_numeric.explain             |     2 +
 .../explain-results/function_hour.explain          |     2 +
 .../explain-results/function_hours.explain         |     2 +
 .../explain-results/function_hypot.explain         |     2 +
 .../explain-results/function_ifnull.explain        |     2 +
 .../explain-results/function_ilike.explain         |     2 +
 .../function_ilike_with_escape.explain             |     2 +
 .../explain-results/function_initcap.explain       |     2 +
 .../explain-results/function_inline.explain        |     3 +
 .../explain-results/function_inline_outer.explain  |     3 +
 .../function_input_file_block_length.explain       |     2 +
 .../function_input_file_block_start.explain        |     2 +
 .../function_input_file_name.explain               |     2 +
 .../function_is_variant_null.explain               |     2 +
 .../explain-results/function_isnan.explain         |     2 +
 .../explain-results/function_isnotnull.explain     |     2 +
 .../explain-results/function_isnull.explain        |     2 +
 .../explain-results/function_java_method.explain   |     2 +
 .../function_json_array_length.explain             |     2 +
 .../function_json_object_keys.explain              |     2 +
 .../explain-results/function_json_tuple.explain    |     3 +
 .../explain-results/function_kurtosis.explain      |     2 +
 .../explain-results/function_lag.explain           |     5 +
 .../explain-results/function_last_day.explain      |     2 +
 .../function_last_value_with_ignore_nulls.explain  |     2 +
 .../function_last_value_with_respect_nulls.explain |     2 +
 .../function_last_with_ignore_nulls.explain        |     2 +
 .../function_last_with_respect_nulls.explain       |     2 +
 .../explain-results/function_lcase.explain         |     2 +
 .../explain-results/function_lead.explain          |     5 +
 .../explain-results/function_least.explain         |     2 +
 .../explain-results/function_left.explain          |     2 +
 .../explain-results/function_len.explain           |     2 +
 .../explain-results/function_length.explain        |     2 +
 .../explain-results/function_levenshtein.explain   |     2 +
 .../function_levenshtein_with_threshold.explain    |     2 +
 .../explain-results/function_like.explain          |     2 +
 .../function_like_with_escape.explain              |     2 +
 .../explain-results/function_lit.explain           |     2 +
 .../explain-results/function_lit_array.explain     |     2 +
 .../explain-results/function_ln.explain            |     2 +
 .../function_localtimestamp.explain                |     2 +
 .../explain-results/function_locate.explain        |     2 +
 .../function_locate_with_pos.explain               |     2 +
 .../explain-results/function_log.explain           |     2 +
 .../explain-results/function_log10.explain         |     2 +
 .../explain-results/function_log1p.explain         |     2 +
 .../explain-results/function_log2.explain          |     2 +
 .../explain-results/function_log_with_base.explain |     2 +
 .../explain-results/function_lower.explain         |     2 +
 .../explain-results/function_lpad.explain          |     2 +
 .../explain-results/function_lpad_binary.explain   |     2 +
 .../explain-results/function_ltrim.explain         |     2 +
 .../function_ltrim_with_pattern.explain            |     2 +
 .../explain-results/function_make_date.explain     |     2 +
 .../function_make_dt_interval.explain              |     2 +
 .../function_make_dt_interval_days.explain         |     2 +
 .../function_make_dt_interval_days_hours.explain   |     2 +
 ...nction_make_dt_interval_days_hours_mins.explain |     2 +
 ...n_make_dt_interval_days_hours_mins_secs.explain |     2 +
 .../explain-results/function_make_interval.explain |     2 +
 .../function_make_interval_years.explain           |     2 +
 .../function_make_interval_years_months.explain    |     2 +
 ...nction_make_interval_years_months_weeks.explain |     2 +
 ...n_make_interval_years_months_weeks_days.explain |     2 +
 ..._interval_years_months_weeks_days_hours.explain |     2 +
 ...rval_years_months_weeks_days_hours_mins.explain |     2 +
 ...years_months_weeks_days_hours_mins_secs.explain |     2 +
 ...nction_make_timestamp_ltz_with_timezone.explain |     2 +
 ...ion_make_timestamp_ltz_without_timezone.explain |     2 +
 .../function_make_timestamp_ntz.explain            |     2 +
 .../function_make_timestamp_with_timezone.explain  |     2 +
 ...unction_make_timestamp_without_timezone.explain |     2 +
 .../function_make_ym_interval.explain              |     2 +
 .../function_make_ym_interval_years.explain        |     2 +
 .../function_make_ym_interval_years_months.explain |     2 +
 .../explain-results/function_map.explain           |     2 +
 .../explain-results/function_map_concat.explain    |     2 +
 .../function_map_contains_key.explain              |     2 +
 .../explain-results/function_map_entries.explain   |     2 +
 .../explain-results/function_map_filter.explain    |     2 +
 .../function_map_from_arrays.explain               |     2 +
 .../function_map_from_entries.explain              |     2 +
 .../explain-results/function_map_keys.explain      |     2 +
 .../explain-results/function_map_values.explain    |     2 +
 .../explain-results/function_map_zip_with.explain  |     2 +
 .../explain-results/function_mask.explain          |     2 +
 .../function_mask_with_specific_upperChar.explain  |     2 +
 ..._mask_with_specific_upperChar_lowerChar.explain |     2 +
 ..._specific_upperChar_lowerChar_digitChar.explain |     2 +
 ...upperChar_lowerChar_digitChar_otherChar.explain |     2 +
 .../explain-results/function_max.explain           |     2 +
 .../explain-results/function_max_by.explain        |     2 +
 .../explain-results/function_md5.explain           |     2 +
 .../explain-results/function_median.explain        |     2 +
 .../explain-results/function_min.explain           |     2 +
 .../explain-results/function_min_by.explain        |     2 +
 .../explain-results/function_minute.explain        |     2 +
 .../explain-results/function_mode.explain          |     2 +
 .../explain-results/function_mode_orphaned.explain |     2 +
 .../function_monotonically_increasing_id.explain   |     2 +
 .../explain-results/function_month.explain         |     2 +
 .../explain-results/function_monthname.explain     |     2 +
 .../explain-results/function_months.explain        |     2 +
 .../function_months_between.explain                |     2 +
 .../function_months_between_with_roundoff.explain  |     2 +
 .../explain-results/function_named_struct.explain  |     2 +
 .../explain-results/function_nanvl.explain         |     2 +
 .../explain-results/function_negate.explain        |     2 +
 .../explain-results/function_negative.explain      |     2 +
 .../explain-results/function_next_day.explain      |     2 +
 .../explain-results/function_now.explain           |     2 +
 .../explain-results/function_nth_value.explain     |     5 +
 .../explain-results/function_ntile.explain         |     5 +
 .../explain-results/function_nullif.explain        |     2 +
 .../explain-results/function_nvl.explain           |     2 +
 .../explain-results/function_nvl2.explain          |     2 +
 .../explain-results/function_octet_length.explain  |     2 +
 .../explain-results/function_overlay.explain       |     2 +
 .../function_overlay_with_len.explain              |     2 +
 .../explain-results/function_parse_json.explain    |     2 +
 .../explain-results/function_parse_url.explain     |     2 +
 .../function_parse_url_with_key.explain            |     2 +
 .../explain-results/function_percent_rank.explain  |     5 +
 .../function_percentile_approx.explain             |     2 +
 .../function_percentile_with_frequency.explain     |     2 +
 .../function_percentile_without_frequency.explain  |     2 +
 .../explain-results/function_pi.explain            |     2 +
 .../explain-results/function_pmod.explain          |     2 +
 .../explain-results/function_posexplode.explain    |     3 +
 .../function_posexplode_outer.explain              |     3 +
 .../explain-results/function_position.explain      |     2 +
 .../function_position_with_start.explain           |     2 +
 .../explain-results/function_positive.explain      |     2 +
 .../explain-results/function_pow.explain           |     2 +
 .../explain-results/function_power.explain         |     2 +
 .../explain-results/function_printf.explain        |     2 +
 .../explain-results/function_product.explain       |     2 +
 .../explain-results/function_quarter.explain       |     2 +
 .../explain-results/function_radians.explain       |     2 +
 .../explain-results/function_raise_error.explain   |     2 +
 .../function_rand_with_seed.explain                |     2 +
 .../function_randn_with_seed.explain               |     2 +
 .../function_random_with_seed.explain              |     2 +
 .../explain-results/function_rank.explain          |     5 +
 .../explain-results/function_reduce.explain        |     2 +
 .../explain-results/function_reflect.explain       |     2 +
 .../explain-results/function_regexp.explain        |     2 +
 .../explain-results/function_regexp_count.explain  |     2 +
 .../function_regexp_extract.explain                |     2 +
 ...gexp_extract_all_with_regex_group_index.explain |     2 +
 ...p_extract_all_without_regex_group_index.explain |     2 +
 ...ion_regexp_instr_with_regex_group_index.explain |     2 +
 ..._regexp_instr_without_regex_group_index.explain |     2 +
 .../explain-results/function_regexp_like.explain   |     2 +
 .../function_regexp_replace.explain                |     2 +
 .../explain-results/function_regexp_substr.explain |     3 +
 .../explain-results/function_regr_avgx.explain     |     2 +
 .../explain-results/function_regr_avgy.explain     |     2 +
 .../explain-results/function_regr_count.explain    |     2 +
 .../function_regr_intercept.explain                |     2 +
 .../explain-results/function_regr_r2.explain       |     2 +
 .../explain-results/function_regr_slope.explain    |     2 +
 .../explain-results/function_regr_sxx.explain      |     2 +
 .../explain-results/function_regr_sxy.explain      |     2 +
 .../explain-results/function_regr_syy.explain      |     2 +
 .../explain-results/function_replace.explain       |     2 +
 .../function_replace_with_specified_string.explain |     2 +
 .../explain-results/function_reverse.explain       |     2 +
 .../explain-results/function_right.explain         |     2 +
 .../explain-results/function_rint.explain          |     2 +
 .../explain-results/function_rlike.explain         |     2 +
 .../explain-results/function_round.explain         |     2 +
 .../explain-results/function_row_number.explain    |     5 +
 .../explain-results/function_rpad.explain          |     2 +
 .../explain-results/function_rpad_binary.explain   |     2 +
 .../explain-results/function_rtrim.explain         |     2 +
 .../function_rtrim_with_pattern.explain            |     2 +
 .../explain-results/function_schema_of_csv.explain |     2 +
 .../function_schema_of_json.explain                |     2 +
 .../function_schema_of_json_with_options.explain   |     2 +
 .../function_schema_of_variant.explain             |     2 +
 .../function_schema_of_variant_agg.explain         |     2 +
 .../explain-results/function_sec.explain           |     2 +
 .../explain-results/function_second.explain        |     2 +
 .../explain-results/function_sentences.explain     |     2 +
 .../function_sentences_with_language.explain       |     2 +
 ...ion_sentences_with_language_and_country.explain |     2 +
 .../explain-results/function_sequence.explain      |     2 +
 .../explain-results/function_session_user.explain  |     2 +
 .../function_session_window.explain                |     4 +
 .../function_session_window_orphaned.explain       |     4 +
 .../explain-results/function_sha.explain           |     2 +
 .../explain-results/function_sha1.explain          |     2 +
 .../explain-results/function_sha2.explain          |     2 +
 .../explain-results/function_shiftleft.explain     |     2 +
 .../explain-results/function_shiftright.explain    |     2 +
 .../function_shiftrightunsigned.explain            |     2 +
 .../explain-results/function_sign.explain          |     2 +
 .../explain-results/function_signum.explain        |     2 +
 .../explain-results/function_sin.explain           |     2 +
 .../explain-results/function_sinh.explain          |     2 +
 .../explain-results/function_size.explain          |     2 +
 .../explain-results/function_skewness.explain      |     2 +
 .../explain-results/function_slice.explain         |     2 +
 .../explain-results/function_some.explain          |     2 +
 .../explain-results/function_sort_array.explain    |     2 +
 .../function_spark_partition_id.explain            |     2 +
 .../explain-results/function_split.explain         |     2 +
 .../explain-results/function_split_part.explain    |     2 +
 .../function_split_using_columns.explain           |     2 +
 .../function_split_with_limit.explain              |     2 +
 ...function_split_with_limit_using_columns.explain |     2 +
 .../explain-results/function_sqrt.explain          |     2 +
 .../explain-results/function_stack.explain         |     3 +
 .../explain-results/function_startswith.explain    |     2 +
 .../explain-results/function_std.explain           |     2 +
 .../explain-results/function_stddev.explain        |     2 +
 .../explain-results/function_stddev_pop.explain    |     2 +
 .../explain-results/function_stddev_samp.explain   |     2 +
 .../explain-results/function_str_to_map.explain    |     2 +
 ...to_map_with_pair_and_keyValue_delimiter.explain |     2 +
 ...function_str_to_map_with_pair_delimiter.explain |     2 +
 .../explain-results/function_struct.explain        |     2 +
 .../explain-results/function_substr.explain        |     2 +
 .../function_substr_with_len.explain               |     2 +
 .../explain-results/function_substring.explain     |     2 +
 .../function_substring_index.explain               |     2 +
 .../function_substring_using_columns.explain       |     2 +
 .../function_substring_with_columns.explain        |     2 +
 .../explain-results/function_sum.explain           |     2 +
 .../explain-results/function_sum_distinct.explain  |     2 +
 .../explain-results/function_tan.explain           |     2 +
 .../explain-results/function_tanh.explain          |     2 +
 .../explain-results/function_timestamp_add.explain |     2 +
 .../function_timestamp_diff.explain                |     2 +
 .../function_timestamp_micros.explain              |     2 +
 .../function_timestamp_millis.explain              |     2 +
 .../function_timestamp_seconds.explain             |     2 +
 .../explain-results/function_to_binary.explain     |     2 +
 .../function_to_binary_with_format.explain         |     2 +
 .../explain-results/function_to_char.explain       |     2 +
 .../explain-results/function_to_csv.explain        |     2 +
 .../explain-results/function_to_date.explain       |     2 +
 .../function_to_date_with_format.explain           |     2 +
 .../explain-results/function_to_json.explain       |     2 +
 .../explain-results/function_to_number.explain     |     2 +
 .../explain-results/function_to_timestamp.explain  |     2 +
 .../function_to_timestamp_ltz.explain              |     2 +
 .../function_to_timestamp_ltz_with_format.explain  |     2 +
 .../function_to_timestamp_ntz.explain              |     2 +
 .../function_to_timestamp_ntz_with_format.explain  |     2 +
 .../function_to_timestamp_with_format.explain      |     2 +
 .../function_to_unix_timestamp.explain             |     2 +
 .../function_to_unix_timestamp_with_format.explain |     2 +
 .../function_to_utc_timestamp.explain              |     2 +
 .../explain-results/function_to_varchar.explain    |     2 +
 .../explain-results/function_transform.explain     |     2 +
 .../function_transform_keys.explain                |     2 +
 .../function_transform_values.explain              |     2 +
 .../function_transform_with_index.explain          |     2 +
 .../explain-results/function_translate.explain     |     2 +
 .../explain-results/function_trim.explain          |     2 +
 .../function_trim_with_pattern.explain             |     2 +
 .../explain-results/function_trunc.explain         |     2 +
 .../explain-results/function_try_add.explain       |     2 +
 .../function_try_aes_decrypt.explain               |     2 +
 .../function_try_aes_decrypt_with_mode.explain     |     2 +
 ...ction_try_aes_decrypt_with_mode_padding.explain |     2 +
 ...n_try_aes_decrypt_with_mode_padding_aad.explain |     2 +
 .../explain-results/function_try_avg.explain       |     2 +
 .../explain-results/function_try_divide.explain    |     2 +
 .../function_try_element_at_array.explain          |     2 +
 .../function_try_element_at_map.explain            |     2 +
 .../function_try_make_interval_years.explain       |     2 +
 ...function_try_make_interval_years_months.explain |     2 +
 ...on_try_make_interval_years_months_weeks.explain |     2 +
 ...y_make_interval_years_months_weeks_days.explain |     2 +
 ..._interval_years_months_weeks_days_hours.explain |     2 +
 ...rval_years_months_weeks_days_hours_mins.explain |     2 +
 ...years_months_weeks_days_hours_mins_secs.explain |     2 +
 ...on_try_make_timestamp_ltz_with_timezone.explain |     2 +
 ...try_make_timestamp_ltz_without_timezone.explain |     2 +
 .../function_try_make_timestamp_ntz.explain        |     2 +
 ...nction_try_make_timestamp_with_timezone.explain |     2 +
 ...ion_try_make_timestamp_without_timezone.explain |     2 +
 .../explain-results/function_try_multiply.explain  |     2 +
 .../function_try_parse_json.explain                |     2 +
 .../explain-results/function_try_parse_url.explain |     2 +
 .../function_try_parse_url_with_key.explain        |     2 +
 .../explain-results/function_try_reflect.explain   |     2 +
 .../explain-results/function_try_subtract.explain  |     2 +
 .../explain-results/function_try_sum.explain       |     2 +
 .../explain-results/function_try_to_binary.explain |     2 +
 .../function_try_to_binary_without_format.explain  |     2 +
 .../explain-results/function_try_to_date.explain   |     2 +
 .../function_try_to_date_with_format.explain       |     2 +
 .../explain-results/function_try_to_number.explain |     2 +
 .../function_try_to_timestamp.explain              |     2 +
 ...unction_try_to_timestamp_without_format.explain |     2 +
 .../function_try_url_decode.explain                |     2 +
 .../function_try_variant_get.explain               |     2 +
 .../explain-results/function_typedLit.explain      |     2 +
 .../explain-results/function_typeof.explain        |     2 +
 .../explain-results/function_ucase.explain         |     2 +
 .../explain-results/function_unbase64.explain      |     2 +
 .../explain-results/function_unhex.explain         |     2 +
 .../explain-results/function_unix_date.explain     |     2 +
 .../explain-results/function_unix_micros.explain   |     2 +
 .../explain-results/function_unix_millis.explain   |     2 +
 .../explain-results/function_unix_seconds.explain  |     2 +
 .../function_unix_timestamp.explain                |     2 +
 .../function_unix_timestamp_with_format.explain    |     2 +
 .../explain-results/function_upper.explain         |     2 +
 .../explain-results/function_url_decode.explain    |     2 +
 .../explain-results/function_url_encode.explain    |     2 +
 .../explain-results/function_user.explain          |     2 +
 .../explain-results/function_var_pop.explain       |     2 +
 .../explain-results/function_var_samp.explain      |     2 +
 .../explain-results/function_variance.explain      |     2 +
 .../explain-results/function_variant_get.explain   |     2 +
 .../explain-results/function_weekday.explain       |     2 +
 .../explain-results/function_weekofyear.explain    |     2 +
 .../explain-results/function_window.explain        |     4 +
 .../explain-results/function_window_time.explain   |     4 +
 .../explain-results/function_xpath.explain         |     2 +
 .../explain-results/function_xpath_boolean.explain |     2 +
 .../explain-results/function_xpath_double.explain  |     2 +
 .../explain-results/function_xpath_float.explain   |     2 +
 .../explain-results/function_xpath_int.explain     |     2 +
 .../explain-results/function_xpath_long.explain    |     2 +
 .../explain-results/function_xpath_number.explain  |     2 +
 .../explain-results/function_xpath_short.explain   |     2 +
 .../explain-results/function_xpath_string.explain  |     2 +
 .../explain-results/function_xxhash64.explain      |     2 +
 .../explain-results/function_year.explain          |     2 +
 .../explain-results/function_years.explain         |     2 +
 .../explain-results/function_zip_with.explain      |     2 +
 .../explain-results/groupby_agg.explain            |     2 +
 .../explain-results/groupby_agg_columns.explain    |     2 +
 .../explain-results/groupby_agg_orphaned.explain   |     2 +
 .../explain-results/groupby_agg_string.explain     |     2 +
 .../explain-results/groupby_avg.explain            |     2 +
 .../explain-results/groupby_count.explain          |     2 +
 .../explain-results/groupby_max.explain            |     2 +
 .../explain-results/groupby_mean.explain           |     2 +
 .../explain-results/groupby_min.explain            |     2 +
 .../explain-results/groupby_sum.explain            |     2 +
 .../explain-results/groupingSets.explain           |     4 +
 .../grouping_and_grouping_id.explain               |     4 +
 .../query-tests/explain-results/hint.explain       |     2 +
 .../explain-results/hll_sketch_agg.explain         |     2 +
 .../hll_sketch_agg_with_columnName.explain         |     2 +
 ...ketch_agg_with_columnName_lgConfigK_int.explain |     2 +
 .../hll_sketch_agg_with_column_lgConfigK.explain   |     2 +
 ...ll_sketch_agg_with_column_lgConfigK_int.explain |     2 +
 .../explain-results/hll_union_agg.explain          |     2 +
 .../hll_union_agg_with_columnName.explain          |     2 +
 ...umnName_allowDifferentLgConfigK_boolean.explain |     2 +
 ...agg_with_column_allowDifferentLgConfigK.explain |     2 +
 ..._column_allowDifferentLgConfigK_boolean.explain |     2 +
 .../query-tests/explain-results/intersect.explain  |     3 +
 .../explain-results/intersectAll.explain           |     3 +
 .../explain-results/join_condition.explain         |     5 +
 .../explain-results/join_inner_condition.explain   |     5 +
 .../join_inner_no_condition.explain                |     3 +
 .../join_inner_using_multiple_col_array.explain    |     4 +
 .../join_inner_using_multiple_col_seq.explain      |     4 +
 .../join_inner_using_single_col.explain            |     4 +
 .../join_using_multiple_col_array.explain          |     4 +
 .../join_using_multiple_col_seq.explain            |     4 +
 .../explain-results/join_using_single_col.explain  |     4 +
 .../explain-results/json_from_dataset.explain      |     1 +
 .../query-tests/explain-results/limit.explain      |     3 +
 .../explain-results/melt_no_values.explain         |     2 +
 .../explain-results/melt_values.explain            |     2 +
 .../query-tests/explain-results/offset.explain     |     2 +
 .../explain-results/orderBy_columns.explain        |     2 +
 .../explain-results/orderBy_strings.explain        |     2 +
 .../query-tests/explain-results/pivot.explain      |     4 +
 .../pivot_without_column_values.explain            |     4 +
 .../query-tests/explain-results/range.explain      |     1 +
 .../query-tests/explain-results/read.explain       |     1 +
 .../query-tests/explain-results/read_csv.explain   |     1 +
 .../query-tests/explain-results/read_jdbc.explain  |     1 +
 .../read_jdbc_with_partition.explain               |     1 +
 .../read_jdbc_with_predicates.explain              |     1 +
 .../query-tests/explain-results/read_json.explain  |     1 +
 .../query-tests/explain-results/read_orc.explain   |     1 +
 .../explain-results/read_parquet.explain           |     1 +
 .../query-tests/explain-results/read_table.explain |     2 +
 .../query-tests/explain-results/read_text.explain  |     1 +
 .../explain-results/relation_extension.explain     |     1 +
 .../relation_extension_deprecated.explain          |     1 +
 .../explain-results/repartition.explain            |     2 +
 .../explain-results/repartitionById.explain        |     2 +
 .../repartitionByRange_expressions.explain         |     2 +
 ...itionByRange_num_partitions_expressions.explain |     2 +
 .../repartition_expressions.explain                |     2 +
 .../repartition_num_partitions_expressions.explain |     2 +
 .../query-tests/explain-results/replace.explain    |     2 +
 .../explain-results/rollup_column.explain          |     4 +
 .../explain-results/rollup_string.explain          |     4 +
 .../query-tests/explain-results/sampleBy.explain   |     2 +
 .../explain-results/sample_fraction_seed.explain   |     2 +
 .../sample_withReplacement_fraction_seed.explain   |     2 +
 .../query-tests/explain-results/select.explain     |     2 +
 .../query-tests/explain-results/selectExpr.explain |     2 +
 .../explain-results/select_collated_string.explain |     2 +
 .../explain-results/select_strings.explain         |     2 +
 .../explain-results/select_typed_1-arg.explain     |     3 +
 .../explain-results/select_typed_2-arg.explain     |     2 +
 .../explain-results/select_typed_3-arg.explain     |     2 +
 .../explain-results/select_typed_4-arg.explain     |     2 +
 .../explain-results/select_typed_5-arg.explain     |     2 +
 .../sortWithinPartitions_columns.explain           |     2 +
 .../sortWithinPartitions_strings.explain           |     2 +
 .../explain-results/sort_columns.explain           |     2 +
 .../explain-results/sort_strings.explain           |     2 +
 .../streaming_table_API_with_options.explain       |     2 +
 .../query-tests/explain-results/summary.explain    |     5 +
 .../query-tests/explain-results/table.explain      |     2 +
 .../explain-results/table_API_with_options.explain |     2 +
 .../explain-results/test_broadcast.explain         |     5 +
 .../query-tests/explain-results/to.explain         |     2 +
 .../query-tests/explain-results/toDF.explain       |     2 +
 .../query-tests/explain-results/toJSON.explain     |     2 +
 .../explain-results/to_avro_with_schema.explain    |     2 +
 .../explain-results/to_avro_without_schema.explain |     2 +
 .../to_protobuf_messageClassName.explain           |     3 +
 ..._protobuf_messageClassName_descFilePath.explain |     3 +
 ...f_messageClassName_descFilePath_options.explain |     3 +
 .../to_protobuf_messageClassName_options.explain   |     3 +
 .../query-tests/explain-results/union.explain      |     3 +
 .../query-tests/explain-results/unionAll.explain   |     3 +
 .../explain-results/unionByName.explain            |     6 +
 .../unionByName_allowMissingColumns.explain        |     5 +
 .../explain-results/unpivot_no_values.explain      |     2 +
 .../explain-results/unpivot_values.explain         |     2 +
 .../explain-results/where_column.explain           |     2 +
 .../query-tests/explain-results/where_expr.explain |     2 +
 .../explain-results/width_bucket.explain           |     2 +
 .../query-tests/explain-results/window.explain     |     8 +
 .../withColumnRenamed_java_map.explain             |     2 +
 .../withColumnRenamed_scala_map.explain            |     2 +
 .../withColumnRenamed_single.explain               |     2 +
 .../explain-results/withColumn_single.explain      |     2 +
 .../explain-results/withColumns_java_map.explain   |     2 +
 .../explain-results/withColumns_scala_map.explain  |     2 +
 .../explain-results/withMetadata.explain           |     2 +
 .../query-tests/queries/alias_string.json          |    16 +
 .../query-tests/queries/alias_string.proto.bin     |   Bin 0 -> 55 bytes
 .../query-tests/queries/alias_symbol.json          |    16 +
 .../query-tests/queries/alias_symbol.proto.bin     |   Bin 0 -> 54 bytes
 .../test/resources/query-tests/queries/apply.json  |    38 +
 .../resources/query-tests/queries/apply.proto.bin  |   Bin 0 -> 233 bytes
 .../resources/query-tests/queries/as_string.json   |    16 +
 .../query-tests/queries/as_string.proto.bin        |   Bin 0 -> 54 bytes
 .../resources/query-tests/queries/as_symbol.json   |    16 +
 .../query-tests/queries/as_symbol.proto.bin        |   Bin 0 -> 54 bytes
 .../query-tests/queries/between_expr.json          |    37 +
 .../query-tests/queries/between_expr.proto.bin     |   Bin 0 -> 272 bytes
 .../resources/query-tests/queries/coalesce.json    |    17 +
 .../query-tests/queries/coalesce.proto.bin         |   Bin 0 -> 53 bytes
 .../test/resources/query-tests/queries/col.json    |    60 +
 .../resources/query-tests/queries/col.proto.bin    |   Bin 0 -> 430 bytes
 .../resources/query-tests/queries/colRegex.json    |    38 +
 .../query-tests/queries/colRegex.proto.bin         |   Bin 0 -> 249 bytes
 .../resources/query-tests/queries/column_add.json  |    81 +
 .../query-tests/queries/column_add.proto.bin       |   Bin 0 -> 709 bytes
 .../query-tests/queries/column_alias.json          |    59 +
 .../query-tests/queries/column_alias.proto.bin     |   Bin 0 -> 521 bytes
 .../resources/query-tests/queries/column_and.json  |   169 +
 .../query-tests/queries/column_and.proto.bin       |   Bin 0 -> 1451 bytes
 .../query-tests/queries/column_apply.json          |    80 +
 .../query-tests/queries/column_apply.proto.bin     |   Bin 0 -> 713 bytes
 .../query-tests/queries/column_as_multi.json       |    59 +
 .../query-tests/queries/column_as_multi.proto.bin  |   Bin 0 -> 549 bytes
 .../queries/column_as_with_metadata.json           |    60 +
 .../queries/column_as_with_metadata.proto.bin      |   Bin 0 -> 555 bytes
 .../resources/query-tests/queries/column_asc.json  |    42 +
 .../query-tests/queries/column_asc.proto.bin       |   Bin 0 -> 350 bytes
 .../queries/column_asc_nulls_first.json            |    42 +
 .../queries/column_asc_nulls_first.proto.bin       |   Bin 0 -> 350 bytes
 .../query-tests/queries/column_asc_nulls_last.json |    42 +
 .../queries/column_asc_nulls_last.proto.bin        |   Bin 0 -> 350 bytes
 .../query-tests/queries/column_between.json        |   169 +
 .../query-tests/queries/column_between.proto.bin   |   Bin 0 -> 1447 bytes
 .../query-tests/queries/column_bitwiseAND.json     |    81 +
 .../queries/column_bitwiseAND.proto.bin            |   Bin 0 -> 714 bytes
 .../query-tests/queries/column_bitwiseOR.json      |    81 +
 .../query-tests/queries/column_bitwiseOR.proto.bin |   Bin 0 -> 711 bytes
 .../query-tests/queries/column_bitwiseXOR.json     |    81 +
 .../queries/column_bitwiseXOR.proto.bin            |   Bin 0 -> 713 bytes
 .../resources/query-tests/queries/column_cast.json |    62 +
 .../query-tests/queries/column_cast.proto.bin      |   Bin 0 -> 522 bytes
 .../query-tests/queries/column_contains.json       |    81 +
 .../query-tests/queries/column_contains.proto.bin  |   Bin 0 -> 719 bytes
 .../resources/query-tests/queries/column_desc.json |    42 +
 .../query-tests/queries/column_desc.proto.bin      |   Bin 0 -> 350 bytes
 .../queries/column_desc_nulls_first.json           |    42 +
 .../queries/column_desc_nulls_first.proto.bin      |   Bin 0 -> 350 bytes
 .../queries/column_desc_nulls_last.json            |    42 +
 .../queries/column_desc_nulls_last.proto.bin       |   Bin 0 -> 350 bytes
 .../query-tests/queries/column_divide.json         |    81 +
 .../query-tests/queries/column_divide.proto.bin    |   Bin 0 -> 708 bytes
 .../query-tests/queries/column_dropFields.json     |    81 +
 .../queries/column_dropFields.proto.bin            |   Bin 0 -> 713 bytes
 .../query-tests/queries/column_endsWith.json       |    81 +
 .../query-tests/queries/column_endsWith.proto.bin  |   Bin 0 -> 723 bytes
 .../query-tests/queries/column_eqNullSafe.json     |    81 +
 .../queries/column_eqNullSafe.proto.bin            |   Bin 0 -> 722 bytes
 .../query-tests/queries/column_equals.json         |    81 +
 .../query-tests/queries/column_equals.proto.bin    |   Bin 0 -> 713 bytes
 .../resources/query-tests/queries/column_geq.json  |    81 +
 .../query-tests/queries/column_geq.proto.bin       |   Bin 0 -> 716 bytes
 .../query-tests/queries/column_getField.json       |    80 +
 .../query-tests/queries/column_getField.proto.bin  |   Bin 0 -> 703 bytes
 .../query-tests/queries/column_getItem.json        |    80 +
 .../query-tests/queries/column_getItem.proto.bin   |   Bin 0 -> 702 bytes
 .../resources/query-tests/queries/column_gt.json   |    81 +
 .../query-tests/queries/column_gt.proto.bin        |   Bin 0 -> 712 bytes
 .../query-tests/queries/column_ilike.json          |    81 +
 .../query-tests/queries/column_ilike.proto.bin     |   Bin 0 -> 709 bytes
 .../query-tests/queries/column_isNaN.json          |    60 +
 .../query-tests/queries/column_isNaN.proto.bin     |   Bin 0 -> 528 bytes
 .../query-tests/queries/column_isNotNull.json      |    60 +
 .../query-tests/queries/column_isNotNull.proto.bin |   Bin 0 -> 536 bytes
 .../query-tests/queries/column_isNull.json         |    60 +
 .../query-tests/queries/column_isNull.proto.bin    |   Bin 0 -> 530 bytes
 .../resources/query-tests/queries/column_isin.json |   123 +
 .../query-tests/queries/column_isin.proto.bin      |   Bin 0 -> 1071 bytes
 .../resources/query-tests/queries/column_leq.json  |    81 +
 .../query-tests/queries/column_leq.proto.bin       |   Bin 0 -> 713 bytes
 .../resources/query-tests/queries/column_like.json |    81 +
 .../query-tests/queries/column_like.proto.bin      |   Bin 0 -> 709 bytes
 .../resources/query-tests/queries/column_lt.json   |    81 +
 .../query-tests/queries/column_lt.proto.bin        |   Bin 0 -> 709 bytes
 .../query-tests/queries/column_modulo.json         |    81 +
 .../query-tests/queries/column_modulo.proto.bin    |   Bin 0 -> 709 bytes
 .../query-tests/queries/column_multiply.json       |    81 +
 .../query-tests/queries/column_multiply.proto.bin  |   Bin 0 -> 710 bytes
 .../resources/query-tests/queries/column_not.json  |    60 +
 .../query-tests/queries/column_not.proto.bin       |   Bin 0 -> 529 bytes
 .../query-tests/queries/column_not_equals.json     |   104 +
 .../queries/column_not_equals.proto.bin            |   Bin 0 -> 904 bytes
 .../resources/query-tests/queries/column_or.json   |   169 +
 .../query-tests/queries/column_or.proto.bin        |   Bin 0 -> 1450 bytes
 .../query-tests/queries/column_rlike.json          |    81 +
 .../query-tests/queries/column_rlike.proto.bin     |   Bin 0 -> 712 bytes
 .../resources/query-tests/queries/column_star.json |    36 +
 .../query-tests/queries/column_star.proto.bin      |   Bin 0 -> 338 bytes
 .../queries/column_star_with_target.json           |    37 +
 .../queries/column_star_with_target.proto.bin      |   Bin 0 -> 343 bytes
 .../query-tests/queries/column_startsWith.json     |    81 +
 .../queries/column_startsWith.proto.bin            |   Bin 0 -> 729 bytes
 .../query-tests/queries/column_substr.json         |   102 +
 .../query-tests/queries/column_substr.proto.bin    |   Bin 0 -> 890 bytes
 .../query-tests/queries/column_subtract.json       |    81 +
 .../query-tests/queries/column_subtract.proto.bin  |   Bin 0 -> 710 bytes
 .../query-tests/queries/column_try_cast.json       |    63 +
 .../query-tests/queries/column_try_cast.proto.bin  |   Bin 0 -> 528 bytes
 .../query-tests/queries/column_unary_minus.json    |    60 +
 .../queries/column_unary_minus.proto.bin           |   Bin 0 -> 537 bytes
 .../query-tests/queries/column_when_otherwise.json |   232 +
 .../queries/column_when_otherwise.proto.bin        |   Bin 0 -> 1999 bytes
 .../query-tests/queries/column_withField.json      |    81 +
 .../query-tests/queries/column_withField.proto.bin |   Bin 0 -> 712 bytes
 .../resources/query-tests/queries/crossJoin.json   |    24 +
 .../query-tests/queries/crossJoin.proto.bin        |   Bin 0 -> 98 bytes
 .../resources/query-tests/queries/crosstab.json    |    17 +
 .../query-tests/queries/crosstab.proto.bin         |   Bin 0 -> 55 bytes
 .../query-tests/queries/csv_from_dataset.json      |    39 +
 .../query-tests/queries/csv_from_dataset.proto.bin |   Bin 0 -> 169 bytes
 .../resources/query-tests/queries/cube_column.json |   126 +
 .../query-tests/queries/cube_column.proto.bin      |   Bin 0 -> 937 bytes
 .../resources/query-tests/queries/cube_string.json |   128 +
 .../query-tests/queries/cube_string.proto.bin      |   Bin 0 -> 957 bytes
 .../resources/query-tests/queries/describe.json    |    16 +
 .../query-tests/queries/describe.proto.bin         |   Bin 0 -> 56 bytes
 .../resources/query-tests/queries/distinct.json    |    17 +
 .../query-tests/queries/distinct.proto.bin         |   Bin 0 -> 52 bytes
 .../test/resources/query-tests/queries/drop.json   |    17 +
 .../resources/query-tests/queries/drop.proto.bin   |   Bin 0 -> 58 bytes
 .../query-tests/queries/dropDuplicates.json        |    17 +
 .../query-tests/queries/dropDuplicates.proto.bin   |   Bin 0 -> 52 bytes
 .../queries/dropDuplicates_names_array.json        |    17 +
 .../queries/dropDuplicates_names_array.proto.bin   |   Bin 0 -> 57 bytes
 .../queries/dropDuplicates_names_seq.json          |    17 +
 .../queries/dropDuplicates_names_seq.proto.bin     |   Bin 0 -> 56 bytes
 .../queries/dropDuplicates_varargs.json            |    17 +
 .../queries/dropDuplicates_varargs.proto.bin       |   Bin 0 -> 60 bytes
 .../query-tests/queries/drop_multiple_column.json  |    58 +
 .../queries/drop_multiple_column.proto.bin         |   Bin 0 -> 421 bytes
 .../query-tests/queries/drop_multiple_strings.json |    16 +
 .../queries/drop_multiple_strings.proto.bin        |   Bin 0 -> 59 bytes
 .../query-tests/queries/drop_single_column.json    |    37 +
 .../queries/drop_single_column.proto.bin           |   Bin 0 -> 235 bytes
 .../query-tests/queries/drop_single_string.json    |    16 +
 .../queries/drop_single_string.proto.bin           |   Bin 0 -> 52 bytes
 .../test/resources/query-tests/queries/except.json |    25 +
 .../resources/query-tests/queries/except.proto.bin |   Bin 0 -> 94 bytes
 .../resources/query-tests/queries/exceptAll.json   |    25 +
 .../query-tests/queries/exceptAll.proto.bin        |   Bin 0 -> 94 bytes
 .../query-tests/queries/expression_extension.json  |    26 +
 .../queries/expression_extension.proto.bin         |   Bin 0 -> 127 bytes
 .../queries/expression_extension_deprecated.json   |    26 +
 .../expression_extension_deprecated.proto.bin      |   Bin 0 -> 127 bytes
 .../test/resources/query-tests/queries/fill.json   |    19 +
 .../resources/query-tests/queries/fill.proto.bin   |   Bin 0 -> 57 bytes
 .../test/resources/query-tests/queries/filter.json |    81 +
 .../resources/query-tests/queries/filter.proto.bin |   Bin 0 -> 606 bytes
 .../resources/query-tests/queries/filter_expr.json |    37 +
 .../query-tests/queries/filter_expr.proto.bin      |   Bin 0 -> 252 bytes
 .../resources/query-tests/queries/freqItems.json   |    17 +
 .../query-tests/queries/freqItems.proto.bin        |   Bin 0 -> 65 bytes
 .../queries/from_avro_with_options.json            |   188 +
 .../queries/from_avro_with_options.proto.bin       |   Bin 0 -> 1403 bytes
 .../queries/from_avro_without_options.json         |    81 +
 .../queries/from_avro_without_options.proto.bin    |   Bin 0 -> 579 bytes
 ...rom_protobuf_messageClassName_descFilePath.json |   102 +
 ...rotobuf_messageClassName_descFilePath.proto.bin |   Bin 0 -> 994 bytes
 ...obuf_messageClassName_descFilePath_options.json |   167 +
 ...messageClassName_descFilePath_options.proto.bin |   Bin 0 -> 1530 bytes
 .../query-tests/queries/function_abs.json          |    60 +
 .../query-tests/queries/function_abs.proto.bin     |   Bin 0 -> 531 bytes
 .../query-tests/queries/function_acos.json         |    60 +
 .../query-tests/queries/function_acos.proto.bin    |   Bin 0 -> 534 bytes
 .../query-tests/queries/function_acosh.json        |    60 +
 .../query-tests/queries/function_acosh.proto.bin   |   Bin 0 -> 537 bytes
 .../query-tests/queries/function_add_months.json   |    81 +
 .../queries/function_add_months.proto.bin          |   Bin 0 -> 682 bytes
 .../query-tests/queries/function_aes_decrypt.json  |    81 +
 .../queries/function_aes_decrypt.proto.bin         |   Bin 0 -> 732 bytes
 .../queries/function_aes_decrypt_with_mode.json    |   102 +
 .../function_aes_decrypt_with_mode.proto.bin       |   Bin 0 -> 917 bytes
 .../function_aes_decrypt_with_mode_padding.json    |   123 +
 ...unction_aes_decrypt_with_mode_padding.proto.bin |   Bin 0 -> 1102 bytes
 ...function_aes_decrypt_with_mode_padding_aad.json |   144 +
 ...ion_aes_decrypt_with_mode_padding_aad.proto.bin |   Bin 0 -> 1287 bytes
 .../query-tests/queries/function_aes_encrypt.json  |    81 +
 .../queries/function_aes_encrypt.proto.bin         |   Bin 0 -> 732 bytes
 .../queries/function_aes_encrypt_with_mode.json    |   102 +
 .../function_aes_encrypt_with_mode.proto.bin       |   Bin 0 -> 917 bytes
 .../function_aes_encrypt_with_mode_padding.json    |   123 +
 ...unction_aes_encrypt_with_mode_padding.proto.bin |   Bin 0 -> 1102 bytes
 .../function_aes_encrypt_with_mode_padding_iv.json |   144 +
 ...tion_aes_encrypt_with_mode_padding_iv.proto.bin |   Bin 0 -> 1289 bytes
 ...ction_aes_encrypt_with_mode_padding_iv_aad.json |   165 +
 ..._aes_encrypt_with_mode_padding_iv_aad.proto.bin |   Bin 0 -> 1474 bytes
 .../query-tests/queries/function_aggregate.json    |   166 +
 .../queries/function_aggregate.proto.bin           |   Bin 0 -> 1334 bytes
 .../function_aggregate_with_finish_lambda.json     |   210 +
 ...function_aggregate_with_finish_lambda.proto.bin |   Bin 0 -> 1702 bytes
 .../query-tests/queries/function_any.json          |    60 +
 .../query-tests/queries/function_any.proto.bin     |   Bin 0 -> 425 bytes
 .../function_any_value_with_ignore_nulls.json      |    81 +
 .../function_any_value_with_ignore_nulls.proto.bin |   Bin 0 -> 727 bytes
 .../function_any_value_with_respect_nulls.json     |    60 +
 ...function_any_value_with_respect_nulls.proto.bin |   Bin 0 -> 543 bytes
 .../queries/function_approx_count_distinct.json    |    60 +
 .../function_approx_count_distinct.proto.bin       |   Bin 0 -> 585 bytes
 .../function_approx_count_distinct_rsd.json        |    81 +
 .../function_approx_count_distinct_rsd.proto.bin   |   Bin 0 -> 794 bytes
 .../queries/function_approx_percentile.json        |   102 +
 .../queries/function_approx_percentile.proto.bin   |   Bin 0 -> 934 bytes
 .../query-tests/queries/function_array.json        |    81 +
 .../query-tests/queries/function_array.proto.bin   |   Bin 0 -> 724 bytes
 .../query-tests/queries/function_array_agg.json    |    60 +
 .../queries/function_array_agg.proto.bin           |   Bin 0 -> 543 bytes
 .../query-tests/queries/function_array_append.json |    81 +
 .../queries/function_array_append.proto.bin        |   Bin 0 -> 733 bytes
 .../queries/function_array_compact.json            |    60 +
 .../queries/function_array_compact.proto.bin       |   Bin 0 -> 551 bytes
 .../queries/function_array_contains.json           |    81 +
 .../queries/function_array_contains.proto.bin      |   Bin 0 -> 737 bytes
 .../queries/function_array_distinct.json           |    60 +
 .../queries/function_array_distinct.proto.bin      |   Bin 0 -> 553 bytes
 .../query-tests/queries/function_array_except.json |   146 +
 .../queries/function_array_except.proto.bin        |   Bin 0 -> 1295 bytes
 .../query-tests/queries/function_array_insert.json |   102 +
 .../queries/function_array_insert.proto.bin        |   Bin 0 -> 917 bytes
 .../queries/function_array_intersect.json          |   125 +
 .../queries/function_array_intersect.proto.bin     |   Bin 0 -> 1117 bytes
 .../query-tests/queries/function_array_join.json   |    81 +
 .../queries/function_array_join.proto.bin          |   Bin 0 -> 737 bytes
 .../function_array_join_with_null_replacement.json |   102 +
 ...tion_array_join_with_null_replacement.proto.bin |   Bin 0 -> 932 bytes
 .../query-tests/queries/function_array_max.json    |    60 +
 .../queries/function_array_max.proto.bin           |   Bin 0 -> 543 bytes
 .../query-tests/queries/function_array_min.json    |    60 +
 .../queries/function_array_min.proto.bin           |   Bin 0 -> 543 bytes
 .../queries/function_array_position.json           |    81 +
 .../queries/function_array_position.proto.bin      |   Bin 0 -> 748 bytes
 .../queries/function_array_prepend.json            |    81 +
 .../queries/function_array_prepend.proto.bin       |   Bin 0 -> 735 bytes
 .../query-tests/queries/function_array_remove.json |    81 +
 .../queries/function_array_remove.proto.bin        |   Bin 0 -> 743 bytes
 .../query-tests/queries/function_array_repeat.json |    81 +
 .../queries/function_array_repeat.proto.bin        |   Bin 0 -> 742 bytes
 .../query-tests/queries/function_array_size.json   |    60 +
 .../queries/function_array_size.proto.bin          |   Bin 0 -> 545 bytes
 .../query-tests/queries/function_array_sort.json   |    60 +
 .../queries/function_array_sort.proto.bin          |   Bin 0 -> 545 bytes
 .../function_array_sort_with_comparator.json       |   117 +
 .../function_array_sort_with_comparator.proto.bin  |   Bin 0 -> 950 bytes
 .../query-tests/queries/function_array_union.json  |   146 +
 .../queries/function_array_union.proto.bin         |   Bin 0 -> 1293 bytes
 .../queries/function_arrays_overlap.json           |   125 +
 .../queries/function_arrays_overlap.proto.bin      |   Bin 0 -> 1115 bytes
 .../query-tests/queries/function_arrays_zip.json   |   125 +
 .../queries/function_arrays_zip.proto.bin          |   Bin 0 -> 1113 bytes
 .../query-tests/queries/function_asc.json          |    60 +
 .../query-tests/queries/function_asc.proto.bin     |   Bin 0 -> 528 bytes
 .../queries/function_asc_nulls_first.json          |    60 +
 .../queries/function_asc_nulls_first.proto.bin     |   Bin 0 -> 552 bytes
 .../queries/function_asc_nulls_last.json           |    60 +
 .../queries/function_asc_nulls_last.proto.bin      |   Bin 0 -> 550 bytes
 .../query-tests/queries/function_ascii.json        |    60 +
 .../query-tests/queries/function_ascii.proto.bin   |   Bin 0 -> 535 bytes
 .../query-tests/queries/function_asin.json         |    60 +
 .../query-tests/queries/function_asin.proto.bin    |   Bin 0 -> 534 bytes
 .../query-tests/queries/function_asinh.json        |    60 +
 .../query-tests/queries/function_asinh.proto.bin   |   Bin 0 -> 537 bytes
 .../queries/function_assert_true_with_message.json |   125 +
 .../function_assert_true_with_message.proto.bin    |   Bin 0 -> 1112 bytes
 .../query-tests/queries/function_atan.json         |    60 +
 .../query-tests/queries/function_atan.proto.bin    |   Bin 0 -> 534 bytes
 .../query-tests/queries/function_atan2.json        |   106 +
 .../query-tests/queries/function_atan2.proto.bin   |   Bin 0 -> 903 bytes
 .../query-tests/queries/function_atanh.json        |    60 +
 .../query-tests/queries/function_atanh.proto.bin   |   Bin 0 -> 537 bytes
 .../query-tests/queries/function_avg.json          |    60 +
 .../query-tests/queries/function_avg.proto.bin     |   Bin 0 -> 531 bytes
 .../query-tests/queries/function_base64.json       |    85 +
 .../query-tests/queries/function_base64.proto.bin  |   Bin 0 -> 718 bytes
 .../query-tests/queries/function_bin.json          |    60 +
 .../query-tests/queries/function_bin.proto.bin     |   Bin 0 -> 531 bytes
 .../query-tests/queries/function_bit_and.json      |    60 +
 .../query-tests/queries/function_bit_and.proto.bin |   Bin 0 -> 539 bytes
 .../query-tests/queries/function_bit_count.json    |    60 +
 .../queries/function_bit_count.proto.bin           |   Bin 0 -> 543 bytes
 .../query-tests/queries/function_bit_get.json      |    81 +
 .../query-tests/queries/function_bit_get.proto.bin |   Bin 0 -> 723 bytes
 .../query-tests/queries/function_bit_length.json   |    60 +
 .../queries/function_bit_length.proto.bin          |   Bin 0 -> 545 bytes
 .../query-tests/queries/function_bit_or.json       |    60 +
 .../query-tests/queries/function_bit_or.proto.bin  |   Bin 0 -> 537 bytes
 .../query-tests/queries/function_bit_xor.json      |    60 +
 .../query-tests/queries/function_bit_xor.proto.bin |   Bin 0 -> 539 bytes
 .../queries/function_bitmap_bit_position.json      |    60 +
 .../queries/function_bitmap_bit_position.proto.bin |   Bin 0 -> 564 bytes
 .../queries/function_bitmap_bucket_number.json     |    60 +
 .../function_bitmap_bucket_number.proto.bin        |   Bin 0 -> 564 bytes
 .../queries/function_bitmap_construct_agg.json     |    60 +
 .../function_bitmap_construct_agg.proto.bin        |   Bin 0 -> 566 bytes
 .../query-tests/queries/function_bitmap_count.json |    60 +
 .../queries/function_bitmap_count.proto.bin        |   Bin 0 -> 444 bytes
 .../queries/function_bitmap_or_agg.json            |    60 +
 .../queries/function_bitmap_or_agg.proto.bin       |   Bin 0 -> 446 bytes
 .../query-tests/queries/function_bitwise_not.json  |    60 +
 .../queries/function_bitwise_not.proto.bin         |   Bin 0 -> 537 bytes
 .../query-tests/queries/function_bool_and.json     |    60 +
 .../queries/function_bool_and.proto.bin            |   Bin 0 -> 435 bytes
 .../query-tests/queries/function_bool_or.json      |    60 +
 .../query-tests/queries/function_bool_or.proto.bin |   Bin 0 -> 433 bytes
 .../query-tests/queries/function_bround.json       |    81 +
 .../query-tests/queries/function_bround.proto.bin  |   Bin 0 -> 721 bytes
 .../query-tests/queries/function_btrim.json        |    60 +
 .../query-tests/queries/function_btrim.proto.bin   |   Bin 0 -> 535 bytes
 .../function_btrim_with_specified_trim_string.json |    81 +
 ...tion_btrim_with_specified_trim_string.proto.bin |   Bin 0 -> 720 bytes
 .../query-tests/queries/function_bucket.json       |    81 +
 .../query-tests/queries/function_bucket.proto.bin  |   Bin 0 -> 720 bytes
 .../queries/function_call_function.json            |    61 +
 .../queries/function_call_function.proto.bin       |   Bin 0 -> 545 bytes
 .../queries/function_call_function_orphaned.json   |    25 +
 .../function_call_function_orphaned.proto.bin      |   Bin 0 -> 175 bytes
 .../query-tests/queries/function_cardinality.json  |    60 +
 .../queries/function_cardinality.proto.bin         |   Bin 0 -> 547 bytes
 .../query-tests/queries/function_ceil.json         |    60 +
 .../query-tests/queries/function_ceil.proto.bin    |   Bin 0 -> 534 bytes
 .../query-tests/queries/function_ceil_scale.json   |    81 +
 .../queries/function_ceil_scale.proto.bin          |   Bin 0 -> 717 bytes
 .../query-tests/queries/function_ceiling.json      |    60 +
 .../query-tests/queries/function_ceiling.proto.bin |   Bin 0 -> 539 bytes
 .../queries/function_ceiling_scale.json            |    81 +
 .../queries/function_ceiling_scale.proto.bin       |   Bin 0 -> 723 bytes
 .../query-tests/queries/function_char.json         |    60 +
 .../query-tests/queries/function_char.proto.bin    |   Bin 0 -> 533 bytes
 .../query-tests/queries/function_char_length.json  |    60 +
 .../queries/function_char_length.proto.bin         |   Bin 0 -> 547 bytes
 .../queries/function_character_length.json         |    60 +
 .../queries/function_character_length.proto.bin    |   Bin 0 -> 557 bytes
 .../query-tests/queries/function_chr.json          |    60 +
 .../query-tests/queries/function_chr.proto.bin     |   Bin 0 -> 531 bytes
 .../query-tests/queries/function_coalesce.json     |    81 +
 .../queries/function_coalesce.proto.bin            |   Bin 0 -> 725 bytes
 .../query-tests/queries/function_col.json          |    37 +
 .../query-tests/queries/function_col.proto.bin     |   Bin 0 -> 342 bytes
 .../query-tests/queries/function_collate.json      |    81 +
 .../query-tests/queries/function_collate.proto.bin |   Bin 0 -> 734 bytes
 .../query-tests/queries/function_collation.json    |    60 +
 .../queries/function_collation.proto.bin           |   Bin 0 -> 543 bytes
 .../query-tests/queries/function_collect_list.json |    60 +
 .../queries/function_collect_list.proto.bin        |   Bin 0 -> 558 bytes
 .../query-tests/queries/function_collect_set.json  |    60 +
 .../queries/function_collect_set.proto.bin         |   Bin 0 -> 555 bytes
 .../query-tests/queries/function_concat.json       |   190 +
 .../query-tests/queries/function_concat.proto.bin  |   Bin 0 -> 1667 bytes
 .../query-tests/queries/function_concat_ws.json    |   123 +
 .../queries/function_concat_ws.proto.bin           |   Bin 0 -> 1109 bytes
 .../query-tests/queries/function_contains.json     |    81 +
 .../queries/function_contains.proto.bin            |   Bin 0 -> 726 bytes
 .../query-tests/queries/function_conv.json         |   102 +
 .../query-tests/queries/function_conv.proto.bin    |   Bin 0 -> 903 bytes
 ...ion_convert_timezone_with_source_time_zone.json |   102 +
 ...onvert_timezone_with_source_time_zone.proto.bin |   Bin 0 -> 898 bytes
 ..._convert_timezone_without_source_time_zone.json |    81 +
 ...ert_timezone_without_source_time_zone.proto.bin |   Bin 0 -> 700 bytes
 .../query-tests/queries/function_corr.json         |    81 +
 .../query-tests/queries/function_corr.proto.bin    |   Bin 0 -> 720 bytes
 .../query-tests/queries/function_cos.json          |    60 +
 .../query-tests/queries/function_cos.proto.bin     |   Bin 0 -> 531 bytes
 .../query-tests/queries/function_cosh.json         |    60 +
 .../query-tests/queries/function_cosh.proto.bin    |   Bin 0 -> 534 bytes
 .../query-tests/queries/function_cot.json          |    60 +
 .../query-tests/queries/function_cot.proto.bin     |   Bin 0 -> 531 bytes
 .../query-tests/queries/function_count.json        |    60 +
 .../query-tests/queries/function_count.proto.bin   |   Bin 0 -> 535 bytes
 .../queries/function_countDistinct.json            |    82 +
 .../queries/function_countDistinct.proto.bin       |   Bin 0 -> 750 bytes
 .../query-tests/queries/function_count_if.json     |   104 +
 .../queries/function_count_if.proto.bin            |   Bin 0 -> 897 bytes
 .../queries/function_count_min_sketch.json         |   123 +
 .../queries/function_count_min_sketch.proto.bin    |   Bin 0 -> 1123 bytes
 .../query-tests/queries/function_count_typed.json  |    60 +
 .../queries/function_count_typed.proto.bin         |   Bin 0 -> 430 bytes
 .../query-tests/queries/function_covar_pop.json    |    81 +
 .../queries/function_covar_pop.proto.bin           |   Bin 0 -> 740 bytes
 .../query-tests/queries/function_covar_samp.json   |    81 +
 .../queries/function_covar_samp.proto.bin          |   Bin 0 -> 744 bytes
 .../query-tests/queries/function_crc32.json        |    85 +
 .../query-tests/queries/function_crc32.proto.bin   |   Bin 0 -> 716 bytes
 .../query-tests/queries/function_csc.json          |    60 +
 .../query-tests/queries/function_csc.proto.bin     |   Bin 0 -> 531 bytes
 .../query-tests/queries/function_cume_dist.json    |   107 +
 .../queries/function_cume_dist.proto.bin           |   Bin 0 -> 904 bytes
 .../query-tests/queries/function_curdate.json      |    38 +
 .../query-tests/queries/function_curdate.proto.bin |   Bin 0 -> 299 bytes
 .../queries/function_current_catalog.json          |    38 +
 .../queries/function_current_catalog.proto.bin     |   Bin 0 -> 369 bytes
 .../queries/function_current_database.json         |    38 +
 .../queries/function_current_database.proto.bin    |   Bin 0 -> 371 bytes
 .../query-tests/queries/function_current_date.json |    38 +
 .../queries/function_current_date.proto.bin        |   Bin 0 -> 309 bytes
 .../queries/function_current_schema.json           |    38 +
 .../queries/function_current_schema.proto.bin      |   Bin 0 -> 367 bytes
 .../queries/function_current_timestamp.json        |    38 +
 .../queries/function_current_timestamp.proto.bin   |   Bin 0 -> 319 bytes
 .../queries/function_current_timezone.json         |    38 +
 .../queries/function_current_timezone.proto.bin    |   Bin 0 -> 317 bytes
 .../query-tests/queries/function_current_user.json |    38 +
 .../queries/function_current_user.proto.bin        |   Bin 0 -> 363 bytes
 .../query-tests/queries/function_date_add.json     |    81 +
 .../queries/function_date_add.proto.bin            |   Bin 0 -> 676 bytes
 .../query-tests/queries/function_date_diff.json    |   146 +
 .../queries/function_date_diff.proto.bin           |   Bin 0 -> 1244 bytes
 .../query-tests/queries/function_date_format.json  |    81 +
 .../queries/function_date_format.proto.bin         |   Bin 0 -> 695 bytes
 .../queries/function_date_from_unix_date.json      |    60 +
 .../queries/function_date_from_unix_date.proto.bin |   Bin 0 -> 508 bytes
 .../query-tests/queries/function_date_part.json    |    81 +
 .../queries/function_date_part.proto.bin           |   Bin 0 -> 677 bytes
 .../query-tests/queries/function_date_sub.json     |    81 +
 .../queries/function_date_sub.proto.bin            |   Bin 0 -> 676 bytes
 .../query-tests/queries/function_date_trunc.json   |    81 +
 .../queries/function_date_trunc.proto.bin          |   Bin 0 -> 673 bytes
 .../query-tests/queries/function_dateadd.json      |    81 +
 .../query-tests/queries/function_dateadd.proto.bin |   Bin 0 -> 669 bytes
 .../query-tests/queries/function_datediff.json     |   146 +
 .../queries/function_datediff.proto.bin            |   Bin 0 -> 1242 bytes
 .../query-tests/queries/function_datepart.json     |    81 +
 .../queries/function_datepart.proto.bin            |   Bin 0 -> 675 bytes
 .../query-tests/queries/function_day.json          |    60 +
 .../query-tests/queries/function_day.proto.bin     |   Bin 0 -> 477 bytes
 .../query-tests/queries/function_dayname.json      |    60 +
 .../query-tests/queries/function_dayname.proto.bin |   Bin 0 -> 485 bytes
 .../query-tests/queries/function_dayofmonth.json   |    60 +
 .../queries/function_dayofmonth.proto.bin          |   Bin 0 -> 491 bytes
 .../query-tests/queries/function_dayofweek.json    |    60 +
 .../queries/function_dayofweek.proto.bin           |   Bin 0 -> 489 bytes
 .../query-tests/queries/function_dayofyear.json    |    60 +
 .../queries/function_dayofyear.proto.bin           |   Bin 0 -> 489 bytes
 .../query-tests/queries/function_days.json         |    60 +
 .../query-tests/queries/function_days.proto.bin    |   Bin 0 -> 529 bytes
 .../query-tests/queries/function_decode.json       |   106 +
 .../query-tests/queries/function_decode.proto.bin  |   Bin 0 -> 910 bytes
 .../query-tests/queries/function_degrees.json      |    60 +
 .../query-tests/queries/function_degrees.proto.bin |   Bin 0 -> 543 bytes
 .../query-tests/queries/function_dense_rank.json   |   107 +
 .../queries/function_dense_rank.proto.bin          |   Bin 0 -> 906 bytes
 .../query-tests/queries/function_desc.json         |    60 +
 .../query-tests/queries/function_desc.proto.bin    |   Bin 0 -> 530 bytes
 .../queries/function_desc_nulls_first.json         |    60 +
 .../queries/function_desc_nulls_first.proto.bin    |   Bin 0 -> 554 bytes
 .../queries/function_desc_nulls_last.json          |    60 +
 .../queries/function_desc_nulls_last.proto.bin     |   Bin 0 -> 552 bytes
 .../resources/query-tests/queries/function_e.json  |    38 +
 .../query-tests/queries/function_e.proto.bin       |   Bin 0 -> 341 bytes
 .../query-tests/queries/function_element_at.json   |    81 +
 .../queries/function_element_at.proto.bin          |   Bin 0 -> 739 bytes
 .../query-tests/queries/function_elt.json          |   102 +
 .../query-tests/queries/function_elt.proto.bin     |   Bin 0 -> 901 bytes
 .../query-tests/queries/function_encode.json       |    81 +
 .../query-tests/queries/function_encode.proto.bin  |   Bin 0 -> 729 bytes
 .../query-tests/queries/function_endswith.json     |    81 +
 .../queries/function_endswith.proto.bin            |   Bin 0 -> 726 bytes
 .../query-tests/queries/function_equal_null.json   |    81 +
 .../queries/function_equal_null.proto.bin          |   Bin 0 -> 730 bytes
 .../query-tests/queries/function_every.json        |    60 +
 .../query-tests/queries/function_every.proto.bin   |   Bin 0 -> 429 bytes
 .../query-tests/queries/function_exists.json       |   132 +
 .../query-tests/queries/function_exists.proto.bin  |   Bin 0 -> 1107 bytes
 .../query-tests/queries/function_exp.json          |    60 +
 .../query-tests/queries/function_exp.proto.bin     |   Bin 0 -> 531 bytes
 .../query-tests/queries/function_explode.json      |    60 +
 .../query-tests/queries/function_explode.proto.bin |   Bin 0 -> 539 bytes
 .../queries/function_explode_outer.json            |    60 +
 .../queries/function_explode_outer.proto.bin       |   Bin 0 -> 551 bytes
 .../query-tests/queries/function_expm1.json        |    60 +
 .../query-tests/queries/function_expm1.proto.bin   |   Bin 0 -> 537 bytes
 .../query-tests/queries/function_expr.json         |    37 +
 .../query-tests/queries/function_expr.proto.bin    |   Bin 0 -> 346 bytes
 .../query-tests/queries/function_extract.json      |    81 +
 .../query-tests/queries/function_extract.proto.bin |   Bin 0 -> 673 bytes
 .../query-tests/queries/function_factorial.json    |   104 +
 .../queries/function_factorial.proto.bin           |   Bin 0 -> 911 bytes
 .../query-tests/queries/function_filter.json       |   132 +
 .../query-tests/queries/function_filter.proto.bin  |   Bin 0 -> 1107 bytes
 .../queries/function_filter_with_pair_input.json   |   205 +
 .../function_filter_with_pair_input.proto.bin      |   Bin 0 -> 1679 bytes
 .../query-tests/queries/function_find_in_set.json  |    81 +
 .../queries/function_find_in_set.proto.bin         |   Bin 0 -> 732 bytes
 .../function_first_value_with_ignore_nulls.json    |    81 +
 ...unction_first_value_with_ignore_nulls.proto.bin |   Bin 0 -> 731 bytes
 .../function_first_value_with_respect_nulls.json   |    60 +
 ...nction_first_value_with_respect_nulls.proto.bin |   Bin 0 -> 547 bytes
 .../queries/function_first_with_ignore_nulls.json  |    81 +
 .../function_first_with_ignore_nulls.proto.bin     |   Bin 0 -> 723 bytes
 .../queries/function_first_with_respect_nulls.json |    81 +
 .../function_first_with_respect_nulls.proto.bin    |   Bin 0 -> 723 bytes
 .../query-tests/queries/function_flatten.json      |   148 +
 .../query-tests/queries/function_flatten.proto.bin |   Bin 0 -> 1301 bytes
 .../query-tests/queries/function_floor.json        |    60 +
 .../query-tests/queries/function_floor.proto.bin   |   Bin 0 -> 537 bytes
 .../query-tests/queries/function_floor_scale.json  |    81 +
 .../queries/function_floor_scale.proto.bin         |   Bin 0 -> 719 bytes
 .../query-tests/queries/function_forall.json       |   132 +
 .../query-tests/queries/function_forall.proto.bin  |   Bin 0 -> 1107 bytes
 .../queries/function_format_number.json            |    81 +
 .../queries/function_format_number.proto.bin       |   Bin 0 -> 745 bytes
 .../query-tests/queries/function_from_csv.json     |   146 +
 .../queries/function_from_csv.proto.bin            |   Bin 0 -> 1339 bytes
 .../query-tests/queries/function_from_json.json    |    81 +
 .../queries/function_from_json.proto.bin           |   Bin 0 -> 770 bytes
 .../queries/function_from_json_orphaned.json       |    29 +
 .../queries/function_from_json_orphaned.proto.bin  |   Bin 0 -> 394 bytes
 .../function_from_json_with_json_schema.json       |    81 +
 .../function_from_json_with_json_schema.proto.bin  |   Bin 0 -> 934 bytes
 .../queries/function_from_unixtime.json            |    60 +
 .../queries/function_from_unixtime.proto.bin       |   Bin 0 -> 496 bytes
 .../queries/function_from_utc_timestamp.json       |    81 +
 .../queries/function_from_utc_timestamp.proto.bin  |   Bin 0 -> 712 bytes
 .../query-tests/queries/function_from_xml.json     |    81 +
 .../queries/function_from_xml.proto.bin            |   Bin 0 -> 767 bytes
 .../function_from_xml_with_json_schema.json        |    81 +
 .../function_from_xml_with_json_schema.proto.bin   |   Bin 0 -> 932 bytes
 .../query-tests/queries/function_get.json          |    81 +
 .../query-tests/queries/function_get.proto.bin     |   Bin 0 -> 715 bytes
 .../queries/function_get_json_object.json          |    81 +
 .../queries/function_get_json_object.proto.bin     |   Bin 0 -> 764 bytes
 .../query-tests/queries/function_getbit.json       |    81 +
 .../query-tests/queries/function_getbit.proto.bin  |   Bin 0 -> 721 bytes
 .../query-tests/queries/function_greatest.json     |   124 +
 .../queries/function_greatest.proto.bin            |   Bin 0 -> 1088 bytes
 .../query-tests/queries/function_hash.json         |    81 +
 .../query-tests/queries/function_hash.proto.bin    |   Bin 0 -> 719 bytes
 .../query-tests/queries/function_hex.json          |    60 +
 .../query-tests/queries/function_hex.proto.bin     |   Bin 0 -> 531 bytes
 .../queries/function_histogram_numeric.json        |    81 +
 .../queries/function_histogram_numeric.proto.bin   |   Bin 0 -> 743 bytes
 .../query-tests/queries/function_hour.json         |    60 +
 .../query-tests/queries/function_hour.proto.bin    |   Bin 0 -> 479 bytes
 .../query-tests/queries/function_hours.json        |    60 +
 .../query-tests/queries/function_hours.proto.bin   |   Bin 0 -> 531 bytes
 .../query-tests/queries/function_hypot.json        |    81 +
 .../query-tests/queries/function_hypot.proto.bin   |   Bin 0 -> 720 bytes
 .../query-tests/queries/function_ifnull.json       |    81 +
 .../query-tests/queries/function_ifnull.proto.bin  |   Bin 0 -> 722 bytes
 .../query-tests/queries/function_ilike.json        |    81 +
 .../query-tests/queries/function_ilike.proto.bin   |   Bin 0 -> 720 bytes
 .../queries/function_ilike_with_escape.json        |   102 +
 .../queries/function_ilike_with_escape.proto.bin   |   Bin 0 -> 905 bytes
 .../query-tests/queries/function_initcap.json      |    60 +
 .../query-tests/queries/function_initcap.proto.bin |   Bin 0 -> 539 bytes
 .../query-tests/queries/function_inline.json       |    83 +
 .../query-tests/queries/function_inline.proto.bin  |   Bin 0 -> 741 bytes
 .../query-tests/queries/function_inline_outer.json |    83 +
 .../queries/function_inline_outer.proto.bin        |   Bin 0 -> 753 bytes
 .../queries/function_input_file_block_length.json  |    38 +
 .../function_input_file_block_length.proto.bin     |   Bin 0 -> 385 bytes
 .../queries/function_input_file_block_start.json   |    38 +
 .../function_input_file_block_start.proto.bin      |   Bin 0 -> 383 bytes
 .../queries/function_input_file_name.json          |    38 +
 .../queries/function_input_file_name.proto.bin     |   Bin 0 -> 369 bytes
 .../queries/function_is_variant_null.json          |    83 +
 .../queries/function_is_variant_null.proto.bin     |   Bin 0 -> 759 bytes
 .../query-tests/queries/function_isnan.json        |    60 +
 .../query-tests/queries/function_isnan.proto.bin   |   Bin 0 -> 535 bytes
 .../query-tests/queries/function_isnotnull.json    |    60 +
 .../queries/function_isnotnull.proto.bin           |   Bin 0 -> 543 bytes
 .../query-tests/queries/function_isnull.json       |    60 +
 .../query-tests/queries/function_isnull.proto.bin  |   Bin 0 -> 537 bytes
 .../query-tests/queries/function_java_method.json  |   102 +
 .../queries/function_java_method.proto.bin         |   Bin 0 -> 939 bytes
 .../queries/function_json_array_length.json        |    60 +
 .../queries/function_json_array_length.proto.bin   |   Bin 0 -> 559 bytes
 .../queries/function_json_object_keys.json         |    60 +
 .../queries/function_json_object_keys.proto.bin    |   Bin 0 -> 557 bytes
 .../query-tests/queries/function_json_tuple.json   |   123 +
 .../queries/function_json_tuple.proto.bin          |   Bin 0 -> 1122 bytes
 .../query-tests/queries/function_kurtosis.json     |    60 +
 .../queries/function_kurtosis.proto.bin            |   Bin 0 -> 546 bytes
 .../query-tests/queries/function_lag.json          |   199 +
 .../query-tests/queries/function_lag.proto.bin     |   Bin 0 -> 1633 bytes
 .../query-tests/queries/function_last_day.json     |    60 +
 .../queries/function_last_day.proto.bin            |   Bin 0 -> 487 bytes
 .../function_last_value_with_ignore_nulls.json     |    81 +
 ...function_last_value_with_ignore_nulls.proto.bin |   Bin 0 -> 729 bytes
 .../function_last_value_with_respect_nulls.json    |    60 +
 ...unction_last_value_with_respect_nulls.proto.bin |   Bin 0 -> 545 bytes
 .../queries/function_last_with_ignore_nulls.json   |    81 +
 .../function_last_with_ignore_nulls.proto.bin      |   Bin 0 -> 719 bytes
 .../queries/function_last_with_respect_nulls.json  |    81 +
 .../function_last_with_respect_nulls.proto.bin     |   Bin 0 -> 719 bytes
 .../query-tests/queries/function_lcase.json        |    60 +
 .../query-tests/queries/function_lcase.proto.bin   |   Bin 0 -> 535 bytes
 .../query-tests/queries/function_lead.json         |   192 +
 .../query-tests/queries/function_lead.proto.bin    |   Bin 0 -> 1633 bytes
 .../query-tests/queries/function_least.json        |   124 +
 .../query-tests/queries/function_least.proto.bin   |   Bin 0 -> 1082 bytes
 .../query-tests/queries/function_left.json         |    81 +
 .../query-tests/queries/function_left.proto.bin    |   Bin 0 -> 718 bytes
 .../query-tests/queries/function_len.json          |    60 +
 .../query-tests/queries/function_len.proto.bin     |   Bin 0 -> 531 bytes
 .../query-tests/queries/function_length.json       |    60 +
 .../query-tests/queries/function_length.proto.bin  |   Bin 0 -> 537 bytes
 .../query-tests/queries/function_levenshtein.json  |    81 +
 .../queries/function_levenshtein.proto.bin         |   Bin 0 -> 734 bytes
 .../function_levenshtein_with_threshold.json       |   102 +
 .../function_levenshtein_with_threshold.proto.bin  |   Bin 0 -> 926 bytes
 .../query-tests/queries/function_like.json         |    81 +
 .../query-tests/queries/function_like.proto.bin    |   Bin 0 -> 718 bytes
 .../queries/function_like_with_escape.json         |   102 +
 .../queries/function_like_with_escape.proto.bin    |   Bin 0 -> 903 bytes
 .../query-tests/queries/function_lit.json          |   645 +
 .../query-tests/queries/function_lit.proto.bin     |   Bin 0 -> 5400 bytes
 .../query-tests/queries/function_lit_array.json    |   908 +
 .../queries/function_lit_array.proto.bin           |   Bin 0 -> 5116 bytes
 .../resources/query-tests/queries/function_ln.json |    60 +
 .../query-tests/queries/function_ln.proto.bin      |   Bin 0 -> 529 bytes
 .../queries/function_localtimestamp.json           |    38 +
 .../queries/function_localtimestamp.proto.bin      |   Bin 0 -> 313 bytes
 .../query-tests/queries/function_locate.json       |    81 +
 .../query-tests/queries/function_locate.proto.bin  |   Bin 0 -> 727 bytes
 .../queries/function_locate_with_pos.json          |   102 +
 .../queries/function_locate_with_pos.proto.bin     |   Bin 0 -> 914 bytes
 .../query-tests/queries/function_log.json          |    60 +
 .../query-tests/queries/function_log.proto.bin     |   Bin 0 -> 530 bytes
 .../query-tests/queries/function_log10.json        |    60 +
 .../query-tests/queries/function_log10.proto.bin   |   Bin 0 -> 537 bytes
 .../query-tests/queries/function_log1p.json        |    60 +
 .../query-tests/queries/function_log1p.proto.bin   |   Bin 0 -> 537 bytes
 .../query-tests/queries/function_log2.json         |    60 +
 .../query-tests/queries/function_log2.proto.bin    |   Bin 0 -> 534 bytes
 .../queries/function_log_with_base.json            |    81 +
 .../queries/function_log_with_base.proto.bin       |   Bin 0 -> 722 bytes
 .../query-tests/queries/function_lower.json        |    60 +
 .../query-tests/queries/function_lower.proto.bin   |   Bin 0 -> 535 bytes
 .../query-tests/queries/function_lpad.json         |   102 +
 .../query-tests/queries/function_lpad.proto.bin    |   Bin 0 -> 904 bytes
 .../query-tests/queries/function_lpad_binary.json  |   102 +
 .../queries/function_lpad_binary.proto.bin         |   Bin 0 -> 802 bytes
 .../query-tests/queries/function_ltrim.json        |    60 +
 .../query-tests/queries/function_ltrim.proto.bin   |   Bin 0 -> 535 bytes
 .../queries/function_ltrim_with_pattern.json       |    81 +
 .../queries/function_ltrim_with_pattern.proto.bin  |   Bin 0 -> 724 bytes
 .../query-tests/queries/function_make_date.json    |   102 +
 .../queries/function_make_date.proto.bin           |   Bin 0 -> 857 bytes
 .../queries/function_make_dt_interval.json         |    38 +
 .../queries/function_make_dt_interval.proto.bin    |   Bin 0 -> 371 bytes
 .../queries/function_make_dt_interval_days.json    |    60 +
 .../function_make_dt_interval_days.proto.bin       |   Bin 0 -> 557 bytes
 .../function_make_dt_interval_days_hours.json      |    81 +
 .../function_make_dt_interval_days_hours.proto.bin |   Bin 0 -> 742 bytes
 .../function_make_dt_interval_days_hours_mins.json |   102 +
 ...tion_make_dt_interval_days_hours_mins.proto.bin |   Bin 0 -> 927 bytes
 ...tion_make_dt_interval_days_hours_mins_secs.json |   123 +
 ...make_dt_interval_days_hours_mins_secs.proto.bin |   Bin 0 -> 1112 bytes
 .../queries/function_make_interval.json            |    38 +
 .../queries/function_make_interval.proto.bin       |   Bin 0 -> 365 bytes
 .../queries/function_make_interval_years.json      |    60 +
 .../queries/function_make_interval_years.proto.bin |   Bin 0 -> 551 bytes
 .../function_make_interval_years_months.json       |    81 +
 .../function_make_interval_years_months.proto.bin  |   Bin 0 -> 736 bytes
 .../function_make_interval_years_months_weeks.json |   102 +
 ...tion_make_interval_years_months_weeks.proto.bin |   Bin 0 -> 921 bytes
 ...tion_make_interval_years_months_weeks_days.json |   123 +
 ...make_interval_years_months_weeks_days.proto.bin |   Bin 0 -> 1106 bytes
 ...ake_interval_years_months_weeks_days_hours.json |   144 +
 ...nterval_years_months_weeks_days_hours.proto.bin |   Bin 0 -> 1291 bytes
 ...nterval_years_months_weeks_days_hours_mins.json |   165 +
 ...al_years_months_weeks_days_hours_mins.proto.bin |   Bin 0 -> 1476 bytes
 ...al_years_months_weeks_days_hours_mins_secs.json |   186 +
 ...ars_months_weeks_days_hours_mins_secs.proto.bin |   Bin 0 -> 1661 bytes
 .../function_make_timestamp_ltz_with_timezone.json |   186 +
 ...tion_make_timestamp_ltz_with_timezone.proto.bin |   Bin 0 -> 1671 bytes
 ...nction_make_timestamp_ltz_without_timezone.json |   165 +
 ...n_make_timestamp_ltz_without_timezone.proto.bin |   Bin 0 -> 1486 bytes
 .../queries/function_make_timestamp_ntz.json       |   165 +
 .../queries/function_make_timestamp_ntz.proto.bin  |   Bin 0 -> 1486 bytes
 .../function_make_timestamp_with_timezone.json     |   186 +
 ...function_make_timestamp_with_timezone.proto.bin |   Bin 0 -> 1663 bytes
 .../function_make_timestamp_without_timezone.json  |   165 +
 ...ction_make_timestamp_without_timezone.proto.bin |   Bin 0 -> 1478 bytes
 .../queries/function_make_ym_interval.json         |    38 +
 .../queries/function_make_ym_interval.proto.bin    |   Bin 0 -> 371 bytes
 .../queries/function_make_ym_interval_years.json   |    60 +
 .../function_make_ym_interval_years.proto.bin      |   Bin 0 -> 557 bytes
 .../function_make_ym_interval_years_months.json    |    81 +
 ...unction_make_ym_interval_years_months.proto.bin |   Bin 0 -> 742 bytes
 .../query-tests/queries/function_map.json          |   123 +
 .../query-tests/queries/function_map.proto.bin     |   Bin 0 -> 1089 bytes
 .../query-tests/queries/function_map_concat.json   |   256 +
 .../queries/function_map_concat.proto.bin          |   Bin 0 -> 2212 bytes
 .../queries/function_map_contains_key.json         |    81 +
 .../queries/function_map_contains_key.proto.bin    |   Bin 0 -> 757 bytes
 .../query-tests/queries/function_map_entries.json  |    60 +
 .../queries/function_map_entries.proto.bin         |   Bin 0 -> 547 bytes
 .../query-tests/queries/function_map_filter.json   |   134 +
 .../queries/function_map_filter.proto.bin          |   Bin 0 -> 1138 bytes
 .../queries/function_map_from_arrays.json          |   169 +
 .../queries/function_map_from_arrays.proto.bin     |   Bin 0 -> 1500 bytes
 .../queries/function_map_from_entries.json         |   140 +
 .../queries/function_map_from_entries.proto.bin    |   Bin 0 -> 1175 bytes
 .../query-tests/queries/function_map_keys.json     |    60 +
 .../queries/function_map_keys.proto.bin            |   Bin 0 -> 541 bytes
 .../query-tests/queries/function_map_values.json   |    60 +
 .../queries/function_map_values.proto.bin          |   Bin 0 -> 545 bytes
 .../query-tests/queries/function_map_zip_with.json |   226 +
 .../queries/function_map_zip_with.proto.bin        |   Bin 0 -> 1874 bytes
 .../query-tests/queries/function_mask.json         |    60 +
 .../query-tests/queries/function_mask.proto.bin    |   Bin 0 -> 533 bytes
 .../function_mask_with_specific_upperChar.json     |    81 +
 ...function_mask_with_specific_upperChar.proto.bin |   Bin 0 -> 718 bytes
 ...ion_mask_with_specific_upperChar_lowerChar.json |   102 +
 ...ask_with_specific_upperChar_lowerChar.proto.bin |   Bin 0 -> 903 bytes
 ...ith_specific_upperChar_lowerChar_digitChar.json |   123 +
 ...pecific_upperChar_lowerChar_digitChar.proto.bin |   Bin 0 -> 1088 bytes
 ...ic_upperChar_lowerChar_digitChar_otherChar.json |   144 +
 ...perChar_lowerChar_digitChar_otherChar.proto.bin |   Bin 0 -> 1273 bytes
 .../query-tests/queries/function_max.json          |    60 +
 .../query-tests/queries/function_max.proto.bin     |   Bin 0 -> 425 bytes
 .../query-tests/queries/function_max_by.json       |    81 +
 .../query-tests/queries/function_max_by.proto.bin  |   Bin 0 -> 722 bytes
 .../query-tests/queries/function_md5.json          |    85 +
 .../query-tests/queries/function_md5.proto.bin     |   Bin 0 -> 712 bytes
 .../query-tests/queries/function_median.json       |    60 +
 .../query-tests/queries/function_median.proto.bin  |   Bin 0 -> 537 bytes
 .../query-tests/queries/function_min.json          |    60 +
 .../query-tests/queries/function_min.proto.bin     |   Bin 0 -> 531 bytes
 .../query-tests/queries/function_min_by.json       |    81 +
 .../query-tests/queries/function_min_by.proto.bin  |   Bin 0 -> 722 bytes
 .../query-tests/queries/function_minute.json       |    60 +
 .../query-tests/queries/function_minute.proto.bin  |   Bin 0 -> 483 bytes
 .../query-tests/queries/function_mode.json         |    60 +
 .../query-tests/queries/function_mode.proto.bin    |   Bin 0 -> 533 bytes
 .../queries/function_mode_orphaned.json            |    29 +
 .../queries/function_mode_orphaned.proto.bin       |   Bin 0 -> 179 bytes
 .../function_monotonically_increasing_id.json      |    38 +
 .../function_monotonically_increasing_id.proto.bin |   Bin 0 -> 393 bytes
 .../query-tests/queries/function_month.json        |    60 +
 .../query-tests/queries/function_month.proto.bin   |   Bin 0 -> 481 bytes
 .../query-tests/queries/function_monthname.json    |    60 +
 .../queries/function_monthname.proto.bin           |   Bin 0 -> 489 bytes
 .../query-tests/queries/function_months.json       |    60 +
 .../query-tests/queries/function_months.proto.bin  |   Bin 0 -> 533 bytes
 .../queries/function_months_between.json           |    82 +
 .../queries/function_months_between.proto.bin      |   Bin 0 -> 706 bytes
 .../function_months_between_with_roundoff.json     |   103 +
 ...function_months_between_with_roundoff.proto.bin |   Bin 0 -> 901 bytes
 .../query-tests/queries/function_named_struct.json |   123 +
 .../queries/function_named_struct.proto.bin        |   Bin 0 -> 1105 bytes
 .../query-tests/queries/function_nanvl.json        |    81 +
 .../query-tests/queries/function_nanvl.proto.bin   |   Bin 0 -> 726 bytes
 .../query-tests/queries/function_negate.json       |    60 +
 .../query-tests/queries/function_negate.proto.bin  |   Bin 0 -> 539 bytes
 .../query-tests/queries/function_negative.json     |    60 +
 .../queries/function_negative.proto.bin            |   Bin 0 -> 541 bytes
 .../query-tests/queries/function_next_day.json     |    81 +
 .../queries/function_next_day.proto.bin            |   Bin 0 -> 679 bytes
 .../query-tests/queries/function_now.json          |    38 +
 .../query-tests/queries/function_now.proto.bin     |   Bin 0 -> 291 bytes
 .../query-tests/queries/function_nth_value.json    |   171 +
 .../queries/function_nth_value.proto.bin           |   Bin 0 -> 1466 bytes
 .../query-tests/queries/function_ntile.json        |   129 +
 .../query-tests/queries/function_ntile.proto.bin   |   Bin 0 -> 1083 bytes
 .../query-tests/queries/function_nullif.json       |    81 +
 .../query-tests/queries/function_nullif.proto.bin  |   Bin 0 -> 722 bytes
 .../query-tests/queries/function_nvl.json          |    81 +
 .../query-tests/queries/function_nvl.proto.bin     |   Bin 0 -> 716 bytes
 .../query-tests/queries/function_nvl2.json         |   102 +
 .../query-tests/queries/function_nvl2.proto.bin    |   Bin 0 -> 903 bytes
 .../query-tests/queries/function_octet_length.json |    60 +
 .../queries/function_octet_length.proto.bin        |   Bin 0 -> 549 bytes
 .../query-tests/queries/function_overlay.json      |   102 +
 .../query-tests/queries/function_overlay.proto.bin |   Bin 0 -> 910 bytes
 .../queries/function_overlay_with_len.json         |   123 +
 .../queries/function_overlay_with_len.proto.bin    |   Bin 0 -> 1095 bytes
 .../query-tests/queries/function_parse_json.json   |    60 +
 .../queries/function_parse_json.proto.bin          |   Bin 0 -> 545 bytes
 .../query-tests/queries/function_parse_url.json    |    81 +
 .../queries/function_parse_url.proto.bin           |   Bin 0 -> 728 bytes
 .../queries/function_parse_url_with_key.json       |   102 +
 .../queries/function_parse_url_with_key.proto.bin  |   Bin 0 -> 913 bytes
 .../query-tests/queries/function_percent_rank.json |   107 +
 .../queries/function_percent_rank.proto.bin        |   Bin 0 -> 910 bytes
 .../queries/function_percentile_approx.json        |   102 +
 .../queries/function_percentile_approx.proto.bin   |   Bin 0 -> 934 bytes
 .../function_percentile_with_frequency.json        |   102 +
 .../function_percentile_with_frequency.proto.bin   |   Bin 0 -> 920 bytes
 .../function_percentile_without_frequency.json     |    81 +
 ...function_percentile_without_frequency.proto.bin |   Bin 0 -> 736 bytes
 .../resources/query-tests/queries/function_pi.json |    38 +
 .../query-tests/queries/function_pi.proto.bin      |   Bin 0 -> 343 bytes
 .../query-tests/queries/function_pmod.json         |    81 +
 .../query-tests/queries/function_pmod.proto.bin    |   Bin 0 -> 717 bytes
 .../query-tests/queries/function_posexplode.json   |    60 +
 .../queries/function_posexplode.proto.bin          |   Bin 0 -> 545 bytes
 .../queries/function_posexplode_outer.json         |    60 +
 .../queries/function_posexplode_outer.proto.bin    |   Bin 0 -> 557 bytes
 .../query-tests/queries/function_position.json     |    81 +
 .../queries/function_position.proto.bin            |   Bin 0 -> 726 bytes
 .../queries/function_position_with_start.json      |   102 +
 .../queries/function_position_with_start.proto.bin |   Bin 0 -> 911 bytes
 .../query-tests/queries/function_positive.json     |    60 +
 .../queries/function_positive.proto.bin            |   Bin 0 -> 541 bytes
 .../query-tests/queries/function_pow.json          |    81 +
 .../query-tests/queries/function_pow.proto.bin     |   Bin 0 -> 718 bytes
 .../query-tests/queries/function_power.json        |    81 +
 .../query-tests/queries/function_power.proto.bin   |   Bin 0 -> 720 bytes
 .../query-tests/queries/function_printf.json       |   102 +
 .../query-tests/queries/function_printf.proto.bin  |   Bin 0 -> 907 bytes
 .../query-tests/queries/function_product.json      |    60 +
 .../query-tests/queries/function_product.proto.bin |   Bin 0 -> 539 bytes
 .../query-tests/queries/function_quarter.json      |    60 +
 .../query-tests/queries/function_quarter.proto.bin |   Bin 0 -> 485 bytes
 .../query-tests/queries/function_radians.json      |    60 +
 .../query-tests/queries/function_radians.proto.bin |   Bin 0 -> 543 bytes
 .../query-tests/queries/function_raise_error.json  |    60 +
 .../queries/function_raise_error.proto.bin         |   Bin 0 -> 552 bytes
 .../queries/function_rand_with_seed.json           |    60 +
 .../queries/function_rand_with_seed.proto.bin      |   Bin 0 -> 534 bytes
 .../queries/function_randn_with_seed.json          |    60 +
 .../queries/function_randn_with_seed.proto.bin     |   Bin 0 -> 537 bytes
 .../queries/function_random_with_seed.json         |    60 +
 .../queries/function_random_with_seed.proto.bin    |   Bin 0 -> 536 bytes
 .../query-tests/queries/function_rank.json         |   107 +
 .../query-tests/queries/function_rank.proto.bin    |   Bin 0 -> 894 bytes
 .../query-tests/queries/function_reduce.json       |   166 +
 .../query-tests/queries/function_reduce.proto.bin  |   Bin 0 -> 1322 bytes
 .../query-tests/queries/function_reflect.json      |   102 +
 .../query-tests/queries/function_reflect.proto.bin |   Bin 0 -> 931 bytes
 .../query-tests/queries/function_regexp.json       |    81 +
 .../query-tests/queries/function_regexp.proto.bin  |   Bin 0 -> 728 bytes
 .../query-tests/queries/function_regexp_count.json |    81 +
 .../queries/function_regexp_count.proto.bin        |   Bin 0 -> 736 bytes
 .../queries/function_regexp_extract.json           |   102 +
 .../queries/function_regexp_extract.proto.bin      |   Bin 0 -> 954 bytes
 ..._regexp_extract_all_with_regex_group_index.json |   102 +
 ...xp_extract_all_with_regex_group_index.proto.bin |   Bin 0 -> 942 bytes
 ...gexp_extract_all_without_regex_group_index.json |    81 +
 ...extract_all_without_regex_group_index.proto.bin |   Bin 0 -> 758 bytes
 ...nction_regexp_instr_with_regex_group_index.json |   102 +
 ...n_regexp_instr_with_regex_group_index.proto.bin |   Bin 0 -> 927 bytes
 ...ion_regexp_instr_without_regex_group_index.json |    81 +
 ...egexp_instr_without_regex_group_index.proto.bin |   Bin 0 -> 743 bytes
 .../query-tests/queries/function_regexp_like.json  |    81 +
 .../queries/function_regexp_like.proto.bin         |   Bin 0 -> 738 bytes
 .../queries/function_regexp_replace.json           |   102 +
 .../queries/function_regexp_replace.proto.bin      |   Bin 0 -> 951 bytes
 .../queries/function_regexp_substr.json            |    81 +
 .../queries/function_regexp_substr.proto.bin       |   Bin 0 -> 747 bytes
 .../query-tests/queries/function_regr_avgx.json    |    81 +
 .../queries/function_regr_avgx.proto.bin           |   Bin 0 -> 728 bytes
 .../query-tests/queries/function_regr_avgy.json    |    81 +
 .../queries/function_regr_avgy.proto.bin           |   Bin 0 -> 728 bytes
 .../query-tests/queries/function_regr_count.json   |    81 +
 .../queries/function_regr_count.proto.bin          |   Bin 0 -> 730 bytes
 .../queries/function_regr_intercept.json           |    81 +
 .../queries/function_regr_intercept.proto.bin      |   Bin 0 -> 738 bytes
 .../query-tests/queries/function_regr_r2.json      |    81 +
 .../query-tests/queries/function_regr_r2.proto.bin |   Bin 0 -> 724 bytes
 .../query-tests/queries/function_regr_slope.json   |    81 +
 .../queries/function_regr_slope.proto.bin          |   Bin 0 -> 730 bytes
 .../query-tests/queries/function_regr_sxx.json     |    81 +
 .../queries/function_regr_sxx.proto.bin            |   Bin 0 -> 726 bytes
 .../query-tests/queries/function_regr_sxy.json     |    81 +
 .../queries/function_regr_sxy.proto.bin            |   Bin 0 -> 726 bytes
 .../query-tests/queries/function_regr_syy.json     |    81 +
 .../queries/function_regr_syy.proto.bin            |   Bin 0 -> 726 bytes
 .../query-tests/queries/function_replace.json      |    81 +
 .../query-tests/queries/function_replace.proto.bin |   Bin 0 -> 724 bytes
 .../function_replace_with_specified_string.json    |   102 +
 ...unction_replace_with_specified_string.proto.bin |   Bin 0 -> 909 bytes
 .../query-tests/queries/function_reverse.json      |    60 +
 .../query-tests/queries/function_reverse.proto.bin |   Bin 0 -> 539 bytes
 .../query-tests/queries/function_right.json        |    81 +
 .../query-tests/queries/function_right.proto.bin   |   Bin 0 -> 720 bytes
 .../query-tests/queries/function_rint.json         |    60 +
 .../query-tests/queries/function_rint.proto.bin    |   Bin 0 -> 534 bytes
 .../query-tests/queries/function_rlike.json        |    81 +
 .../query-tests/queries/function_rlike.proto.bin   |   Bin 0 -> 726 bytes
 .../query-tests/queries/function_round.json        |    81 +
 .../query-tests/queries/function_round.proto.bin   |   Bin 0 -> 721 bytes
 .../query-tests/queries/function_row_number.json   |   107 +
 .../queries/function_row_number.proto.bin          |   Bin 0 -> 906 bytes
 .../query-tests/queries/function_rpad.json         |   102 +
 .../query-tests/queries/function_rpad.proto.bin    |   Bin 0 -> 904 bytes
 .../query-tests/queries/function_rpad_binary.json  |   102 +
 .../queries/function_rpad_binary.proto.bin         |   Bin 0 -> 802 bytes
 .../query-tests/queries/function_rtrim.json        |    60 +
 .../query-tests/queries/function_rtrim.proto.bin   |   Bin 0 -> 535 bytes
 .../queries/function_rtrim_with_pattern.json       |    81 +
 .../queries/function_rtrim_with_pattern.proto.bin  |   Bin 0 -> 724 bytes
 .../queries/function_schema_of_csv.json            |   125 +
 .../queries/function_schema_of_csv.proto.bin       |   Bin 0 -> 1147 bytes
 .../queries/function_schema_of_json.json           |    60 +
 .../queries/function_schema_of_json.proto.bin      |   Bin 0 -> 564 bytes
 .../function_schema_of_json_with_options.json      |   125 +
 .../function_schema_of_json_with_options.proto.bin |   Bin 0 -> 1183 bytes
 .../queries/function_schema_of_variant.json        |    83 +
 .../queries/function_schema_of_variant.proto.bin   |   Bin 0 -> 763 bytes
 .../queries/function_schema_of_variant_agg.json    |    83 +
 .../function_schema_of_variant_agg.proto.bin       |   Bin 0 -> 771 bytes
 .../query-tests/queries/function_sec.json          |    60 +
 .../query-tests/queries/function_sec.proto.bin     |   Bin 0 -> 531 bytes
 .../query-tests/queries/function_second.json       |    60 +
 .../query-tests/queries/function_second.proto.bin  |   Bin 0 -> 483 bytes
 .../query-tests/queries/function_sentences.json    |    60 +
 .../queries/function_sentences.proto.bin           |   Bin 0 -> 543 bytes
 .../queries/function_sentences_with_language.json  |    81 +
 .../function_sentences_with_language.proto.bin     |   Bin 0 -> 729 bytes
 ...nction_sentences_with_language_and_country.json |   102 +
 ...n_sentences_with_language_and_country.proto.bin |   Bin 0 -> 915 bytes
 .../query-tests/queries/function_sequence.json     |    81 +
 .../queries/function_sequence.proto.bin            |   Bin 0 -> 724 bytes
 .../query-tests/queries/function_session_user.json |    38 +
 .../queries/function_session_user.proto.bin        |   Bin 0 -> 363 bytes
 .../queries/function_session_window.json           |    81 +
 .../queries/function_session_window.proto.bin      |   Bin 0 -> 704 bytes
 .../queries/function_session_window_orphaned.json  |    34 +
 .../function_session_window_orphaned.proto.bin     |   Bin 0 -> 165 bytes
 .../query-tests/queries/function_sha.json          |    60 +
 .../query-tests/queries/function_sha.proto.bin     |   Bin 0 -> 531 bytes
 .../query-tests/queries/function_sha1.json         |    85 +
 .../query-tests/queries/function_sha1.proto.bin    |   Bin 0 -> 714 bytes
 .../query-tests/queries/function_sha2.json         |   106 +
 .../query-tests/queries/function_sha2.proto.bin    |   Bin 0 -> 900 bytes
 .../query-tests/queries/function_shiftleft.json    |    81 +
 .../queries/function_shiftleft.proto.bin           |   Bin 0 -> 733 bytes
 .../query-tests/queries/function_shiftright.json   |    81 +
 .../queries/function_shiftright.proto.bin          |   Bin 0 -> 736 bytes
 .../queries/function_shiftrightunsigned.json       |    81 +
 .../queries/function_shiftrightunsigned.proto.bin  |   Bin 0 -> 760 bytes
 .../query-tests/queries/function_sign.json         |    60 +
 .../query-tests/queries/function_sign.proto.bin    |   Bin 0 -> 533 bytes
 .../query-tests/queries/function_signum.json       |    60 +
 .../query-tests/queries/function_signum.proto.bin  |   Bin 0 -> 540 bytes
 .../query-tests/queries/function_sin.json          |    60 +
 .../query-tests/queries/function_sin.proto.bin     |   Bin 0 -> 531 bytes
 .../query-tests/queries/function_sinh.json         |    60 +
 .../query-tests/queries/function_sinh.proto.bin    |   Bin 0 -> 534 bytes
 .../query-tests/queries/function_size.json         |    60 +
 .../query-tests/queries/function_size.proto.bin    |   Bin 0 -> 533 bytes
 .../query-tests/queries/function_skewness.json     |    60 +
 .../queries/function_skewness.proto.bin            |   Bin 0 -> 546 bytes
 .../query-tests/queries/function_slice.json        |   102 +
 .../query-tests/queries/function_slice.proto.bin   |   Bin 0 -> 907 bytes
 .../query-tests/queries/function_some.json         |    60 +
 .../query-tests/queries/function_some.proto.bin    |   Bin 0 -> 427 bytes
 .../query-tests/queries/function_sort_array.json   |    81 +
 .../queries/function_sort_array.proto.bin          |   Bin 0 -> 736 bytes
 .../queries/function_spark_partition_id.json       |    38 +
 .../queries/function_spark_partition_id.proto.bin  |   Bin 0 -> 375 bytes
 .../query-tests/queries/function_split.json        |    81 +
 .../query-tests/queries/function_split.proto.bin   |   Bin 0 -> 722 bytes
 .../query-tests/queries/function_split_part.json   |   102 +
 .../queries/function_split_part.proto.bin          |   Bin 0 -> 915 bytes
 .../queries/function_split_using_columns.json      |    81 +
 .../queries/function_split_using_columns.proto.bin |   Bin 0 -> 720 bytes
 .../queries/function_split_with_limit.json         |   102 +
 .../queries/function_split_with_limit.proto.bin    |   Bin 0 -> 908 bytes
 .../function_split_with_limit_using_columns.json   |   102 +
 ...nction_split_with_limit_using_columns.proto.bin |   Bin 0 -> 905 bytes
 .../query-tests/queries/function_sqrt.json         |    60 +
 .../query-tests/queries/function_sqrt.proto.bin    |   Bin 0 -> 534 bytes
 .../query-tests/queries/function_stack.json        |   123 +
 .../query-tests/queries/function_stack.proto.bin   |   Bin 0 -> 1089 bytes
 .../query-tests/queries/function_startswith.json   |    81 +
 .../queries/function_startswith.proto.bin          |   Bin 0 -> 730 bytes
 .../query-tests/queries/function_std.json          |    60 +
 .../query-tests/queries/function_std.proto.bin     |   Bin 0 -> 531 bytes
 .../query-tests/queries/function_stddev.json       |    60 +
 .../query-tests/queries/function_stddev.proto.bin  |   Bin 0 -> 540 bytes
 .../query-tests/queries/function_stddev_pop.json   |    60 +
 .../queries/function_stddev_pop.proto.bin          |   Bin 0 -> 552 bytes
 .../query-tests/queries/function_stddev_samp.json  |    60 +
 .../queries/function_stddev_samp.proto.bin         |   Bin 0 -> 555 bytes
 .../query-tests/queries/function_str_to_map.json   |    60 +
 .../queries/function_str_to_map.proto.bin          |   Bin 0 -> 545 bytes
 ...tr_to_map_with_pair_and_keyValue_delimiter.json |    81 +
 ..._map_with_pair_and_keyValue_delimiter.proto.bin |   Bin 0 -> 730 bytes
 .../function_str_to_map_with_pair_delimiter.json   |   102 +
 ...nction_str_to_map_with_pair_delimiter.proto.bin |   Bin 0 -> 915 bytes
 .../query-tests/queries/function_struct.json       |    81 +
 .../query-tests/queries/function_struct.proto.bin  |   Bin 0 -> 728 bytes
 .../query-tests/queries/function_substr.json       |    81 +
 .../query-tests/queries/function_substr.proto.bin  |   Bin 0 -> 722 bytes
 .../queries/function_substr_with_len.json          |   102 +
 .../queries/function_substr_with_len.proto.bin     |   Bin 0 -> 907 bytes
 .../query-tests/queries/function_substring.json    |   102 +
 .../queries/function_substring.proto.bin           |   Bin 0 -> 923 bytes
 .../queries/function_substring_index.json          |   102 +
 .../queries/function_substring_index.proto.bin     |   Bin 0 -> 948 bytes
 .../queries/function_substring_using_columns.json  |   102 +
 .../function_substring_using_columns.proto.bin     |   Bin 0 -> 913 bytes
 .../query-tests/queries/function_sum.json          |    60 +
 .../query-tests/queries/function_sum.proto.bin     |   Bin 0 -> 531 bytes
 .../query-tests/queries/function_sum_distinct.json |    61 +
 .../queries/function_sum_distinct.proto.bin        |   Bin 0 -> 542 bytes
 .../query-tests/queries/function_tan.json          |    60 +
 .../query-tests/queries/function_tan.proto.bin     |   Bin 0 -> 531 bytes
 .../query-tests/queries/function_tanh.json         |    60 +
 .../query-tests/queries/function_tanh.proto.bin    |   Bin 0 -> 534 bytes
 .../queries/function_timestamp_add.json            |   102 +
 .../queries/function_timestamp_add.proto.bin       |   Bin 0 -> 879 bytes
 .../queries/function_timestamp_diff.json           |   102 +
 .../queries/function_timestamp_diff.proto.bin      |   Bin 0 -> 882 bytes
 .../queries/function_timestamp_micros.json         |    60 +
 .../queries/function_timestamp_micros.proto.bin    |   Bin 0 -> 503 bytes
 .../queries/function_timestamp_millis.json         |    60 +
 .../queries/function_timestamp_millis.proto.bin    |   Bin 0 -> 503 bytes
 .../queries/function_timestamp_seconds.json        |    60 +
 .../queries/function_timestamp_seconds.proto.bin   |   Bin 0 -> 505 bytes
 .../query-tests/queries/function_to_binary.json    |    60 +
 .../queries/function_to_binary.proto.bin           |   Bin 0 -> 543 bytes
 .../queries/function_to_binary_with_format.json    |    81 +
 .../function_to_binary_with_format.proto.bin       |   Bin 0 -> 732 bytes
 .../query-tests/queries/function_to_char.json      |    81 +
 .../query-tests/queries/function_to_char.proto.bin |   Bin 0 -> 729 bytes
 .../query-tests/queries/function_to_csv.json       |   125 +
 .../query-tests/queries/function_to_csv.proto.bin  |   Bin 0 -> 1108 bytes
 .../query-tests/queries/function_to_date.json      |    60 +
 .../query-tests/queries/function_to_date.proto.bin |   Bin 0 -> 485 bytes
 .../queries/function_to_date_with_format.json      |    81 +
 .../queries/function_to_date_with_format.proto.bin |   Bin 0 -> 683 bytes
 .../query-tests/queries/function_to_json.json      |   125 +
 .../query-tests/queries/function_to_json.proto.bin |   Bin 0 -> 1134 bytes
 .../query-tests/queries/function_to_number.json    |    81 +
 .../queries/function_to_number.proto.bin           |   Bin 0 -> 733 bytes
 .../query-tests/queries/function_to_timestamp.json |    60 +
 .../queries/function_to_timestamp.proto.bin        |   Bin 0 -> 495 bytes
 .../queries/function_to_timestamp_ltz.json         |    60 +
 .../queries/function_to_timestamp_ltz.proto.bin    |   Bin 0 -> 557 bytes
 .../function_to_timestamp_ltz_with_format.json     |    81 +
 ...function_to_timestamp_ltz_with_format.proto.bin |   Bin 0 -> 742 bytes
 .../queries/function_to_timestamp_ntz.json         |    60 +
 .../queries/function_to_timestamp_ntz.proto.bin    |   Bin 0 -> 557 bytes
 .../function_to_timestamp_ntz_with_format.json     |    81 +
 ...function_to_timestamp_ntz_with_format.proto.bin |   Bin 0 -> 742 bytes
 .../queries/function_to_timestamp_with_format.json |    81 +
 .../function_to_timestamp_with_format.proto.bin    |   Bin 0 -> 712 bytes
 .../queries/function_to_unix_timestamp.json        |    60 +
 .../queries/function_to_unix_timestamp.proto.bin   |   Bin 0 -> 559 bytes
 .../function_to_unix_timestamp_with_format.json    |    81 +
 ...unction_to_unix_timestamp_with_format.proto.bin |   Bin 0 -> 744 bytes
 .../queries/function_to_utc_timestamp.json         |    81 +
 .../queries/function_to_utc_timestamp.proto.bin    |   Bin 0 -> 706 bytes
 .../query-tests/queries/function_to_varchar.json   |    81 +
 .../queries/function_to_varchar.proto.bin          |   Bin 0 -> 735 bytes
 .../query-tests/queries/function_transform.json    |   132 +
 .../queries/function_transform.proto.bin           |   Bin 0 -> 1110 bytes
 .../queries/function_transform_keys.json           |   160 +
 .../queries/function_transform_keys.proto.bin      |   Bin 0 -> 1338 bytes
 .../queries/function_transform_values.json         |   117 +
 .../queries/function_transform_values.proto.bin    |   Bin 0 -> 971 bytes
 .../queries/function_transform_with_index.json     |   117 +
 .../function_transform_with_index.proto.bin        |   Bin 0 -> 946 bytes
 .../query-tests/queries/function_translate.json    |   102 +
 .../queries/function_translate.proto.bin           |   Bin 0 -> 929 bytes
 .../query-tests/queries/function_trim.json         |    60 +
 .../query-tests/queries/function_trim.proto.bin    |   Bin 0 -> 533 bytes
 .../queries/function_trim_with_pattern.json        |    81 +
 .../queries/function_trim_with_pattern.proto.bin   |   Bin 0 -> 721 bytes
 .../query-tests/queries/function_trunc.json        |    81 +
 .../query-tests/queries/function_trunc.proto.bin   |   Bin 0 -> 669 bytes
 .../query-tests/queries/function_try_add.json      |    81 +
 .../query-tests/queries/function_try_add.proto.bin |   Bin 0 -> 724 bytes
 .../queries/function_try_aes_decrypt.json          |    81 +
 .../queries/function_try_aes_decrypt.proto.bin     |   Bin 0 -> 740 bytes
 .../function_try_aes_decrypt_with_mode.json        |   102 +
 .../function_try_aes_decrypt_with_mode.proto.bin   |   Bin 0 -> 925 bytes
 ...function_try_aes_decrypt_with_mode_padding.json |   123 +
 ...ion_try_aes_decrypt_with_mode_padding.proto.bin |   Bin 0 -> 1110 bytes
 ...tion_try_aes_decrypt_with_mode_padding_aad.json |   144 +
 ...try_aes_decrypt_with_mode_padding_aad.proto.bin |   Bin 0 -> 1295 bytes
 .../query-tests/queries/function_try_avg.json      |    60 +
 .../query-tests/queries/function_try_avg.proto.bin |   Bin 0 -> 539 bytes
 .../query-tests/queries/function_try_divide.json   |    81 +
 .../queries/function_try_divide.proto.bin          |   Bin 0 -> 730 bytes
 .../queries/function_try_element_at_array.json     |    81 +
 .../function_try_element_at_array.proto.bin        |   Bin 0 -> 738 bytes
 .../queries/function_try_element_at_map.json       |    81 +
 .../queries/function_try_element_at_map.proto.bin  |   Bin 0 -> 738 bytes
 .../queries/function_try_make_interval_years.json  |    60 +
 .../function_try_make_interval_years.proto.bin     |   Bin 0 -> 559 bytes
 .../function_try_make_interval_years_months.json   |    81 +
 ...nction_try_make_interval_years_months.proto.bin |   Bin 0 -> 744 bytes
 ...ction_try_make_interval_years_months_weeks.json |   102 +
 ..._try_make_interval_years_months_weeks.proto.bin |   Bin 0 -> 929 bytes
 ..._try_make_interval_years_months_weeks_days.json |   123 +
 ...make_interval_years_months_weeks_days.proto.bin |   Bin 0 -> 1114 bytes
 ...ake_interval_years_months_weeks_days_hours.json |   144 +
 ...nterval_years_months_weeks_days_hours.proto.bin |   Bin 0 -> 1299 bytes
 ...nterval_years_months_weeks_days_hours_mins.json |   165 +
 ...al_years_months_weeks_days_hours_mins.proto.bin |   Bin 0 -> 1484 bytes
 ...al_years_months_weeks_days_hours_mins_secs.json |   186 +
 ...ars_months_weeks_days_hours_mins_secs.proto.bin |   Bin 0 -> 1669 bytes
 ...ction_try_make_timestamp_ltz_with_timezone.json |   186 +
 ..._try_make_timestamp_ltz_with_timezone.proto.bin |   Bin 0 -> 1679 bytes
 ...on_try_make_timestamp_ltz_without_timezone.json |   165 +
 ...y_make_timestamp_ltz_without_timezone.proto.bin |   Bin 0 -> 1494 bytes
 .../queries/function_try_make_timestamp_ntz.json   |   165 +
 .../function_try_make_timestamp_ntz.proto.bin      |   Bin 0 -> 1494 bytes
 .../function_try_make_timestamp_with_timezone.json |   186 +
 ...tion_try_make_timestamp_with_timezone.proto.bin |   Bin 0 -> 1671 bytes
 ...nction_try_make_timestamp_without_timezone.json |   165 +
 ...n_try_make_timestamp_without_timezone.proto.bin |   Bin 0 -> 1486 bytes
 .../query-tests/queries/function_try_multiply.json |    81 +
 .../queries/function_try_multiply.proto.bin        |   Bin 0 -> 734 bytes
 .../queries/function_try_parse_json.json           |    60 +
 .../queries/function_try_parse_json.proto.bin      |   Bin 0 -> 553 bytes
 .../queries/function_try_parse_url.json            |    81 +
 .../queries/function_try_parse_url.proto.bin       |   Bin 0 -> 736 bytes
 .../queries/function_try_parse_url_with_key.json   |   102 +
 .../function_try_parse_url_with_key.proto.bin      |   Bin 0 -> 921 bytes
 .../query-tests/queries/function_try_reflect.json  |   102 +
 .../queries/function_try_reflect.proto.bin         |   Bin 0 -> 939 bytes
 .../query-tests/queries/function_try_subtract.json |    81 +
 .../queries/function_try_subtract.proto.bin        |   Bin 0 -> 734 bytes
 .../query-tests/queries/function_try_sum.json      |    60 +
 .../query-tests/queries/function_try_sum.proto.bin |   Bin 0 -> 539 bytes
 .../queries/function_try_to_binary.json            |    81 +
 .../queries/function_try_to_binary.proto.bin       |   Bin 0 -> 741 bytes
 .../function_try_to_binary_without_format.json     |    60 +
 ...function_try_to_binary_without_format.proto.bin |   Bin 0 -> 551 bytes
 .../query-tests/queries/function_try_to_date.json  |    60 +
 .../queries/function_try_to_date.proto.bin         |   Bin 0 -> 493 bytes
 .../queries/function_try_to_date_with_format.json  |    81 +
 .../function_try_to_date_with_format.proto.bin     |   Bin 0 -> 695 bytes
 .../queries/function_try_to_number.json            |    81 +
 .../queries/function_try_to_number.proto.bin       |   Bin 0 -> 741 bytes
 .../queries/function_try_to_timestamp.json         |    81 +
 .../queries/function_try_to_timestamp.proto.bin    |   Bin 0 -> 742 bytes
 .../function_try_to_timestamp_without_format.json  |    60 +
 ...ction_try_to_timestamp_without_format.proto.bin |   Bin 0 -> 557 bytes
 .../queries/function_try_url_decode.json           |    60 +
 .../queries/function_try_url_decode.proto.bin      |   Bin 0 -> 553 bytes
 .../queries/function_try_variant_get.json          |   125 +
 .../queries/function_try_variant_get.proto.bin     |   Bin 0 -> 1155 bytes
 .../query-tests/queries/function_typedLit.json     |  2061 ++
 .../queries/function_typedLit.proto.bin            |   Bin 0 -> 11681 bytes
 .../query-tests/queries/function_typeof.json       |    60 +
 .../query-tests/queries/function_typeof.proto.bin  |   Bin 0 -> 537 bytes
 .../query-tests/queries/function_ucase.json        |    60 +
 .../query-tests/queries/function_ucase.proto.bin   |   Bin 0 -> 535 bytes
 .../query-tests/queries/function_unbase64.json     |    60 +
 .../queries/function_unbase64.proto.bin            |   Bin 0 -> 541 bytes
 .../query-tests/queries/function_unhex.json        |    60 +
 .../query-tests/queries/function_unhex.proto.bin   |   Bin 0 -> 535 bytes
 .../query-tests/queries/function_unix_date.json    |   104 +
 .../queries/function_unix_date.proto.bin           |   Bin 0 -> 885 bytes
 .../query-tests/queries/function_unix_micros.json  |   104 +
 .../queries/function_unix_micros.proto.bin         |   Bin 0 -> 918 bytes
 .../query-tests/queries/function_unix_millis.json  |   104 +
 .../queries/function_unix_millis.proto.bin         |   Bin 0 -> 918 bytes
 .../query-tests/queries/function_unix_seconds.json |   104 +
 .../queries/function_unix_seconds.proto.bin        |   Bin 0 -> 920 bytes
 .../queries/function_unix_timestamp.json           |    61 +
 .../queries/function_unix_timestamp.proto.bin      |   Bin 0 -> 528 bytes
 .../function_unix_timestamp_with_format.json       |    81 +
 .../function_unix_timestamp_with_format.proto.bin  |   Bin 0 -> 718 bytes
 .../query-tests/queries/function_upper.json        |    60 +
 .../query-tests/queries/function_upper.proto.bin   |   Bin 0 -> 535 bytes
 .../query-tests/queries/function_url_decode.json   |    60 +
 .../queries/function_url_decode.proto.bin          |   Bin 0 -> 545 bytes
 .../query-tests/queries/function_url_encode.json   |    60 +
 .../queries/function_url_encode.proto.bin          |   Bin 0 -> 545 bytes
 .../query-tests/queries/function_user.json         |    38 +
 .../query-tests/queries/function_user.proto.bin    |   Bin 0 -> 347 bytes
 .../query-tests/queries/function_var_pop.json      |    60 +
 .../query-tests/queries/function_var_pop.proto.bin |   Bin 0 -> 543 bytes
 .../query-tests/queries/function_var_samp.json     |    60 +
 .../queries/function_var_samp.proto.bin            |   Bin 0 -> 546 bytes
 .../query-tests/queries/function_variance.json     |    60 +
 .../queries/function_variance.proto.bin            |   Bin 0 -> 546 bytes
 .../query-tests/queries/function_variant_get.json  |   125 +
 .../queries/function_variant_get.proto.bin         |   Bin 0 -> 1139 bytes
 .../query-tests/queries/function_weekday.json      |    60 +
 .../query-tests/queries/function_weekday.proto.bin |   Bin 0 -> 485 bytes
 .../query-tests/queries/function_weekofyear.json   |    60 +
 .../queries/function_weekofyear.proto.bin          |   Bin 0 -> 491 bytes
 .../query-tests/queries/function_window.json       |   123 +
 .../query-tests/queries/function_window.proto.bin  |   Bin 0 -> 1068 bytes
 .../query-tests/queries/function_window_time.json  |    94 +
 .../queries/function_window_time.proto.bin         |   Bin 0 -> 739 bytes
 .../query-tests/queries/function_xpath.json        |    81 +
 .../query-tests/queries/function_xpath.proto.bin   |   Bin 0 -> 675 bytes
 .../queries/function_xpath_boolean.json            |    81 +
 .../queries/function_xpath_boolean.proto.bin       |   Bin 0 -> 684 bytes
 .../query-tests/queries/function_xpath_double.json |    81 +
 .../queries/function_xpath_double.proto.bin        |   Bin 0 -> 682 bytes
 .../query-tests/queries/function_xpath_float.json  |    81 +
 .../queries/function_xpath_float.proto.bin         |   Bin 0 -> 680 bytes
 .../query-tests/queries/function_xpath_int.json    |    81 +
 .../queries/function_xpath_int.proto.bin           |   Bin 0 -> 676 bytes
 .../query-tests/queries/function_xpath_long.json   |    81 +
 .../queries/function_xpath_long.proto.bin          |   Bin 0 -> 678 bytes
 .../query-tests/queries/function_xpath_number.json |    81 +
 .../queries/function_xpath_number.proto.bin        |   Bin 0 -> 682 bytes
 .../query-tests/queries/function_xpath_short.json  |    81 +
 .../queries/function_xpath_short.proto.bin         |   Bin 0 -> 680 bytes
 .../query-tests/queries/function_xpath_string.json |    81 +
 .../queries/function_xpath_string.proto.bin        |   Bin 0 -> 682 bytes
 .../query-tests/queries/function_xxhash64.json     |   123 +
 .../queries/function_xxhash64.proto.bin            |   Bin 0 -> 1097 bytes
 .../query-tests/queries/function_year.json         |    60 +
 .../query-tests/queries/function_year.proto.bin    |   Bin 0 -> 479 bytes
 .../query-tests/queries/function_years.json        |    60 +
 .../query-tests/queries/function_years.proto.bin   |   Bin 0 -> 531 bytes
 .../query-tests/queries/function_zip_with.json     |   138 +
 .../queries/function_zip_with.proto.bin            |   Bin 0 -> 1128 bytes
 .../resources/query-tests/queries/groupby_agg.json |   398 +
 .../query-tests/queries/groupby_agg.proto.bin      |   Bin 0 -> 3139 bytes
 .../query-tests/queries/groupby_agg_columns.json   |   127 +
 .../queries/groupby_agg_columns.proto.bin          |   Bin 0 -> 983 bytes
 .../query-tests/queries/groupby_agg_orphaned.json  |   101 +
 .../queries/groupby_agg_orphaned.proto.bin         |   Bin 0 -> 210 bytes
 .../query-tests/queries/groupby_agg_string.json    |   152 +
 .../queries/groupby_agg_string.proto.bin           |   Bin 0 -> 1161 bytes
 .../resources/query-tests/queries/groupby_avg.json |   129 +
 .../query-tests/queries/groupby_avg.proto.bin      |   Bin 0 -> 1017 bytes
 .../query-tests/queries/groupby_count.json         |   105 +
 .../query-tests/queries/groupby_count.proto.bin    |   Bin 0 -> 757 bytes
 .../resources/query-tests/queries/groupby_max.json |   129 +
 .../query-tests/queries/groupby_max.proto.bin      |   Bin 0 -> 1017 bytes
 .../query-tests/queries/groupby_mean.json          |   129 +
 .../query-tests/queries/groupby_mean.proto.bin     |   Bin 0 -> 1017 bytes
 .../resources/query-tests/queries/groupby_min.json |   129 +
 .../query-tests/queries/groupby_min.proto.bin      |   Bin 0 -> 1017 bytes
 .../resources/query-tests/queries/groupby_sum.json |   129 +
 .../query-tests/queries/groupby_sum.proto.bin      |   Bin 0 -> 1017 bytes
 .../query-tests/queries/groupingSets.json          |   154 +
 .../query-tests/queries/groupingSets.proto.bin     |   Bin 0 -> 1147 bytes
 .../queries/grouping_and_grouping_id.json          |   215 +
 .../queries/grouping_and_grouping_id.proto.bin     |   Bin 0 -> 1805 bytes
 .../test/resources/query-tests/queries/hint.json   |    38 +
 .../resources/query-tests/queries/hint.proto.bin   |   Bin 0 -> 248 bytes
 .../query-tests/queries/hll_sketch_agg.json        |    60 +
 .../query-tests/queries/hll_sketch_agg.proto.bin   |   Bin 0 -> 448 bytes
 .../queries/hll_sketch_agg_with_columnName.json    |    60 +
 .../hll_sketch_agg_with_columnName.proto.bin       |   Bin 0 -> 459 bytes
 ...l_sketch_agg_with_columnName_lgConfigK_int.json |    81 +
 ...tch_agg_with_columnName_lgConfigK_int.proto.bin |   Bin 0 -> 654 bytes
 .../hll_sketch_agg_with_column_lgConfigK.json      |    81 +
 .../hll_sketch_agg_with_column_lgConfigK.proto.bin |   Bin 0 -> 632 bytes
 .../hll_sketch_agg_with_column_lgConfigK_int.json  |    81 +
 ..._sketch_agg_with_column_lgConfigK_int.proto.bin |   Bin 0 -> 643 bytes
 .../query-tests/queries/hll_union_agg.json         |    60 +
 .../query-tests/queries/hll_union_agg.proto.bin    |   Bin 0 -> 446 bytes
 .../queries/hll_union_agg_with_columnName.json     |    60 +
 .../hll_union_agg_with_columnName.proto.bin        |   Bin 0 -> 456 bytes
 ...columnName_allowDifferentLgConfigK_boolean.json |    81 +
 ...nName_allowDifferentLgConfigK_boolean.proto.bin |   Bin 0 -> 650 bytes
 ...on_agg_with_column_allowDifferentLgConfigK.json |    81 +
 ...g_with_column_allowDifferentLgConfigK.proto.bin |   Bin 0 -> 630 bytes
 ...ith_column_allowDifferentLgConfigK_boolean.json |    81 +
 ...olumn_allowDifferentLgConfigK_boolean.proto.bin |   Bin 0 -> 640 bytes
 .../resources/query-tests/queries/intersect.json   |    25 +
 .../query-tests/queries/intersect.proto.bin        |   Bin 0 -> 94 bytes
 .../query-tests/queries/intersectAll.json          |    25 +
 .../query-tests/queries/intersectAll.proto.bin     |   Bin 0 -> 94 bytes
 .../query-tests/queries/join_condition.json        |   106 +
 .../query-tests/queries/join_condition.proto.bin   |   Bin 0 -> 686 bytes
 .../query-tests/queries/join_inner_condition.json  |   106 +
 .../queries/join_inner_condition.proto.bin         |   Bin 0 -> 684 bytes
 .../queries/join_inner_no_condition.json           |    24 +
 .../queries/join_inner_no_condition.proto.bin      |   Bin 0 -> 98 bytes
 .../join_inner_using_multiple_col_array.json       |    25 +
 .../join_inner_using_multiple_col_array.proto.bin  |   Bin 0 -> 105 bytes
 .../queries/join_inner_using_multiple_col_seq.json |    25 +
 .../join_inner_using_multiple_col_seq.proto.bin    |   Bin 0 -> 105 bytes
 .../queries/join_inner_using_single_col.json       |    25 +
 .../queries/join_inner_using_single_col.proto.bin  |   Bin 0 -> 102 bytes
 .../queries/join_using_multiple_col_array.json     |    25 +
 .../join_using_multiple_col_array.proto.bin        |   Bin 0 -> 105 bytes
 .../queries/join_using_multiple_col_seq.json       |    25 +
 .../queries/join_using_multiple_col_seq.proto.bin  |   Bin 0 -> 105 bytes
 .../query-tests/queries/join_using_single_col.json |    25 +
 .../queries/join_using_single_col.proto.bin        |   Bin 0 -> 102 bytes
 .../query-tests/queries/json_from_dataset.json     |    39 +
 .../queries/json_from_dataset.proto.bin            |   Bin 0 -> 180 bytes
 .../test/resources/query-tests/queries/limit.json  |    16 +
 .../resources/query-tests/queries/limit.proto.bin  |   Bin 0 -> 50 bytes
 .../query-tests/queries/melt_no_values.json        |    60 +
 .../query-tests/queries/melt_no_values.proto.bin   |   Bin 0 -> 434 bytes
 .../resources/query-tests/queries/melt_values.json |    63 +
 .../query-tests/queries/melt_values.proto.bin      |   Bin 0 -> 437 bytes
 .../test/resources/query-tests/queries/offset.json |    16 +
 .../resources/query-tests/queries/offset.proto.bin |   Bin 0 -> 51 bytes
 .../query-tests/queries/orderBy_columns.json       |    92 +
 .../query-tests/queries/orderBy_columns.proto.bin  |   Bin 0 -> 628 bytes
 .../query-tests/queries/orderBy_strings.json       |    92 +
 .../query-tests/queries/orderBy_strings.proto.bin  |   Bin 0 -> 640 bytes
 .../test/resources/query-tests/queries/pivot.json  |   115 +
 .../resources/query-tests/queries/pivot.proto.bin  |   Bin 0 -> 801 bytes
 .../queries/pivot_without_column_values.json       |   108 +
 .../queries/pivot_without_column_values.proto.bin  |   Bin 0 -> 789 bytes
 .../test/resources/query-tests/queries/range.json  |    11 +
 .../resources/query-tests/queries/range.proto.bin  |   Bin 0 -> 14 bytes
 .../test/resources/query-tests/queries/read.json   |    16 +
 .../resources/query-tests/queries/read.proto.bin   |   Bin 0 -> 142 bytes
 .../resources/query-tests/queries/read_csv.json    |    11 +
 .../query-tests/queries/read_csv.proto.bin         |   Bin 0 -> 76 bytes
 .../resources/query-tests/queries/read_jdbc.json   |    14 +
 .../query-tests/queries/read_jdbc.proto.bin        |   Bin 0 -> 101 bytes
 .../queries/read_jdbc_with_partition.json          |    18 +
 .../queries/read_jdbc_with_partition.proto.bin     |   Bin 0 -> 177 bytes
 .../queries/read_jdbc_with_predicates.json         |    15 +
 .../queries/read_jdbc_with_predicates.proto.bin    |   Bin 0 -> 121 bytes
 .../resources/query-tests/queries/read_json.json   |    11 +
 .../query-tests/queries/read_json.proto.bin        |   Bin 0 -> 78 bytes
 .../resources/query-tests/queries/read_orc.json    |    11 +
 .../query-tests/queries/read_orc.proto.bin         |   Bin 0 -> 75 bytes
 .../query-tests/queries/read_parquet.json          |    11 +
 .../query-tests/queries/read_parquet.proto.bin     |   Bin 0 -> 83 bytes
 .../resources/query-tests/queries/read_table.json  |    10 +
 .../query-tests/queries/read_table.proto.bin       |   Bin 0 -> 17 bytes
 .../resources/query-tests/queries/read_text.json   |    11 +
 .../query-tests/queries/read_text.proto.bin        |   Bin 0 -> 77 bytes
 .../query-tests/queries/relation_extension.json    |    16 +
 .../queries/relation_extension.proto.bin           |   Bin 0 -> 108 bytes
 .../queries/relation_extension_deprecated.json     |    16 +
 .../relation_extension_deprecated.proto.bin        |   Bin 0 -> 108 bytes
 .../resources/query-tests/queries/repartition.json |    17 +
 .../query-tests/queries/repartition.proto.bin      |   Bin 0 -> 53 bytes
 .../query-tests/queries/repartitionById.json       |    67 +
 .../query-tests/queries/repartitionById.proto.bin  |   Bin 0 -> 426 bytes
 .../queries/repartitionByRange_expressions.json    |   104 +
 .../repartitionByRange_expressions.proto.bin       |   Bin 0 -> 794 bytes
 ...artitionByRange_num_partitions_expressions.json |   105 +
 ...ionByRange_num_partitions_expressions.proto.bin |   Bin 0 -> 821 bytes
 .../queries/repartition_expressions.json           |    58 +
 .../queries/repartition_expressions.proto.bin      |   Bin 0 -> 421 bytes
 .../repartition_num_partitions_expressions.json    |    59 +
 ...epartition_num_partitions_expressions.proto.bin |   Bin 0 -> 423 bytes
 .../resources/query-tests/queries/replace.json     |    24 +
 .../query-tests/queries/replace.proto.bin          |   Bin 0 -> 77 bytes
 .../query-tests/queries/rollup_column.json         |   126 +
 .../query-tests/queries/rollup_column.proto.bin    |   Bin 0 -> 937 bytes
 .../query-tests/queries/rollup_string.json         |   128 +
 .../query-tests/queries/rollup_string.proto.bin    |   Bin 0 -> 961 bytes
 .../resources/query-tests/queries/sampleBy.json    |    49 +
 .../query-tests/queries/sampleBy.proto.bin         |   Bin 0 -> 306 bytes
 .../query-tests/queries/sample_fraction_seed.json  |    18 +
 .../queries/sample_fraction_seed.proto.bin         |   Bin 0 -> 64 bytes
 .../sample_withReplacement_fraction_seed.json      |    18 +
 .../sample_withReplacement_fraction_seed.proto.bin |   Bin 0 -> 62 bytes
 .../test/resources/query-tests/queries/select.json |    37 +
 .../resources/query-tests/queries/select.proto.bin |   Bin 0 -> 235 bytes
 .../resources/query-tests/queries/selectExpr.json  |    58 +
 .../query-tests/queries/selectExpr.proto.bin       |   Bin 0 -> 462 bytes
 .../queries/select_collated_string.json            |    37 +
 .../queries/select_collated_string.proto.bin       |   Bin 0 -> 243 bytes
 .../query-tests/queries/select_strings.json        |    58 +
 .../query-tests/queries/select_strings.proto.bin   |   Bin 0 -> 432 bytes
 .../query-tests/queries/select_typed_1-arg.json    |   127 +
 .../queries/select_typed_1-arg.proto.bin           |   Bin 0 -> 1013 bytes
 .../query-tests/queries/select_typed_2-arg.json    |   146 +
 .../queries/select_typed_2-arg.proto.bin           |   Bin 0 -> 1182 bytes
 .../query-tests/queries/select_typed_3-arg.json    |   211 +
 .../queries/select_typed_3-arg.proto.bin           |   Bin 0 -> 1749 bytes
 .../query-tests/queries/select_typed_4-arg.json    |   276 +
 .../queries/select_typed_4-arg.proto.bin           |   Bin 0 -> 2317 bytes
 .../query-tests/queries/select_typed_5-arg.json    |   341 +
 .../queries/select_typed_5-arg.proto.bin           |   Bin 0 -> 2882 bytes
 .../queries/sortWithinPartitions_columns.json      |    67 +
 .../queries/sortWithinPartitions_columns.proto.bin |   Bin 0 -> 436 bytes
 .../queries/sortWithinPartitions_strings.json      |    67 +
 .../queries/sortWithinPartitions_strings.proto.bin |   Bin 0 -> 476 bytes
 .../query-tests/queries/sort_columns.json          |    67 +
 .../query-tests/queries/sort_columns.proto.bin     |   Bin 0 -> 436 bytes
 .../query-tests/queries/sort_strings.json          |    67 +
 .../query-tests/queries/sort_strings.proto.bin     |   Bin 0 -> 443 bytes
 .../queries/streaming_table_API_with_options.json  |    15 +
 .../streaming_table_API_with_options.proto.bin     |   Bin 0 -> 55 bytes
 .../resources/query-tests/queries/summary.json     |    16 +
 .../query-tests/queries/summary.proto.bin          |   Bin 0 -> 60 bytes
 .../test/resources/query-tests/queries/table.json  |    10 +
 .../resources/query-tests/queries/table.proto.bin  |   Bin 0 -> 17 bytes
 .../queries/table_API_with_options.json            |    14 +
 .../queries/table_API_with_options.proto.bin       |   Bin 0 -> 44 bytes
 .../query-tests/queries/test_broadcast.json        |    33 +
 .../query-tests/queries/test_broadcast.proto.bin   |   Bin 0 -> 122 bytes
 .../src/test/resources/query-tests/queries/to.json |    34 +
 .../resources/query-tests/queries/to.proto.bin     |   Bin 0 -> 77 bytes
 .../test/resources/query-tests/queries/toDF.json   |    16 +
 .../resources/query-tests/queries/toDF.proto.bin   |   Bin 0 -> 61 bytes
 .../test/resources/query-tests/queries/toJSON.json |   105 +
 .../resources/query-tests/queries/toJSON.proto.bin |   Bin 0 -> 941 bytes
 .../query-tests/queries/to_avro_with_schema.json   |    81 +
 .../queries/to_avro_with_schema.proto.bin          |   Bin 0 -> 565 bytes
 .../queries/to_avro_without_schema.json            |    60 +
 .../queries/to_avro_without_schema.proto.bin       |   Bin 0 -> 387 bytes
 .../queries/to_protobuf_messageClassName.json      |   197 +
 .../queries/to_protobuf_messageClassName.proto.bin |   Bin 0 -> 1740 bytes
 .../to_protobuf_messageClassName_descFilePath.json |   218 +
 ...rotobuf_messageClassName_descFilePath.proto.bin |   Bin 0 -> 2135 bytes
 ...obuf_messageClassName_descFilePath_options.json |   283 +
 ...messageClassName_descFilePath_options.proto.bin |   Bin 0 -> 2654 bytes
 .../to_protobuf_messageClassName_options.json      |   262 +
 .../to_protobuf_messageClassName_options.proto.bin |   Bin 0 -> 2270 bytes
 .../queries/transpose_index_column.json            |    37 +
 .../queries/transpose_index_column.proto.bin       |   Bin 0 -> 236 bytes
 .../queries/transpose_no_index_column.json         |    15 +
 .../queries/transpose_no_index_column.proto.bin    |   Bin 0 -> 49 bytes
 .../test/resources/query-tests/queries/union.json  |    25 +
 .../resources/query-tests/queries/union.proto.bin  |   Bin 0 -> 94 bytes
 .../resources/query-tests/queries/unionAll.json    |    25 +
 .../query-tests/queries/unionAll.proto.bin         |   Bin 0 -> 94 bytes
 .../resources/query-tests/queries/unionByName.json |    43 +
 .../query-tests/queries/unionByName.proto.bin      |   Bin 0 -> 135 bytes
 .../queries/unionByName_allowMissingColumns.json   |    27 +
 .../unionByName_allowMissingColumns.proto.bin      |   Bin 0 -> 104 bytes
 .../query-tests/queries/unpivot_no_values.json     |    39 +
 .../queries/unpivot_no_values.proto.bin            |   Bin 0 -> 249 bytes
 .../query-tests/queries/unpivot_values.json        |    84 +
 .../query-tests/queries/unpivot_values.proto.bin   |   Bin 0 -> 622 bytes
 .../query-tests/queries/where_column.json          |    81 +
 .../query-tests/queries/where_column.proto.bin     |   Bin 0 -> 606 bytes
 .../resources/query-tests/queries/where_expr.json  |    37 +
 .../query-tests/queries/where_expr.proto.bin       |   Bin 0 -> 251 bytes
 .../query-tests/queries/width_bucket.json          |   123 +
 .../query-tests/queries/width_bucket.proto.bin     |   Bin 0 -> 997 bytes
 .../test/resources/query-tests/queries/window.json |   745 +
 .../resources/query-tests/queries/window.proto.bin |   Bin 0 -> 5728 bytes
 .../queries/withColumnRenamed_java_map.json        |    22 +
 .../queries/withColumnRenamed_java_map.proto.bin   |   Bin 0 -> 72 bytes
 .../queries/withColumnRenamed_scala_map.json       |    22 +
 .../queries/withColumnRenamed_scala_map.proto.bin  |   Bin 0 -> 72 bytes
 .../queries/withColumnRenamed_single.json          |    19 +
 .../queries/withColumnRenamed_single.proto.bin     |   Bin 0 -> 60 bytes
 .../query-tests/queries/withColumn_single.json     |    40 +
 .../queries/withColumn_single.proto.bin            |   Bin 0 -> 248 bytes
 .../query-tests/queries/withColumns_java_map.json  |    64 +
 .../queries/withColumns_java_map.proto.bin         |   Bin 0 -> 435 bytes
 .../query-tests/queries/withColumns_scala_map.json |    64 +
 .../queries/withColumns_scala_map.proto.bin        |   Bin 0 -> 446 bytes
 .../query-tests/queries/withMetadata.json          |    42 +
 .../query-tests/queries/withMetadata.proto.bin     |   Bin 0 -> 294 bytes
 .../resources/query-tests/test-data/people.csv     |     3 +
 .../resources/query-tests/test-data/people.json    |     3 +
 .../resources/query-tests/test-data/people.txt     |     3 +
 .../query-tests/test-data/streaming/csv/people.csv |     3 +
 .../query-tests/test-data/streaming/txt/people.txt |     3 +
 .../test/resources/query-tests/test-data/users.orc |   Bin 0 -> 547 bytes
 .../resources/query-tests/test-data/users.parquet  |   Bin 0 -> 615 bytes
 .../test/scala/org/apache/spark/sql/TestUDFs.scala |    43 +
 sql/connect/docs/adding-proto-messages.md          |    40 +
 sql/connect/docs/client-connection-string.md       |   144 +
 sql/connect/server/README.md                       |     5 +
 sql/connect/server/pom.xml                         |   394 +
 .../spark/sql/connect/plugin/CommandPlugin.java    |    32 +
 .../spark/sql/connect/plugin/ExpressionPlugin.java |    34 +
 .../spark/sql/connect/plugin/MLBackendPlugin.java  |    37 +
 .../spark/sql/connect/plugin/RelationPlugin.java   |    35 +
 .../sql/connect/SimpleSparkConnectService.scala    |    64 +
 .../spark/sql/connect/SparkConnectPlugin.scala     |    58 +
 .../apache/spark/sql/connect/config/Connect.scala  |   453 +
 .../config/ConnectPlanCompressionAlgorithm.scala   |    21 +
 .../connect/execution/CachedStreamResponse.scala   |    32 +
 .../ConnectProgressExecutionListener.scala         |   197 +
 .../execution/ExecuteGrpcResponseSender.scala      |   418 +
 .../execution/ExecuteResponseObserver.scala        |   367 +
 .../connect/execution/ExecuteThreadRunner.scala    |   367 +
 .../execution/SparkConnectPlanExecution.scala      |   368 +
 .../org/apache/spark/sql/connect/ml/MLCache.scala  |   283 +
 .../apache/spark/sql/connect/ml/MLException.scala  |    59 +
 .../apache/spark/sql/connect/ml/MLHandler.scala    |   567 +
 .../org/apache/spark/sql/connect/ml/MLUtils.scala  |   742 +
 .../apache/spark/sql/connect/ml/Serializer.scala   |   217 +
 .../connect/pipelines/DataflowGraphRegistry.scala  |    77 +
 .../connect/pipelines/PipelineEventSender.scala    |   176 +
 .../sql/connect/pipelines/PipelinesHandler.scala   |   531 +
 .../sql/connect/planner/InvalidInputErrors.scala   |   213 +
 .../planner/LiteralExpressionProtoConverter.scala  |    39 +
 .../spark/sql/connect/planner/MockObserver.scala   |    43 +
 .../sql/connect/planner/SaveModeConverter.scala    |    49 +
 .../sql/connect/planner/SparkConnectPlanner.scala  |  4335 +++
 ...SparkConnectStreamingQueryListenerHandler.scala |   128 +
 .../planner/StreamingForeachBatchHelper.scala      |   281 +
 .../planner/StreamingQueryListenerHelper.scala     |   109 +
 .../connect/planner/TableSaveMethodConverter.scala |    40 +
 .../plugin/SparkConnectPluginRegistry.scala        |   185 +
 .../sql/connect/service/ExecuteEventsManager.scala |   477 +
 .../spark/sql/connect/service/ExecuteHolder.scala  |   414 +
 .../LocalPropertiesCleanupInterceptor.scala        |    51 +
 .../sql/connect/service/LoggingInterceptor.scala   |    78 +
 .../PreSharedKeyAuthenticationInterceptor.scala    |    47 +
 .../sql/connect/service/SessionEventsManager.scala |   126 +
 .../spark/sql/connect/service/SessionHolder.scala  |   622 +
 .../service/SparkConnectAddArtifactsHandler.scala  |   320 +
 .../service/SparkConnectAnalyzeHandler.scala       |   238 +
 .../SparkConnectArtifactStatusesHandler.scala      |    71 +
 .../service/SparkConnectCloneSessionHandler.scala  |    62 +
 .../service/SparkConnectConfigHandler.scala        |   204 +
 .../service/SparkConnectExecutePlanHandler.scala   |    62 +
 .../service/SparkConnectExecutionManager.scala     |   366 +
 .../SparkConnectFetchErrorDetailsHandler.scala     |    59 +
 .../service/SparkConnectInterceptorRegistry.scala  |   117 +
 .../service/SparkConnectInterruptHandler.scala     |    74 +
 .../service/SparkConnectListenerBusListener.scala  |   177 +
 .../SparkConnectReattachExecuteHandler.scala       |    67 +
 .../SparkConnectReleaseExecuteHandler.scala        |    70 +
 .../SparkConnectReleaseSessionHandler.scala        |    46 +
 .../sql/connect/service/SparkConnectServer.scala   |    58 +
 .../sql/connect/service/SparkConnectService.scala  |   567 +
 .../service/SparkConnectSessionManager.scala       |   403 +
 .../service/SparkConnectStreamingQueryCache.scala  |   338 +
 .../ui/SparkConnectServerAppStatusStore.scala      |   131 +
 .../ui/SparkConnectServerHistoryServerPlugin.scala |    41 +
 .../connect/ui/SparkConnectServerListener.scala    |   452 +
 .../sql/connect/ui/SparkConnectServerPage.scala    |   526 +
 .../connect/ui/SparkConnectServerSessionPage.scala |   131 +
 .../sql/connect/ui/SparkConnectServerTab.scala     |    59 +
 .../org/apache/spark/sql/connect/ui/ToolTips.scala |    39 +
 .../spark/sql/connect/utils/ErrorUtils.scala       |   350 +
 .../spark/sql/connect/utils/MetricGenerator.scala  |    78 +
 .../utils/PipelineAnalysisContextUtils.scala       |    85 +
 .../sql/connect/utils/PlanCompressionUtils.scala   |   121 +
 .../services/org.apache.spark.ml.Estimator         |    20 +
 .../services/org.apache.spark.ml.Transformer       |    21 +
 .../org.apache.spark.ml.evaluation.Evaluator       |    21 +
 .../server/src/test/resources/log4j2.properties    |    44 +
 sql/connect/server/src/test/resources/udf          |   Bin 0 -> 973 bytes
 .../sql/connect/ProtoToParsedPlanTestSuite.scala   |   264 +
 .../spark/sql/connect/PythonTestDepsChecker.scala  |    44 +
 .../apache/spark/sql/connect/ResourceHelper.scala  |    32 +
 .../spark/sql/connect/SparkConnectServerTest.scala |   394 +
 .../sql/connect/SparkConnectServerTestSuite.scala  |   207 +
 .../spark/sql/connect/SparkConnectTestUtils.scala  |    36 +
 .../org/apache/spark/sql/connect/dsl/package.scala |  1207 +
 .../ConnectProgressExecutionListenerSuite.scala    |   177 +
 .../execution/ReattachableExecuteSuite.scala       |   506 +
 .../sql/connect/messages/AbbreviateSuite.scala     |   333 +
 .../messages/ConnectProtoMessagesSuite.scala       |    89 +
 .../spark/sql/connect/ml/MLBackendSuite.scala      |   199 +
 .../org/apache/spark/sql/connect/ml/MLHelper.scala |   423 +
 .../org/apache/spark/sql/connect/ml/MLSuite.scala  |   490 +
 .../sql/connect/pipelines/EndToEndAPISuite.scala   |   181 +
 .../pipelines/PipelineEventSenderSuite.scala       |   231 +
 .../pipelines/PipelineEventStreamSuite.scala       |   176 +
 .../pipelines/PipelineRefreshFunctionalSuite.scala |   361 +
 .../connect/pipelines/PythonPipelineSuite.scala    |  1180 +
 .../SparkDeclarativePipelinesServerSuite.scala     |   947 +
 .../SparkDeclarativePipelinesServerTest.scala      |   166 +
 .../connect/pipelines/TestPipelineDefinition.scala |   152 +
 .../connect/planner/InvalidInputErrorsSuite.scala  |   144 +
 .../LiteralExpressionProtoConverterSuite.scala     |   233 +
 .../connect/planner/SparkConnectPlannerSuite.scala |  1072 +
 .../planner/SparkConnectPlannerTestUtils.scala     |    62 +
 .../connect/planner/SparkConnectProtoSuite.scala   |  1158 +
 .../connect/planner/SparkConnectServiceSuite.scala |  1125 +
 .../SparkConnectWithSessionExtensionSuite.scala    |    87 +
 .../planner/StreamingForeachBatchHelperSuite.scala |    81 +
 .../plugin/SparkConnectPluginRegistrySuite.scala   |   270 +
 .../connect/service/AddArtifactsHandlerSuite.scala |   504 +
 .../service/ArtifactStatusesHandlerSuite.scala     |    99 +
 .../service/ExecuteEventsManagerSuite.scala        |   384 +
 .../service/FetchErrorDetailsHandlerSuite.scala    |   372 +
 .../connect/service/InterceptorRegistrySuite.scala |   197 +
 .../service/SessionEventsManagerSuite.scala        |   102 +
 .../connect/service/SparkConnectAuthSuite.scala    |    46 +
 .../service/SparkConnectCloneSessionSuite.scala    |   278 +
 .../SparkConnectListenerBusListenerSuite.scala     |   241 +
 .../service/SparkConnectServiceE2ESuite.scala      |   530 +
 .../SparkConnectServiceInternalServerSuite.scala   |   321 +
 .../service/SparkConnectSessionHolderSuite.scala   |   486 +
 .../service/SparkConnectSessionManagerSuite.scala  |   222 +
 .../SparkConnectStreamingQueryCacheSuite.scala     |   159 +
 .../ui/SparkConnectServerListenerSuite.scala       |   237 +
 .../connect/ui/SparkConnectServerPageSuite.scala   |   134 +
 sql/connect/shims/README.md                        |     1 +
 sql/connect/shims/pom.xml                          |    48 +
 .../src/main/scala/org/apache/spark/shims.scala    |    49 +
 .../AggregateBenchmark-jdk21-results.txt           |   148 +
 sql/core/benchmarks/AggregateBenchmark-results.txt |   148 +
 .../AnsiIntervalSortBenchmark-jdk21-results.txt    |    28 +
 .../AnsiIntervalSortBenchmark-results.txt          |    28 +
 .../benchmarks/Base64Benchmark-jdk21-results.txt   |    56 +
 sql/core/benchmarks/Base64Benchmark-results.txt    |    56 +
 .../BloomFilterBenchmark-jdk21-results.txt         |   196 +
 .../benchmarks/BloomFilterBenchmark-results.txt    |   196 +
 ...iltInDataSourceWriteBenchmark-jdk21-results.txt |    70 +
 .../BuiltInDataSourceWriteBenchmark-results.txt    |    70 +
 .../ByteArrayBenchmark-jdk21-results.txt           |    27 +
 sql/core/benchmarks/ByteArrayBenchmark-results.txt |    27 +
 sql/core/benchmarks/CSVBenchmark-jdk21-results.txt |    77 +
 sql/core/benchmarks/CSVBenchmark-results.txt       |    77 +
 .../CharVarcharBenchmark-jdk21-results.txt         |   122 +
 .../benchmarks/CharVarcharBenchmark-results.txt    |   122 +
 .../CollationBenchmark-jdk21-results.txt           |    88 +
 sql/core/benchmarks/CollationBenchmark-results.txt |    88 +
 .../CollationNonASCIIBenchmark-jdk21-results.txt   |    88 +
 .../CollationNonASCIIBenchmark-results.txt         |    88 +
 .../ColumnarBatchBenchmark-jdk21-results.txt       |    59 +
 .../benchmarks/ColumnarBatchBenchmark-results.txt  |    59 +
 .../CompressionSchemeBenchmark-jdk21-results.txt   |   137 +
 .../CompressionSchemeBenchmark-results.txt         |   137 +
 ...ConstantColumnVectorBenchmark-jdk21-results.txt |   280 +
 .../ConstantColumnVectorBenchmark-results.txt      |   280 +
 .../DataSourceReadBenchmark-jdk21-results.txt      |   438 +
 .../benchmarks/DataSourceReadBenchmark-results.txt |   438 +
 .../benchmarks/DatasetBenchmark-jdk21-results.txt  |    46 +
 sql/core/benchmarks/DatasetBenchmark-results.txt   |    46 +
 .../benchmarks/DateTimeBenchmark-jdk21-results.txt |   461 +
 sql/core/benchmarks/DateTimeBenchmark-results.txt  |   461 +
 .../DateTimeRebaseBenchmark-jdk21-results.txt      |   154 +
 .../benchmarks/DateTimeRebaseBenchmark-results.txt |   154 +
 .../benchmarks/EncodeBenchmark-jdk21-results.txt   |     8 +
 sql/core/benchmarks/EncodeBenchmark-results.txt    |     8 +
 ...ndOnlyUnsafeRowArrayBenchmark-jdk21-results.txt |    45 +
 ...alAppendOnlyUnsafeRowArrayBenchmark-results.txt |    45 +
 .../benchmarks/ExtractBenchmark-jdk21-results.txt  |   104 +
 sql/core/benchmarks/ExtractBenchmark-results.txt   |   104 +
 .../FilterPushdownBenchmark-jdk21-results.txt      |   734 +
 .../benchmarks/FilterPushdownBenchmark-results.txt |   734 +
 .../GenerateExecBenchmark-jdk21-results.txt        |    12 +
 .../benchmarks/GenerateExecBenchmark-results.txt   |    12 +
 ...ashedRelationMetricsBenchmark-jdk21-results.txt |    11 +
 .../HashedRelationMetricsBenchmark-results.txt     |    11 +
 .../InExpressionBenchmark-jdk21-results.txt        |   740 +
 .../benchmarks/InExpressionBenchmark-results.txt   |   740 +
 .../InMemoryColumnarBenchmark-jdk21-results.txt    |    12 +
 .../InMemoryColumnarBenchmark-results.txt          |    12 +
 ...ithDynamicPartitionsBenchmark-jdk21-results.txt |     8 +
 ...TableWithDynamicPartitionsBenchmark-results.txt |     8 +
 .../benchmarks/IntervalBenchmark-jdk21-results.txt |    40 +
 sql/core/benchmarks/IntervalBenchmark-results.txt  |    40 +
 .../benchmarks/JoinBenchmark-jdk21-results.txt     |    82 +
 sql/core/benchmarks/JoinBenchmark-results.txt      |    82 +
 .../benchmarks/JsonBenchmark-jdk21-results.txt     |   130 +
 sql/core/benchmarks/JsonBenchmark-results.txt      |   130 +
 .../benchmarks/LargeRowBenchmark-jdk21-results.txt |    26 +
 sql/core/benchmarks/LargeRowBenchmark-results.txt  |    26 +
 .../MakeDateTimeBenchmark-jdk21-results.txt        |    22 +
 .../benchmarks/MakeDateTimeBenchmark-results.txt   |    22 +
 .../MetadataStructBenchmark-jdk21-results.txt      |    46 +
 .../benchmarks/MetadataStructBenchmark-results.txt |    46 +
 .../MetricsAggregationBenchmark-jdk21-results.txt  |    12 +
 .../MetricsAggregationBenchmark-results.txt        |    12 +
 .../benchmarks/MiscBenchmark-jdk21-results.txt     |   127 +
 sql/core/benchmarks/MiscBenchmark-results.txt      |   127 +
 ...cNestedSchemaPruningBenchmark-jdk21-results.txt |    53 +
 .../OrcNestedSchemaPruningBenchmark-results.txt    |    53 +
 ...2NestedSchemaPruningBenchmark-jdk21-results.txt |    53 +
 .../OrcV2NestedSchemaPruningBenchmark-results.txt  |    53 +
 ...tedPredicatePushDownBenchmark-jdk21-results.txt |    21 +
 ...uetNestedPredicatePushDownBenchmark-results.txt |    21 +
 ...tNestedSchemaPruningBenchmark-jdk21-results.txt |    53 +
 ...ParquetNestedSchemaPruningBenchmark-results.txt |    53 +
 .../PrimitiveArrayBenchmark-jdk21-results.txt      |    12 +
 .../benchmarks/PrimitiveArrayBenchmark-results.txt |    12 +
 .../benchmarks/RangeBenchmark-jdk21-results.txt    |    15 +
 sql/core/benchmarks/RangeBenchmark-results.txt     |    15 +
 .../RecursiveCTEBenchmark-jdk21-results.txt        |    95 +
 .../benchmarks/RecursiveCTEBenchmark-results.txt   |    95 +
 .../SetOperationsBenchmark-jdk21-results.txt       |    13 +
 .../benchmarks/SetOperationsBenchmark-results.txt  |    13 +
 .../benchmarks/SortBenchmark-jdk21-results.txt     |    16 +
 sql/core/benchmarks/SortBenchmark-results.txt      |    16 +
 ...StoreBasicOperationsBenchmark-jdk21-results.txt |   144 +
 .../StateStoreBasicOperationsBenchmark-results.txt |   144 +
 .../StringFunctionsBenchmark-jdk21-results.txt     |    11 +
 .../StringFunctionsBenchmark-results.txt           |    11 +
 .../SubExprEliminationBenchmark-jdk21-results.txt  |    25 +
 .../SubExprEliminationBenchmark-results.txt        |    25 +
 .../TPCDSQueryBenchmark-jdk21-results.txt          |   810 +
 .../benchmarks/TPCDSQueryBenchmark-results.txt     |   810 +
 ...akeOrderedAndProjectBenchmark-jdk21-results.txt |    12 +
 .../TakeOrderedAndProjectBenchmark-results.txt     |    12 +
 .../benchmarks/TopKBenchmark-jdk21-results.txt     |    22 +
 sql/core/benchmarks/TopKBenchmark-results.txt      |    22 +
 sql/core/benchmarks/UDFBenchmark-jdk21-results.txt |    59 +
 sql/core/benchmarks/UDFBenchmark-results.txt       |    59 +
 .../UnsafeArrayDataBenchmark-jdk21-results.txt     |    33 +
 .../UnsafeArrayDataBenchmark-results.txt           |    33 +
 .../UpdateFieldsBenchmark-jdk21-results.txt        |    26 +
 .../benchmarks/UpdateFieldsBenchmark-results.txt   |    26 +
 .../V2FunctionBenchmark-jdk21-results.txt          |    44 +
 .../benchmarks/V2FunctionBenchmark-results.txt     |    44 +
 .../WideSchemaBenchmark-jdk21-results.txt          |   158 +
 .../benchmarks/WideSchemaBenchmark-results.txt     |   158 +
 .../WideTableBenchmark-jdk21-results.txt           |    17 +
 sql/core/benchmarks/WideTableBenchmark-results.txt |    17 +
 sql/core/pom.xml                                   |   435 +
 sql/core/src/main/buf.gen.yaml                     |    24 +
 sql/core/src/main/buf.work.yaml                    |    19 +
 .../parquet/filter2/predicate/SparkFilterApi.java  |    55 +
 .../spark/sql/avro/AvroCompressionCodec.java       |    71 +
 .../spark/sql/avro/SparkAvroKeyOutputFormat.java   |    99 +
 .../apache/spark/sql/connector/read/V1Scan.java    |    43 +
 .../apache/spark/sql/connector/write/V1Write.java  |    35 +
 .../spark/sql/execution/BufferedRowIterator.java   |    98 +
 .../sql/execution/RecordBinaryComparator.java      |    87 +
 .../sql/execution/UnsafeExternalRowSorter.java     |   259 +
 .../execution/UnsafeFixedWidthAggregationMap.java  |   249 +
 .../sql/execution/UnsafeKVExternalSorter.java      |   335 +
 .../sql/execution/columnar/ColumnDictionary.java   |    65 +
 .../datasources/HadoopLineRecordReader.java        |   298 +
 .../SchemaColumnConvertNotSupportedException.java  |    63 +
 .../datasources/orc/OrcArrayColumnVector.java      |   113 +
 .../datasources/orc/OrcAtomicColumnVector.java     |   162 +
 .../datasources/orc/OrcColumnStatistics.java       |    80 +
 .../execution/datasources/orc/OrcColumnVector.java |    83 +
 .../datasources/orc/OrcColumnVectorUtils.java      |    66 +
 .../datasources/orc/OrcColumnarBatchReader.java    |   238 +
 .../datasources/orc/OrcCompressionCodec.java       |    61 +
 .../execution/datasources/orc/OrcFooterReader.java |    67 +
 .../datasources/orc/OrcMapColumnVector.java        |   116 +
 .../datasources/orc/OrcStructColumnVector.java     |   106 +
 .../datasources/parquet/OpenedParquetFooter.java   |    34 +
 .../datasources/parquet/ParquetColumnVector.java   |   432 +
 .../parquet/ParquetCompressionCodec.java           |    76 +
 .../datasources/parquet/ParquetDictionary.java     |    76 +
 .../datasources/parquet/ParquetFooterReader.java   |   112 +
 .../datasources/parquet/ParquetReadState.java      |   182 +
 .../datasources/parquet/ParquetVectorUpdater.java  |    94 +
 .../parquet/ParquetVectorUpdaterFactory.java       |  1772 ++
 .../parquet/SpecificParquetRecordReaderBase.java   |   302 +
 .../parquet/VectorizedColumnReader.java            |   434 +
 .../parquet/VectorizedDeltaBinaryPackedReader.java |   331 +
 .../parquet/VectorizedDeltaByteArrayReader.java    |   144 +
... 3833781 lines suppressed ...


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org