[jira] [Assigned] (SPARK-57452) Auditing the migration guide

["Wenchen Fan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-57452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wenchen Fan reassigned SPARK-57452:
-----------------------------------

    Assignee: Wenchen Fan

> Auditing the migration guide
> ----------------------------
>
>                 Key: SPARK-57452
>                 URL: https://issues.apache.org/jira/browse/SPARK-57452
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Kubernetes, MLlib, PySpark, Spark Core, SQL, Structured Streaming, Web UI
>    Affects Versions: 4.2.0
>            Reporter: Xiao Li
>            Assignee: Wenchen Fan
>            Priority: Blocker
>              Labels: pull-request-available
>
> I used AI to analyze more than 1,900 commits from the Spark 4.2.0 release and identified 38 changes that appear to be missing from the migration guide.
> The JIRAs listed below were identified through this analysis. However, this may not be a complete list, so please also review the remaining commits for any additional migration guide updates that may be required.
>  
>  * *SPARK-55314[CONNECT] Propagate observed metrics errors to client*
> *Component:* CONNECT
> *Why no migration-guide note needed:* Should be documented: Observation.get (Scala and Connect) now raises the underlying exception when metric collection fails instead of returning an empty map; code that tolerated an empty result on failure now sees a thrown exception.
> *Proposed migration-guide message:* [SQL] Since Spark 4.2, {{Observation.get}} (Scala and Spark Connect) raises the underlying exception when observed-metric collection fails instead of silently returning an empty map. Code that tolerated an empty result on failure now sees a thrown exception. There is no opt-out.
>  * *SPARK-55655[MLLIB] Make {{CountVectorizer}} vocabulary deterministic when counts are equal*
> *Component:* MLLIB
> *Why no migration-guide note needed:* Should be documented: CountVectorizer now breaks ties between equal-count terms lexicographically, making the vocabulary deterministic; this can change vocabulary term order and feature indices versus prior (non-deterministic) output. No opt-out.
> *Proposed migration-guide message:* [Core] Since Spark 4.2, {{CountVectorizer}} breaks ties between equal-count terms lexicographically so the vocabulary is deterministic. This can change vocabulary term order and feature indices compared with prior (non-deterministic) output. There is no opt-out.
>  * *SPARK-47997[PS] Add errors parameter to DataFrame.drop and Series.drop*
> *Component:* PS
> *Why no migration-guide note needed:* Should be documented: ps DataFrame.drop/Series.drop now raise KeyError if ANY label is missing (previously only if all missing), for all pandas versions; pass errors='ignore' to skip missing labels.
> *Proposed migration-guide message:* [PySpark] In Spark 4.2, pandas-on-Spark DataFrame.drop/Series.drop add an {{errors}} parameter defaulting to {{'raise'}} and now raise KeyError if any requested label is missing (previously only if all were missing), across all pandas versions. To skip missing labels, pass {{{}errors='ignore'{}}}.
>  * *SPARK-56219[PS] Align groupby idxmax and idxmin skipna=False behavior with pandas 2/3*
> *Component:* PS
> *Why no migration-guide note needed:* Should be documented: pandas-on-Spark groupby idxmax/idxmin with skipna=False now returns null for NA groups (pandas 2) or raises on NA inputs (pandas 3) instead of a label; no opt-out, results change for existing skipna=False users.
> *Proposed migration-guide message:* [PySpark] In Spark 4.2, pandas-on-Spark {{GroupBy.idxmax}} and {{GroupBy.idxmin}} with {{skipna=False}} now follow pandas semantics for NA: with pandas 2 they return null for groups containing NA values, and with pandas 3 they raise on NA-containing inputs, instead of returning an index label. There is no opt-out.
>  * *SPARK-55977[PS] Fix isin() to use strict type matching like pandas*
> *Component:* PS
> *Why no migration-guide note needed:* Should be documented: ps Series/DataFrame.isin() now uses strict Python-type matching (e.g. 1 no longer matches '1'), changing results for all pandas versions; this is pandas-parity, no opt-out config.
> *Proposed migration-guide message:* [PySpark] In Spark 4.2, pandas-on-Spark Series/DataFrame {{isin()}} uses strict Python-type matching like pandas, so values of incompatible types no longer match (for example integer 1 no longer matches string '1'). Results change across all pandas versions. There is no opt-out.
>  * *SPARK-54568[PYTHON] Avoid unnecessary pandas conversion in create dataframe from ndarray*
> *Component:* PYTHON
> *Why no migration-guide note needed:* Should be documented: createDataFrame from a numpy ndarray now requires pyarrow and converts ndarray to Arrow directly, dropping np.dtype-based StructType inference so inferred schema can differ.
> *Proposed migration-guide message:* [PySpark] In Spark 4.2, createDataFrame from a NumPy ndarray converts the array directly to an Arrow Table and now requires PyArrow. The previous np.dtype-based StructType inference is dropped, so the inferred schema may differ. To control the schema, pass an explicit schema.
>  * *SPARK-56186[PYTHON] Retire pypy*
> *Component:* PYTHON
> *Why no migration-guide note needed:* Should be documented: PyPy is no longer officially supported in PySpark (CI, docker image, classifier, and PyPy-specific code removed); PyPy users should migrate to CPython.
> *Proposed migration-guide message:* [PySpark] In Spark 4.2, PyPy is no longer officially supported in PySpark: PyPy CI, the PyPy docker image, the setup.py classifier, and PyPy-specific code/test skips have been removed. PyPy users should migrate to CPython. There is no opt-out.
>  * *SPARK-55096[PYTHON] Update pandas minimum version in {{connect/setup.py}}*
> *Component:* PYTHON
> *Why no migration-guide note needed:* Should be documented: minimum pandas raised to 2.2.0 for Spark Connect (was 2.0.0); pandas <2.2 is no longer supported on Connect.
> *Proposed migration-guide message:* [PySpark] In Spark 4.2, the minimum supported version for pandas on Spark Connect has been raised from 2.0.0 to 2.2.0, matching the minimum already required by PySpark.
>  * *SPARK-54962[PYTHON] Fix nullable integers handling in Pandas UDF*
> *Component:* PYTHON
> *Why no migration-guide note needed:* Should be documented: Pandas UDFs on nullable integer columns containing nulls now use a nullable Int extension dtype instead of float64, so values/dtype inside the UDF change (fixing precision loss for large integers); no opt-out.
> *Proposed migration-guide message:* [PySpark] In Spark 4.2, Pandas UDFs on a nullable integer column that contains nulls receive a pandas nullable integer extension dtype (e.g. Int64) instead of float64, fixing precision loss for large integers. The dtype and values seen inside the UDF change accordingly. There is no opt-out configuration.
>  * *SPARK-55583[PYTHON] Validate Arrow schema types in Python data source*
> *Component:* PYTHON
> *Why no migration-guide note needed:* Should be documented: a Python Data Source read returning a pa.RecordBatch whose Arrow types differ from the declared schema now fails with DATA_SOURCE_RETURN_SCHEMA_MISMATCH; type-mismatched batches that previously loaded by coincidence now error.
> *Proposed migration-guide message:* [PySpark] In Spark 4.2, a Python data source read that returns a {{pa.RecordBatch}} whose Arrow types differ from the declared schema now fails with {{{}DATA_SOURCE_RETURN_SCHEMA_MISMATCH{}}}. Type-mismatched batches that previously loaded by coincidence now error. There is no opt-out.
>  * *SPARK-55416[SS][PYTHON] Streaming Python Data Source memory leak when end-offset is not updated*
> *Component:* SS,PYTHON
> *Why no migration-guide note needed:* Should be documented: a Streaming Python Data Source SimpleDataSourceStreamReader whose read() returns a non-empty batch with end==start now fails with STREAM_READER_OFFSET_DID_NOT_ADVANCE instead of leaking memory; affects existing (buggy) reader impls.
> *Proposed migration-guide message:* [SS] Since Spark 4.2, a streaming Python data source SimpleDataSourceStreamReader whose {{read()}} returns a non-empty batch with end offset equal to start now fails with {{SIMPLE_STREAM_READER_OFFSET_DID_NOT_ADVANCE}} instead of leaking driver memory. Empty batches with end == start are still allowed. There is no opt-out.
>  * *SPARK-56206[SQL] Fix case-insensitive duplicate CTE name detection*
> *Component:* SQL
> *Why no migration-guide note needed:* Should be documented: duplicate CTE names differing only in case (e.g. WITH cte1, CTE1) now raise DUPLICATED_CTE_NAMES instead of silently overwriting; previously-accepted queries now fail. No opt-out.
> *Proposed migration-guide message:* [SQL] Since Spark 4.2, duplicate CTE name detection is case-insensitive. CTE definitions whose names differ only in case (e.g. {{{}WITH cte AS (...), CTE AS (...){}}}) now raise {{DUPLICATED_CTE_NAMES}} instead of silently overwriting the earlier definition. There is no opt-out; rename the conflicting CTEs.
>  * *SPARK-56652[SQL] Always emit RELY/NORELY in DESCRIBE EXTENDED constraint output*
> *Component:* SQL
> *Why no migration-guide note needed:* Should be documented: DESCRIBE EXTENDED now always prints RELY/NORELY for table constraints (previously omitted the default NORELY), changing the command output text for tools parsing it.
> *Proposed migration-guide message:* [SQL] Since Spark 4.2, {{DESCRIBE EXTENDED}} always emits the {{{}RELY{}}}/{{{}NORELY{}}} token for table constraints, including {{NORELY}} for the default state which was previously omitted. This matches {{SHOW CREATE TABLE}} output and changes the command's constraint output text for tools parsing it.
>  * *SPARK-55019[SQL] Allow DROP TABLE to drop VIEW*
> *Component:* SQL
> *Why no migration-guide note needed:* Should be documented: DROP TABLE on a view now drops the view by default instead of raising WRONG_COMMAND_FOR_OBJECT_TYPE; restore via spark.sql.dropTableOnView.enabled=false.
> *Proposed migration-guide message:* [SQL] Since Spark 4.2, DROP TABLE on a view drops the view by default instead of raising {{{}WRONG_COMMAND_FOR_OBJECT_TYPE{}}}. To restore the previous behavior, set {{spark.sql.dropTableOnView.enabled}} to {{{}false{}}}.
>  * *SPARK-54853[SQL] Always check {{hive.exec.max.dynamic.partitions}} on the spark side*
> *Component:* SQL
> *Why no migration-guide note needed:* Should be documented: hive.exec.max.dynamic.partitions is now always enforced Spark-side and the session-level value is honored, changing when the limit error fires; error renamed to DYNAMIC_PARTITION_WRITE_PARTITION_NUM_LIMIT_EXCEEDED.
> *Proposed migration-guide message:* [SQL] Since Spark 4.2, the {{hive.exec.max.dynamic.partitions}} limit for dynamic partition writes to Hive SerDe tables is always enforced on the Spark side and honors the session-level value, changing when the limit is checked. The error is now reported as {{{}DYNAMIC_PARTITION_WRITE_PARTITION_NUM_LIMIT_EXCEEDED{}}}.
>  * *SPARK-55372[SQL] Fix {{SHOW CREATE TABLE}} for tables / views with default collation*
> *Component:* SQL
> *Why no migration-guide note needed:* Should be documented: typeName/toString of an explicitly UTF8_BINARY-collated StringType/CharType now render 'string collate UTF8_BINARY' not 'string' (default non-collated unchanged), changing SHOW CREATE TABLE and schema output for such columns.
> *Proposed migration-guide message:* [SQL] Since Spark 4.2, a StringType/CharType/VarcharType with an explicit {{UTF8_BINARY}} collation renders its collation in {{{}typeName{}}}/{{{}toString{}}} (for example {{{}string collate UTF8_BINARY{}}}), changing SHOW CREATE TABLE and schema output for such columns. Default non-collated strings are unchanged. No opt-out.
>  * *SPARK-54918[SQL] Normalize floating numbers in array set operations*
> *Component:* SQL
> *Why no migration-guide note needed:* Should be documented: array_distinct/union/intersect/except and arrays_overlap now normalize floats so 0.0/-0.0 and differently-bit NaNs are treated as equal, changing results of these array set operations. No opt-out.
> *Proposed migration-guide message:* [SQL] Since Spark 4.2, the array set functions {{{}array_distinct{}}}, {{{}array_union{}}}, {{{}array_intersect{}}}, {{{}array_except{}}}, and {{arrays_overlap}} normalize floating-point values, so {{0.0}} and {{-0.0}} and differently-bit NaN values are treated as equal. This changes the results of these functions; there is no opt-out.
>  * *SPARK-54777[SQL] Changed dropTable error handling in JDBCTableCatalog.dropTable(...)*
> *Component:* SQL
> *Why no migration-guide note needed:* Should be documented: JDBC DROP TABLE now only swallows object-not-found errors; other failures (permission, etc.) propagate instead of silently returning, so a drop that previously appeared to succeed now throws.
> *Proposed migration-guide message:* [SQL] Since Spark 4.2, the JDBC table catalog only swallows object-not-found errors when running DROP TABLE; other failures such as permission-denied or constraint violations now propagate instead of silently returning success. A DROP TABLE that previously appeared to succeed may now throw.
>  * *SPARK-57040[SQL] JDBC connector supports pushdown TABLESAMPLE SYSTEM*
> *Component:* SQL
> *Why no migration-guide note needed:* Should be documented: V2 JDBC TABLESAMPLE with withReplacement=true is no longer pushed down (correctness fix; pushdown default-on), so .sample(withReplacement=true) on JDBC tables now returns different (correct) results.
> *Proposed migration-guide message:* [SQL] Since Spark 4.2, the JDBC connector no longer pushes down {{TABLESAMPLE}} when {{withReplacement=true}} (a correctness fix, as no mainstream RDBMS supports sampling with replacement), and adds {{TABLESAMPLE SYSTEM}} pushdown for PostgreSQL. Results of sample-with-replacement on JDBC tables change accordingly.
>  * *SPARK-56031[SQL] Make Natural Join column matching respect case sensitivity conf*
> *Component:* SQL
> *Why no migration-guide note needed:* Should be documented: NATURAL JOIN now respects spark.sql.caseSensitive (default false), so joins on case-differing common columns instead of degrading to CROSS JOIN, changing results; set spark.sql.caseSensitive=true to match case-sensitively.
> *Proposed migration-guide message:* [SQL] Since Spark 4.2, NATURAL JOIN respects {{spark.sql.caseSensitive}} (default {{{}false{}}}), so common columns that differ only in case are joined instead of degrading to a CROSS JOIN, changing results. To match columns case-sensitively, set {{spark.sql.caseSensitive}} to {{{}true{}}}.
>  * *SPARK-31561[SQL] Add QUALIFY Clause*
> *Component:* SQL
> *Why no migration-guide note needed:* Should be documented: QUALIFY is now a (non-reserved) clause keyword, so a query using unquoted QUALIFY as a trailing table alias (FROM t QUALIFY) now parses as a QUALIFY clause; quote the identifier to restore.
> *Proposed migration-guide message:* [SQL] Since Spark 4.2, the {{QUALIFY}} clause is supported, and {{QUALIFY}} becomes a (non-reserved) clause keyword. A query using unquoted {{QUALIFY}} as a trailing table alias (e.g. {{{}FROM t QUALIFY{}}}) is now parsed as a {{QUALIFY}} clause. To restore the previous behavior, quote the alias (e.g. {{{}`QUALIFY`{}}}).
>  * *SPARK-57188[SQL] Parameterless function takes precedence over UDF parameter*
> *Component:* SQL
> *Why no migration-guide note needed:* Should be documented: a parameterless built-in (current_user, current_date, etc.) now takes precedence over a same-named SQL UDF parameter, changing UDF body results. Set spark.sql.legacy.allowUdfParameterToShadowParameterlessFunction=true to restore.
> *Proposed migration-guide message:* [SQL] Since Spark 4.2, a parameterless built-in function ({{{}current_user{}}}, {{{}current_date{}}}, {{{}session_user{}}}, etc.) takes precedence over a same-named SQL UDF parameter in the function body. To restore the previous behavior, set {{spark.sql.legacy.allowUdfParameterToShadowParameterlessFunction}} to {{{}true{}}}.
>  * *SPARK-56045[SQL] Add flag for ignoring Parquet UNKNOWN type annotation and revert to old behavior*
> *Component:* SQL
> *Why no migration-guide note needed:* Should be documented: reading Parquet files with UNKNOWN logical-type annotation now infers physical type (e.g. IntegerType) instead of NullType shipped in v4.1.0; opt back into NullType via spark.sql.parquet.reader.respectUnknownTypeAnnotation.enabled=true.
> *Proposed migration-guide message:* [SQL] Since Spark 4.2, reading Parquet files with the {{UNKNOWN}} logical-type annotation infers the physical type (for example IntegerType) instead of the NullType used in 4.1.0. To restore the 4.1.0 behavior of inferring NullType, set {{spark.sql.parquet.reader.respectUnknownTypeAnnotation.enabled}} to {{{}true{}}}.
>  * *SPARK-56414[SQL] Per-write options should take precedence over session config in file source writes*
> *Component:* SQL
> *Why no migration-guide note needed:* Should be documented: per-write options (e.g. parquet.outputTimestampType) now override the matching session SQLConf in Parquet/Avro writes; previously such options were silently ignored, so written file format can change when both are set.
> *Proposed migration-guide message:* [SQL] Since Spark 4.2, per-write options take precedence over session config for several Parquet/Avro write keys (e.g. {{{}spark.sql.parquet.outputTimestampType{}}}, {{{}spark.sql.parquet.writeLegacyFormat{}}}). Previously such options were silently ignored, so the written file format can change when both are set.
>  * *SPARK-56251[SQL] Add default fetchSize for postgres to avoid loading all data in memory*
> *Component:* SQL
> *Why no migration-guide note needed:* Should be documented: Postgres JDBC reads now default fetchSize to 1000 (was 0/all-in-memory), enabling cursor fetch with autoCommit=false; changes default read behavior. Set fetchsize=0 to restore.
> *Proposed migration-guide message:* [SQL] Since Spark 4.2, the PostgreSQL JDBC dialect defaults the read {{fetchsize}} to {{1000}} (was {{{}0{}}}), enabling cursor-based fetching with {{autoCommit=false}} to avoid loading the whole table into memory. To restore the previous behavior, set the {{fetchsize}} option to {{{}0{}}}.
>  * *SPARK-55155[SQL] Support foldable expressions in {{SET CATALOG}} statement*
> *Component:* SQL
> *Why no migration-guide note needed:* Should be documented: SET CATALOG with a bare name now resolves to a session variable of that name first (if one exists) before treating it as a catalog name; there is no opt-out config. Edge case but a default behavior change.
> *Proposed migration-guide message:* [SQL] Since Spark 4.2, SET CATALOG accepts foldable expressions and a bare name is first resolved as a session variable of that name (if one exists) before being treated as a catalog name. There is no opt-out; a session variable that shadows a catalog name changes which catalog is set.
>  * *SPARK-51518[SQL] Support | as an alternative to |> for the SQL pipe operator token*
> *Component:* SQL
> *Why no migration-guide note needed:* Should be documented: single-char '|' is now a SQL pipe-operator token by default, so a query using a pipe keyword as a column name after bitwise-OR may reparse; restore via spark.sql.parser.singleCharacterPipeOperator.enabled=false.
> *Proposed migration-guide message:* [SQL] Since Spark 4.2, the SQL parser accepts single-character {{|}} as an alternative to {{|>}} for the pipe operator token by default. A pipe keyword used as a column name after a bitwise-OR {{|}} may now reparse. To restore the previous behavior, set {{spark.sql.parser.singleCharacterPipeOperator.enabled}} to {{{}false{}}}.
>  * *SPARK-52812[SQL] Make Spark Connect Catalog.createTable eager*
> *Component:* SQL
> *Why no migration-guide note needed:* Should be documented: Spark Connect Catalog.createTable now executes eagerly instead of lazily, so the table is created (and errors like already-exists surface) immediately at the call rather than on a later action.
> *Proposed migration-guide message:* [SQL] Since Spark 4.2, Spark Connect {{Catalog.createTable}} executes eagerly: the table is created (and errors such as table-already-exists surface) immediately at the call rather than lazily on a later action. Code relying on the previous lazy behavior is affected.
>  * *SPARK-55198[SQL] spark-sql should skip comment line with leading whitespaces*
> *Component:* SQL
> *Why no migration-guide note needed:* Should be documented: spark-sql CLI now skips comment lines that have leading whitespace before – (line.trim startsWith --), matching Hive/beeline; such lines were previously sent as SQL. CLI-only, no opt-out.
> *Proposed migration-guide message:* [SQL] Since Spark 4.2, the spark-sql CLI skips comment lines whose first non-whitespace characters are {{-{-}{-}}} (i.e. {{line.trim}} starts with {{{}-{}}}), aligning with Hive and beeline. Previously such leading-whitespace comment lines were sent as SQL. There is no opt-out.
>  * *SPARK-49110[SQL] Simplify SubqueryAlias.metadataOutput to always propagate metadata columns*
> *Component:* SQL
> *Why no migration-guide note needed:* Should be documented: SubqueryAlias now always propagates metadata columns by default; queries that failed with AnalysisException may now succeed and joins may newly raise ambiguous-column errors. Restore via subqueryAliasAlwaysPropagateMetadataColumns=false.
> *Proposed migration-guide message:* [SQL] Since Spark 4.2, {{SubqueryAlias}} always propagates metadata columns from its child, so some queries that previously failed with AnalysisException now succeed and joins may raise new ambiguous-column errors. To restore the legacy behavior, set {{{}spark.sql.analyzer.subqueryAliasAlwaysPropagateMetadataColumns=false{}}}.
>  * *SPARK-56678[SQL] Use structured Catalog/Namespace/Table rows in DESCRIBE TABLE EXTENDED for v2 tables and views*
> *Component:* SQL
> *Why no migration-guide note needed:* Should be documented: DESCRIBE TABLE EXTENDED for v2 tables/views now emits structured Catalog/Namespace/Database/Table rows instead of a single Name/Identifier row; consumers parsing the output are affected.
> *Proposed migration-guide message:* [SQL] Since Spark 4.2, {{DESCRIBE TABLE EXTENDED}} for v2 tables and views emits structured {{{}Catalog{}}}, {{{}Namespace{}}}, {{{}Database{}}}, and {{{}Table{}}}/{{{}View{}}} rows instead of a single {{{}Name{}}}/{{{}Identifier{}}} row. Consumers that parse the command output may be affected.
>  * *SPARK-56654[SQL] Reject unpaired UTF-16 surrogates in Variant JSON parsing*
> *Component:* SQL
> *Why no migration-guide note needed:* Should be documented: parse_json/try_parse_json/from_json('variant') now reject unpaired UTF-16 surrogates (error/NULL) instead of substituting U+FFFD; previously-accepted JSON now fails. Set spark.sql.variant.validateUnicodeInJsonParsing=false to restore.
> *Proposed migration-guide message:* [SQL] Since Spark 4.2, {{{}parse_json{}}}, {{{}try_parse_json{}}}, and {{from_json}} to variant reject unpaired UTF-16 surrogates (raising an error or returning NULL) instead of silently substituting U+FFFD. To restore the previous permissive behavior, set {{spark.sql.variant.validateUnicodeInJsonParsing}} to {{{}false{}}}.
>  * *SPARK-56554[SQL] Respect inferSchema option when parsing XML as variant*
> *Component:* SQL
> *Why no migration-guide note needed:* Should be documented: reading XML as Variant now honors inferSchema=false (leaf/attribute text kept as strings, not inferred boolean/long/decimal), changing results. Set spark.sql.xml.variant.respectInferSchema=false to restore.
> *Proposed migration-guide message:* [SQL] Since Spark 4.2, reading XML as Variant honors {{{}inferSchema=false{}}}, keeping leaf text and attribute values as strings instead of inferring boolean/long/decimal. To restore the previous behavior of always inferring types, set {{spark.sql.xml.variant.respectInferSchema}} to {{{}false{}}}.
>  * *SPARK-54718[SQL] Preserve attributes names during CTE newInstance()*
> *Component:* SQL
> *Why no migration-guide note needed:* Should be documented: CTE newInstance() now preserves attribute name casing by default, changing output column-name casing for self-joins on a CTE with case-differing duplicate columns; restore via spark.sql.legacy.cteDuplicateAttributeNames.
> *Proposed migration-guide message:* [SQL] Since Spark 4.2, CTE relation references preserve attribute name casing when re-instantiated, so self-joins on a CTE with case-differing duplicate columns keep the original column-name casing in the output. To restore the previous behavior, set {{spark.sql.legacy.cteDuplicateAttributeNames}} to {{{}true{}}}.
>  * *SPARK-56280[SS] normalize NaN and +/-0.0 in streaming dedupe node*
> *Component:* SS
> *Why no migration-guide note needed:* Should be documented: streaming dropDuplicates on float/double keys now normalizes NaN and +/-0.0, so differently-bit NaNs and signed zeros are treated as duplicates; dedup results change with no opt-out.
> *Proposed migration-guide message:* [SS] Since Spark 4.2, streaming {{{}dropDuplicates{}}}/{{{}dropDuplicatesWithinWatermark{}}} on float or double key columns normalize NaN and signed zero, so differently-bit NaN values and {{{}+0.0{}}}/{{{}-0.0{}}} are treated as duplicates. This changes deduplication results for queries with floating-point keys; there is no opt-out.
>  * *SPARK-55058[SS] Throw error on inconsistent checkpoint metadata*
> *Component:* SS
> *Why no migration-guide note needed:* Should be documented: restarting a streaming query whose checkpoint has offset/commit logs but no metadata file now fails with MISSING_METADATA_FILE by default instead of starting a new query id; disable via the verifyMetadataExists.enabled config=false.
> *Proposed migration-guide message:* [SS] Since Spark 4.2, restarting a streaming query whose checkpoint has offset and commit logs but no metadata file fails with {{STREAMING_CHECKPOINT_MISSING_METADATA_FILE}} instead of silently generating a new query id. To restore the previous behavior, set {{spark.sql.streaming.checkpoint.verifyMetadataExists.enabled}} to {{{}false{}}}.
>  * *SPARK-56239[UI] Fix SQL tab DataTables: API default limit, date format, and appId resolution*
> *Component:* UI
> *Why no migration-guide note needed:* Should be documented: the long-existing /applications/\{appId}/sql REST endpoint default length changed from 20 to -1, so it now returns all SQL executions by default; clients relying on the 20-row default get more rows.
> *Proposed migration-guide message:* [Core] Since Spark 4.2, the {{/applications/\{appId}/sql}} REST endpoint defaults the {{length}} parameter to {{-1}} (was {{{}20{}}}), so it returns all SQL executions by default. To restore the previous behavior, pass {{length=20}} (any {{length <= 0}} returns all executions).
>  * *SPARK-55075[K8S] Track executor pod creation errors with ExecutorFailureTracker*
> *Component:* K8S
> *Why no migration-guide note needed:* Should be documented: on K8s, executor pod-creation failures are now caught, logged and counted by ExecutorFailureTracker (continue until max failures) instead of being rethrown immediately, changing default failure semantics for K8s deployments.
> *Proposed migration-guide message:* [Core] Since Spark 4.2, on Kubernetes executor pod-creation failures are caught, logged, and counted by ExecutorFailureTracker (allocation continues until the max-failures threshold) instead of being rethrown immediately. There is no opt-out.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org