You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by "LuciferYang (via GitHub)" <gi...@apache.org> on 2026/03/25 04:05:14 UTC

[PR] [SPARK-56171][SQL] Enable V2 file write path for non-partitioned DataFrame API writes and delete `FallBackFileSourceV2` [spark]

LuciferYang opened a new pull request, #54998:
URL: https://github.com/apache/spark/pull/54998

   ### What changes were proposed in this pull request?
   This PR enables the V2 file write path for non-partitioned `df.write.mode("append"/"overwrite").save(path)` operations across all built-in file formats (Parquet, ORC, JSON, CSV, Text, Avro), and deletes the `FallBackFileSourceV2` analysis rule which is now redundant.
   
   ### Key changes
   
   **V2 write foundation (`FileTable`, `FileWrite`, `*Write`, `*Table`)**
   - `FileTable.createFileWriteBuilder`: new infrastructure for creating `WriteBuilder` with `SupportsTruncate` and `SupportsDynamicOverwrite` capabilities
   - `FileWrite`: partition schema support, truncation (overwrite), dynamic partition overwrite, schema validation (nested column name duplication, data type, collation in map keys)
   - All 6 format-specific `Write` case classes (`ParquetWrite`, `OrcWrite`, `JsonWrite`, `CSVWrite`, `TextWrite`, `AvroWrite`) accept new parameters from `createFileWriteBuilder`
   - All 6 format-specific `Table` classes implement `newWriteBuilder` via `createFileWriteBuilder`
   
   **Delete `FallBackFileSourceV2`**
   - Remove the analysis rule and its registrations in `BaseSessionStateBuilder` and `HiveSessionStateBuilder`
   - This rule was redundant: `USE_V1_SOURCE_LIST` (defaulting to all built-in formats) already prevents V2 file tables from being created, and `DataSourceV2Utils.getTableProvider` gates catalog table loading
   
   **Cache invalidation (`DataSourceV2Strategy`)**
   - `refreshCache`: use `recacheByPath` (instead of `recacheByPlan`) for `FileTable` writes, with `fileIndex.refresh()` to update the cached file listing
   
   **Error handling (`FileFormatDataWriter`)**
   - Override `writeAll` to wrap errors with `TASK_WRITE_FAILED`, consistent with the per-row `write()` method
   
   **V2 write gating (`DataFrameWriter`, `DataSourceV2Utils`)**
   - `DataFrameWriter.lookupV2Provider`: allow V2 for `FileDataSourceV2` only when mode is `Append` or `Overwrite` AND no `partitionBy` is specified; fall back to V1 for `ErrorIfExists`/`Ignore` (TODO: SPARK-56174) and partitioned writes (TODO: SPARK-56185)
   - `DataFrameWriter.saveAsTable` / `insertInto`: always fall back to V1 for `FileDataSourceV2` (TODO: SPARK-56185)
   - `DataSourceV2Utils.getTableProvider`: return `None` for `FileDataSourceV2` to prevent V2 catalog table loading until stats, partition management, and data type validation gaps are addressed (TODO: SPARK-56185)
   
   **Data type validation (`V2SessionCatalog`)**
   - Add V1 `FileFormat.supportDataType` validation in the `createTable` fallback branch, ensuring `CREATE TABLE` with unsupported types (e.g., Variant in CSV) is rejected consistently
   
   **Avro Table**
   - Fix `formatName` from `"AVRO"` to `"Avro"` to match V1's `AvroFileFormat.toString`
   
   
   ### Why are the changes needed?
   The V2 Data Source API provides a cleaner, more extensible write path than V1's `InsertIntoHadoopFsRelationCommand`. Enabling V2 writes for built-in file formats is a step toward fully migrating file sources to V2, which will simplify the codebase and enable future optimizations.
   
   `FallBackFileSourceV2` was an analysis rule that converted V2 file `InsertIntoStatement` back to V1. It is no longer needed because:
   1. `USE_V1_SOURCE_LIST` (default: all built-in formats) prevents V2 file tables from being created for reads or writes
   2. `DataSourceV2Utils.getTableProvider` gates V2 catalog table loading
   3. The DataFrame API path uses `AppendData`/`OverwriteByExpression`, not `InsertIntoStatement`
   
   
   ### Does this PR introduce _any_ user-facing change?
   No. With default configuration (`spark.sql.sources.useV1SourceList` = `"avro,csv,json,kafka,orc,parquet,text"`), all file writes continue to use the V1 path. The V2 write path is only activated when a user explicitly clears `USE_V1_SOURCE_LIST` and uses `df.write.mode("append"/"overwrite").save(path)` without `partitionBy`.
   
   
   ### How was this patch tested?
   - Pass Github Actions
   - New test suite `FileDataSourceV2WriteSuite` (23 tests) covering:
     - V1 fallback behavior for all save modes
     - V2 non-partitioned append and overwrite across parquet, orc, json, csv
     - V2 cache invalidation on append and overwrite
     - Partitioned writes, dynamic partition overwrite, multi-level partitioning (via V1)
     - Catalog table INSERT INTO and CTAS (via V1)
   
   
   ### Was this patch authored or co-authored using generative AI tooling?
   Generated-by:  Claude Code 4.6
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org