[jira] [Created] (FLINK-39083) Support field-level error tolerance for CSV format deserialization

["Liu (Jira)" <ji...@apache.org>]

Liu created FLINK-39083:
---------------------------

             Summary: Support field-level error tolerance for CSV format deserialization
                 Key: FLINK-39083
                 URL: https://issues.apache.org/jira/browse/FLINK-39083
             Project: Flink
          Issue Type: Improvement
          Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
            Reporter: Liu
         Attachments: image-2026-02-13-10-08-41-570.png

h1. Motivation

Currently, the csv.ignore-parse-errors option in CSV format provides only two behaviors:
 * false (default): Any field-level parse error causes the entire job to fail.
 * true: Any field-level parse error causes the entire row to be discarded (returns null).

This "all-or-nothing" approach is problematic in production ETL scenarios. For example, consider a CSV table with 50 columns where only 1 TIMESTAMP column occasionally has malformed values. With ignore-parse-errors=true, the entire row—including the 49 correctly parsed fields—is silently dropped. This leads to significant and unnecessary data loss.
h1. Proposal

Introduce a new configuration option csv.ignore-single-field-parse-error (boolean, default false) that provides field-level error tolerance:
 * When enabled, if a single field fails type conversion (e.g., "abc" for an INT column), only that field is set to null, and the rest of the row is preserved.
 * Jackson-level parsing errors (e.g., malformed CSV structure) are not affected by this option and continue to be governed by ignore-parse-errors.

Behavior matrix:

!image-2026-02-13-10-08-41-570.png|width=532,height=149!

*Scope of Changes*
 # CsvFormatOptions — Add new ConfigOption<Boolean> for ignore-single-field-parse-error.
 # CsvCommons — Register the new option in optionalOptions() and forwardOptions().
 # CsvToRowDataConverters — Core change: add ignoreSingleFieldParseErrors flag; modify the catch block in createRowConverter() to set the failed field to null instead of re-throwing.
 # CsvRowDataDeserializationSchema — Add builder setter; pass flag through to CsvToRowDataConverters.
 # CsvFormatFactory — Wire the new config option to the deserialization schema builder.
 # CsvFileFormatFactory — Wire the new config option in the bulk decoding path.
 # Tests — Add unit tests covering all four combinations in the behavior matrix.
 # Documentation — Update English and Chinese CSV format docs.

h1. Compatibility
 * Fully backward compatible: The new option defaults to false, preserving existing behavior.
 * No changes to serialization path: This option only affects deserialization.
 * No public API changes: Only new optional configuration added.

h1. Discussion Points

I'd like to get community feedback on the following before proceeding with implementation:
 # Option naming: Is csv.ignore-single-field-parse-error clear enough? Alternatives considered: csv.field-error-as-null, csv.partial-parse-errors.
 # Interaction with ignore-parse-errors: Should ignore-single-field-parse-error=true implicitly suppress field-level errors even when ignore-parse-errors=false? Or should it only take effect when ignore-parse-errors is also true?
 # Cross-format consistency: Should we consider a similar option for the JSON format (JsonToRowDataConverters) in a follow-up JIRA? The JSON format has a similar "all-or-nothing" behavior today.
 # Logging: The proposed implementation logs a WARN for each field-level error. Should this be configurable or use a different log level (e.g., DEBUG) to avoid log flooding?

h1. Related

Depends on / follows: https://issues.apache.org/jira/browse/FLINK-39065



--
This message was sent by Atlassian Jira
(v8.20.10#820010)