[jira] [Created] (SPARK-60217) Add a decompressionCodec read option to text-based file sources to override extension-based codec detection

["Sandip Agarwala (Jira)" <ji...@apache.org>]

Sandip Agarwala created SPARK-60217:
---------------------------------------

             Summary: Add a decompressionCodec read option to text-based file sources to override extension-based codec detection
                 Key: SPARK-60217
                 URL: https://issues.apache.org/jira/browse/SPARK-60217
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Sandip Agarwala


Text-based file sources (CSV, JSON, XML, text) choose the decompression codec for each input file solely from its file name extension (e.g. `.gz` -> gzip). Files whose extension does not match their actual compression cannot be read correctly: a gzip-compressed file named `data.csv` is read as garbage, and a plain-text file named `data.csv.gz` fails to decompress. Writers already support a `compression` option, but readers have no way to override the extension-based detection.

This ticket proposes a new read-only data source option, `decompressionCodec`, for CSV, JSON, XML and text:

{code}
spark.read.option("decompressionCodec", "gzip").csv("/path/to/data.csv")
{code}

* Accepts the short names supported by the `compression` write option (`none`, `uncompressed`, `bzip2`, `deflate`, `gzip`, `lz4`, `snappy`), plus `zstd`/`zstandard`, or a fully qualified Hadoop `CompressionCodec` class name.
* When set, the codec is applied to every input file instead of inferring it from the extension; `none`/`uncompressed` reads files as-is even if they carry a compression extension.
* Applies uniformly to line-based, multiLine, wholetext and schema-inference reads, for both DSv1 and DSv2.
* Splittability follows the forced codec (e.g. a forced bzip2 file remains splittable, a forced gzip file does not).
* The existing write-side `compression` option keeps being ignored on read. A separate option name is used deliberately so that tables and pipelines that persist or share `compression` (e.g. `saveAsTable` / `CREATE TABLE ... OPTIONS (compression ...)`) do not change behavior.
* Not supported (fails with a clear error) for archive inputs (`.tar`, `.tar.gz`, `.tgz`, `.zip`, `.7z`), whose container format is determined by the extension, and when the legacy Hadoop `LineRecordReader` is enabled (`spark.sql.execution.datasources.hadoopLineRecordReader.enabled=false`), which cannot honor the option.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org