You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Akshat Shenoi (Jira)" <ji...@apache.org> on 2026/06/12 18:04:00 UTC

[jira] [Created] (SPARK-57419) [SQL] Support read & schema inference of JSON files inside tar archives

Akshat Shenoi created SPARK-57419:
-------------------------------------

             Summary: [SQL] Support read & schema inference of JSON files inside tar archives
                 Key: SPARK-57419
                 URL: https://issues.apache.org/jira/browse/SPARK-57419
             Project: Spark
          Issue Type: New Feature
          Components: SQL
    Affects Versions: 4.3.0
            Reporter: Akshat Shenoi
            Assignee: Akshat Shenoi
             Fix For: 4.3.0


SPARK-57135 added opt-in reading of CSV files packaged in tar archives (.tar, .tar.gz, .tgz), but only with an explicit schema — schema inference was out of scope, and inferring without a schema errors (UNABLE_TO_INFER_SCHEMA).

This follow-up adds schema inference for tar archives, so spark.read.csv("data.tar") (with spark.sql.files.archive.reader.enabled=true) infers a schema instead of erroring, matching how a directory of the same CSV files is inferred:

- CSVDataSource.inferSchema streams each archive's entries through the existing ArchiveReader (never unpacking to disk), tokenizes each entry like a standalone CSV file (dropping its header row when header=true), and feeds all entries' rows into a single CSVInferSchema pass keyed on the first entry's header — the same first-header, type-widening model used for a multi-file CSV read.
- When archives and loose CSV files are read together, the two inferred schemas are merged positionally with CSV-aware type widening.
- ignoreCorruptFiles / ignoreMissingFiles are honored at archive granularity, matching the loose-file path.
- Reuses the spark.sql.files.archive.reader.enabled config from SPARK-57135; no new config.

Scope: CSV over tar, building on SPARK-57135. Inference for other file formats (JSON, text, XML) follows their respective read support.

Tests: directory parity, all archive formats agree, corrupt-archive skip among good ones, cross-entry type widening, and mixed archive + loose-file inference.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org