You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Akshat Shenoi (Jira)" <ji...@apache.org> on 2026/06/15 21:27:00 UTC
[jira] [Updated] (SPARK-57478) [SQL] Read text files from tar archives
[ https://issues.apache.org/jira/browse/SPARK-57478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Akshat Shenoi updated SPARK-57478:
----------------------------------
Description:
SPARK-57135 / SPARK-57321 added reading and schema inference for CSV files packed in tar archives (.tar/.tar.gz/.tgz), and SPARK-57419 did the same for JSON, gated by spark.sql.files.archive.reader.enabled. This extends the same capability to the text data source.
When the flag is enabled, the V1 text source treats a tar archive as a directory of its entries: each entry is streamed through the ArchiveReader (never unpacked to disk) and read exactly like a standalone text file -- one row per line, or a single row holding the whole entry when wholeText is set. The whole archive is one non-splittable unit (isSplitable returns false for an archive path).
Text has a fixed `value STRING` schema, so there is no schema inference. Archive scanning is wired into the V1 file source only; the DSv2 reader is left untouched.
was:SPARK-57135 added support for reading CSV files packed in tar archives (.tar/.tar.gz/.tgz) and SPARK-57321 added schema inference for them, both gated by spark.sql.files.archive.reader.enabled; this extends the same capability to the JSON data source. When spark.sql.files.archive.reader.enabled is true, the V1 JSON data source reads a tar archive as if it were a directory of its entries: each entry is streamed through ArchiveReader (never unpacked to disk) and parsed exactly like a standalone JSON file, for both line-delimited and multi-line JSON. Schema inference reads every archive entry together with any loose files alongside it in a single JsonInferSchema pass, so the inferred schema matches a directory read of the same files. The whole archive is a single non-splittable unit, and a corrupt/missing archive is skipped as a unit under ignoreCorruptFiles/ignoreMissingFiles. The DSv2 JSON reader does not support archives, so it refuses to infer a schema for archive inputs (raising UNABLE_TO_INFER_SCHEMA) rather than mis-reading raw archive bytes. Unlike CSV, JSON needs no per-entry header handling (records are self-describing, so one parser serves every entry) and no mergeSchema-style branching (JsonInferSchema already merges record types by field name across all inputs, so one pass is itself the union). This change also unifies the archive test suites: the format-agnostic inference and complex-type tests are hoisted into ArchiveReadSuiteBase behind capability hooks (supportsSchemaInference, supportsComplexTypes) so CSV, JSON, and future archive formats share them rather than each duplicating them.
> [SQL] Read text files from tar archives
> ---------------------------------------
>
> Key: SPARK-57478
> URL: https://issues.apache.org/jira/browse/SPARK-57478
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Akshat Shenoi
> Assignee: Akshat Shenoi
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> SPARK-57135 / SPARK-57321 added reading and schema inference for CSV files packed in tar archives (.tar/.tar.gz/.tgz), and SPARK-57419 did the same for JSON, gated by spark.sql.files.archive.reader.enabled. This extends the same capability to the text data source.
> When the flag is enabled, the V1 text source treats a tar archive as a directory of its entries: each entry is streamed through the ArchiveReader (never unpacked to disk) and read exactly like a standalone text file -- one row per line, or a single row holding the whole entry when wholeText is set. The whole archive is one non-splittable unit (isSplitable returns false for an archive path).
> Text has a fixed `value STRING` schema, so there is no schema inference. Archive scanning is wired into the V1 file source only; the DSv2 reader is left untouched.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org