You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Akshat Shenoi (Jira)" <ji...@apache.org> on 2026/06/26 23:28:00 UTC
[jira] [Updated] (SPARK-57591) [SQL] Read and infer ORC schema from tar archives
[ https://issues.apache.org/jira/browse/SPARK-57591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Akshat Shenoi updated SPARK-57591:
----------------------------------
Description:
Extend the ORC data source to read and infer schema from .tar/.tar.gz/.tgz archives, gated by spark.sql.files.archive.reader.enabled.
was:
Following the CSV (SPARK-57135 / SPARK-57321) and JSON (SPARK-57419) tar-archive support, extend the Avro data source to read and infer schema from .tar/.tar.gz/.tgz archives, gated by spark.sql.files.archive.reader.enabled.
When the flag is enabled, the V1 Avro source treats a tar archive as a directory of its entries. Reads stream each entry through a forward-only DataFileStream (never unpacked to disk, memory stays bounded); a fresh datum reader and deserializer are built per entry since each entry carries its own writer schema in its header. Schema inference reads each entry's writer schema from the Avro header (records are never scanned) and, like a directory read, uses the first readable schema for the whole dataset -- Avro does not merge schemas across files (schema evolution is not supported).
The shared ArchiveReader is hardened to open the first entry eagerly and release the stream on a corrupt archive, so corruption surfaces (and is cleaned up) on the driver-side, header-only inference path that has no task-completion listener.
The whole archive is one non-splittable unit. Avro has no DSv2 reader, so the archive scan is V1-only.
> [SQL] Read and infer ORC schema from tar archives
> -------------------------------------------------
>
> Key: SPARK-57591
> URL: https://issues.apache.org/jira/browse/SPARK-57591
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Akshat Shenoi
> Assignee: Akshat Shenoi
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> Extend the ORC data source to read and infer schema from .tar/.tar.gz/.tgz archives, gated by spark.sql.files.archive.reader.enabled.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org