You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "akshatshenoi-eng (via GitHub)" <gi...@apache.org> on 2026/05/28 21:29:58 UTC
[PR] [SPARK-57135][SQL] Add ArchiveFormat for reading .tar/.tar.gz/.tgz archives as files [spark]
akshatshenoi-eng opened a new pull request, #56193:
URL: https://github.com/apache/spark/pull/56193
### What changes were proposed in this pull request?
Adds `ArchiveFormat`, a format-agnostic archive layer that lets any V1 `FileFormat` transparently read tar archives (`.tar`, `.tar.gz`, `.tgz`) without per-format changes.
**New file `ArchiveFormat.scala`:**
- `readArchive` materializes one entry at a time to a local temp file and invokes the caller-supplied `readFn` against a synthetic `PartitionedFile` pointing at it. Only one entry's bytes live on disk at a time per task; the temp dir is cleaned up on iterator close and on task completion.
- `expandArchives` does the same materialization on the driver for `inferSchema`, substituting entry `FileStatus`es for the archive's.
- `isArchivePath` matches `.tar`, `.tar.gz`, and `.tgz`. `.tar.gz` is auto-decompressed by Hadoop's `CompressionCodecFactory` via `CodecStreams`; `.tgz` is not a registered Hadoop codec extension, so the gzip layer is unwrapped explicitly with `GZIPInputStream`.
- Entries whose basename starts with `.` are skipped (covers macOS AppleDouble sidecars, `.DS_Store`, etc.).
Materializing to disk (rather than streaming) means formats that need random access (Parquet/ORC footers) work without modification, at the cost of one entry's worth of disk per task.
**Pipeline hooks:**
- `PartitionedFileUtil.splitFiles`: archive paths are forced to a single split so the archive layer can stream entries sequentially.
- `FileScanRDD.readCurrentFile`: archive paths are routed through `ArchiveFormat.readArchive` instead of calling the format reader directly.
- `DataSource.resolve`: both `inferSchema` call sites expand archives before delegating to the format.
The feature is gated behind `spark.sql.files.archive.enabled` (default `false`).
### Why are the changes needed?
A common ingestion pattern stores many small files inside tar archives to reduce namespace pressure (fewer files in object stores or HDFS). Today there is no way to read these without first unpacking them externally. This change lets users point any V1 datasource reader directly at a tar archive and have the entries read transparently.
### Does this PR introduce _any_ user-facing change?
Yes — when `spark.sql.files.archive.enabled=true`, `.tar`, `.tar.gz`, and `.tgz` paths are transparently expanded during both schema inference and scan. The config defaults to `false` so existing behaviour is unchanged.
### How was this patch tested?
Two new test suites:
- `ArchiveFormatSuite`: unit tests for `isArchivePath`, `readArchive` (empty, single, multi, gzip, lazy materialization, sidecar skipping, temp-file lifecycle, partition value propagation, TaskContext cleanup), and `expandArchives`.
- `ArchiveReadSuite`: end-to-end reads through CSV, JSON, Parquet, and ORC readers against `.tar`, `.tar.gz`, and `.tgz` inputs, plus schema inference parity, splittability, and mixed-archive/non-archive partition layout tests.
### Was this patch authored or co-authored by anyone who is not a member of the [Spark community](https://spark.apache.org/committers.html)?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57135][SQL] Add ArchiveFormat for reading .tar/.tar.gz/.tgz archives as files [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #56193:
URL: https://github.com/apache/spark/pull/56193#issuecomment-4568690743
Con't we already support compression codec in CSV, JSON and text? I think we should rather add an option there instead of introducing a new datasource
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57135][SQL] Support reading CSV files inside tar archives [spark]
Posted by "akshatshenoi-db (via GitHub)" <gi...@apache.org>.
akshatshenoi-db commented on code in PR #56193:
URL: https://github.com/apache/spark/pull/56193#discussion_r3364731287
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala:
##########
@@ -44,6 +44,11 @@ case class CSVFileFormat() extends TextBasedFileFormat with DataSourceRegister {
options: Map[String, String],
path: Path): Boolean = {
val parsedOptions = getCsvOptions(sparkSession, options)
+ // A tar archive is decompressed/unpacked as a sequential stream, so it must be read as a
+ // single split rather than carved into byte ranges.
+ if (parsedOptions.archiveFormatEnabled && ArchiveReader.isArchivePath(path)) {
Review Comment:
deferred to when json and text are implemented
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57135][SQL] Support reading CSV files inside tar archives [spark]
Posted by "akshatshenoi-db (via GitHub)" <gi...@apache.org>.
akshatshenoi-db commented on code in PR #56193:
URL: https://github.com/apache/spark/pull/56193#discussion_r3364592096
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala:
##########
@@ -119,24 +124,26 @@ case class CSVFileFormat() extends TextBasedFileFormat with DataSourceRegister {
dataSchema.filterNot(_.name == parsedOptions.columnNameOfCorruptRecord))
val actualRequiredSchema = StructType(
requiredSchema.filterNot(_.name == parsedOptions.columnNameOfCorruptRecord))
- val parser = new UnivocityParser(
- actualDataSchema,
- actualRequiredSchema,
- parsedOptions,
- actualFilters)
// Use column pruning when specified by Catalyst, except when one or more columns have
// existence default value(s), since in that case we instruct the CSV parser to disable column
// pruning and instead read each entire row in order to correctly assign the default value(s).
val schema = if (isColumnPruningEnabled) actualRequiredSchema else actualDataSchema
- val isStartOfFile = file.start == 0
- val headerChecker = new CSVHeaderChecker(
- schema, parsedOptions, source = s"CSV file: ${file.urlEncodedPath}", isStartOfFile)
- CSVDataSource(parsedOptions).readFile(
- conf,
- file,
- parser,
- headerChecker,
- requiredSchema)
+
+ def newParser(): UnivocityParser =
+ new UnivocityParser(actualDataSchema, actualRequiredSchema, parsedOptions, actualFilters)
+ def getHeaderChecker(isStartOfFile: Boolean, source: String): CSVHeaderChecker =
+ new CSVHeaderChecker(schema, parsedOptions, source, isStartOfFile)
+
+ // A tar archive (always a single split, see `isSplitable`) is streamed entry by entry when
+ // archive reads are enabled; otherwise the file is parsed directly.
+ if (parsedOptions.archiveFormatEnabled && ArchiveReader.isArchivePath(file.toPath)) {
+ CSVDataSource(parsedOptions).readArchive(
Review Comment:
will implement schema inference in a follow up pr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57135][SQL] Support reading CSV files inside tar archives [spark]
Posted by "akshatshenoi-eng (via GitHub)" <gi...@apache.org>.
akshatshenoi-eng commented on PR #56193:
URL: https://github.com/apache/spark/pull/56193#issuecomment-4597256255
> @akshatshenoi-eng I think we should support this all in Text and JSON as well with sharing the same codebase. Also do you mind explaining how it's going to work? e.g., if partitioned table is all tar-gzed, would Spark recognize the structure? Or would you read all them in single dataframe?
>
> In addition, how do we handle the physical partitions? Would we distribute them quite well?
Text and JSON support are both planned. I made ArchiveReader and streamArchiveEntries format agnostic so adding that support should be straightforward since both have stream-based parsers (same as CSV). Parquet, ORC, Avro, XML, and Excel are also planned I'm still figuring Parquet out since it can't be streamed like CSV is. I just wanted to start with CSV to validate the streaming design end-to-end before scaling to other formats.
Spark recognizes the partition structure correctly. Partition discovery happens at the directory level, independent of file format. If the layout is:
s3://bucket/dt=2024-01-01/data.tar.gz
s3://bucket/dt=2024-01-02/data.tar.gz
each archive becomes a PartitionedFile with its partition values already attached (dt=2024-01-01, etc.). When the archive is streamed, every row produced from its entries inherits those partition values automatically.
Each archive is a single Spark partition because tar is a sequential stream (isSplitable returns false, so Spark can't carve it into byte-range splits). The distribution across executors scales with the number of archive files: 10 archives → 10 tasks, which distribute across the cluster normally. The current limitation is that a single large archive isn't parallelized but that is also on the roadmap to be handled later.
Sorry for anything else that may be vague or not yet implemented my intern project is enabling multi-file archive read support for tar, tar.gz, zip and 7z.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57135][SQL] Support reading CSV files inside tar archives [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #56193: [SPARK-57135][SQL] Support reading CSV files inside tar archives
URL: https://github.com/apache/spark/pull/56193
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57135][SQL] Add ArchiveFormat for reading .tar/.tar.gz/.tgz archives as files [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #56193:
URL: https://github.com/apache/spark/pull/56193#issuecomment-4570774674
in addition to gzip tarball, can it be extended to support other codec? at least I think zstd should be supported, a similar request was raised in the Hadoop dev list recently
https://lists.apache.org/thread/ntlx40h3vn6k7q3y5qf22vm815nw8lkz
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57135][SQL] Support reading CSV files inside tar archives [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #56193:
URL: https://github.com/apache/spark/pull/56193#issuecomment-4597055602
@akshatshenoi-eng I think we should support this all in Text and JSON as well with sharing the same codebase. Also do you mind explaining how it's going to work? e.g., if partitioned table is all tar-gzed, would Spark recognize the structure? Or would you read all them in single dataframe?
In addition, how do we handle the physical partitions? Would we distribute them quite well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57135][SQL] Support reading CSV files inside tar archives [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #56193:
URL: https://github.com/apache/spark/pull/56193#discussion_r3360232824
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala:
##########
@@ -44,6 +44,11 @@ case class CSVFileFormat() extends TextBasedFileFormat with DataSourceRegister {
options: Map[String, String],
path: Path): Boolean = {
val parsedOptions = getCsvOptions(sparkSession, options)
+ // A tar archive is decompressed/unpacked as a sequential stream, so it must be read as a
+ // single split rather than carved into byte ranges.
+ if (parsedOptions.archiveFormatEnabled && ArchiveReader.isArchivePath(path)) {
Review Comment:
On reusability for later formats (per the offline discussion — CSV-only now is fine): the reusable core is in good shape. `ArchiveReader`/`TarArchiveReader`/`lineIterator` live in `execution.datasources` and are format-agnostic, so a future `JsonDataSource.readArchive` can call `ArchiveReader(path).readEntries(conf) { (name, in) => ... }` directly, and `archiveFormatEnabled` already sits on the shared `FileSourceOptions`.
The one piece that isn't general is this per-`FileFormat` gating — the `isSplitable` archive guard here plus the `buildReader` archive-vs-normal branch. `JsonFileFormat`/`TextFileFormat` have structurally identical `isSplitable`/`buildReader` today, so they'd copy these verbatim. Optionally lift the gating into a shared helper/trait on `TextBasedFileFormat` (e.g. `ArchiveReadableFileFormat`) so a new format supplies only its per-entry parser. Minor; the bigger reuse gap is schema inference (see the comment at the archive branch in `buildReader`).
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala:
##########
@@ -119,24 +124,26 @@ case class CSVFileFormat() extends TextBasedFileFormat with DataSourceRegister {
dataSchema.filterNot(_.name == parsedOptions.columnNameOfCorruptRecord))
val actualRequiredSchema = StructType(
requiredSchema.filterNot(_.name == parsedOptions.columnNameOfCorruptRecord))
- val parser = new UnivocityParser(
- actualDataSchema,
- actualRequiredSchema,
- parsedOptions,
- actualFilters)
// Use column pruning when specified by Catalyst, except when one or more columns have
// existence default value(s), since in that case we instruct the CSV parser to disable column
// pruning and instead read each entire row in order to correctly assign the default value(s).
val schema = if (isColumnPruningEnabled) actualRequiredSchema else actualDataSchema
- val isStartOfFile = file.start == 0
- val headerChecker = new CSVHeaderChecker(
- schema, parsedOptions, source = s"CSV file: ${file.urlEncodedPath}", isStartOfFile)
- CSVDataSource(parsedOptions).readFile(
- conf,
- file,
- parser,
- headerChecker,
- requiredSchema)
+
+ def newParser(): UnivocityParser =
+ new UnivocityParser(actualDataSchema, actualRequiredSchema, parsedOptions, actualFilters)
+ def getHeaderChecker(isStartOfFile: Boolean, source: String): CSVHeaderChecker =
+ new CSVHeaderChecker(schema, parsedOptions, source, isStartOfFile)
+
+ // A tar archive (always a single split, see `isSplitable`) is streamed entry by entry when
+ // archive reads are enabled; otherwise the file is parsed directly.
+ if (parsedOptions.archiveFormatEnabled && ArchiveReader.isArchivePath(file.toPath)) {
+ CSVDataSource(parsedOptions).readArchive(
Review Comment:
**Schema inference is not archive-aware — the key framework-completeness gap.** `inferSchema` → `CSVDataSource.inferSchema` → `infer` reads raw file bytes (`TextFileFormat` for non-multiline, `BinaryFileRDD` for multiline); none go through `ArchiveReader`. So with the config enabled and no user-supplied schema (`spark.read.csv("data.tar")`), inference reads tar headers/bytes as CSV and yields a garbage schema rather than erroring. Every test passes `.schema(...)` explicitly, so this path is untested, and the description doesn't state a schema is required.
This matters most for reuse: `JsonDataSource`/`TextFileFormat` infer through the *same* `createBaseDataset`/`BinaryFileRDD` raw-bytes path, so leaving it unsolved means every future format re-hits (or re-fixes) the same bug. Worth solving once in the reusable layer — a shared "infer from the first archive entry via `ArchiveReader`" path, or at minimum a shared clear "schema required for archive reads" error — plus a test and a description note.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala:
##########
@@ -144,6 +184,39 @@ object TextInputCSVDataSource extends CSVDataSource {
UnivocityParser.parseIterator(lines, parser, headerChecker, requiredSchema)
}
+ override def readArchive(
+ conf: Configuration,
+ file: PartitionedFile,
+ getParser: () => UnivocityParser,
+ getHeaderChecker: (Boolean, String) => CSVHeaderChecker,
+ requiredSchema: StructType): Iterator[InternalRow] =
+ // Stream each tar entry through the line-based parser, treating the entry exactly like a
+ // standalone CSV file (a fresh parser/header checker is built per entry).
+ streamArchiveEntries(conf, file, getParser, getHeaderChecker) { (parser, headerChecker, in) =>
+ UnivocityParser.parseIterator(
+ entryLines(in, parser.options), parser, headerChecker, requiredSchema)
+ }
+
+ /**
+ * Decodes one archive entry's bytes into the same CSV line strings the non-archive [[readFile]]
+ * path feeds to the parser: [[ArchiveReader.lineIterator]] splits the entry into lines (honoring
+ * a custom line separator), each line is decoded with the configured charset, and the separator
+ * is re-appended so `UnivocityParser` does not raise EOF on the final line.
+ *
+ * @param in bytes of one already-decompressed archive entry; not closed here (the archive owns
+ * the underlying stream).
+ * @param options CSV options supplying the read line separator and charset.
+ * @return an iterator over the entry's lines, each terminated with the line separator.
+ */
+ private def entryLines(in: InputStream, options: CSVOptions): Iterator[String] = {
+ val newline = options.lineSeparatorInRead.getOrElse(
+ Array(options.asParserSettings.getFormat.getNormalizedNewline.toByte))
+ ArchiveReader.lineIterator(in, options.lineSeparatorInRead).map { line =>
+ line.append(newline, 0, newline.length)
Review Comment:
Nit: the comment says the trailing newline is appended "so `UnivocityParser` does not raise EOF on the final line," but the non-archive `readFile` path feeds lines to the *same* `parseIterator` (via `HadoopFileLinesReader`) **without** re-appending a terminator. So either the EOF claim is inaccurate (and this append is unnecessary), or there's a subtlety worth spelling out. The single-byte `getNormalizedNewline` append also diverges from `readFile` for non-UTF-8 multi-byte charsets. Could you double-check — if the append isn't needed, dropping it keeps the archive path identical to `readFile`.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala:
##########
@@ -83,6 +83,46 @@ abstract class CSVDataSource extends Serializable {
sparkSession: SparkSession,
inputPaths: Seq[FileStatus],
parsedOptions: CSVOptions): StructType
+
+ /**
+ * Streams a tar archive (`.tar`/`.tar.gz`/`.tgz`) entry by entry through the CSV parser without
+ * unpacking it to disk. The whole archive is a single split (see `CSVFileFormat.isSplitable`); a
+ * fresh header checker and parser are built per entry so each entry is parsed exactly like a
+ * standalone CSV file -- its header, if any, validated and dropped independently. The
+ * mode-specific implementation turns one entry into rows via `parseStream` / `parseIterator`.
+ *
+ * @param getParser builds a fresh [[UnivocityParser]].
+ * @param getHeaderChecker builds a fresh [[CSVHeaderChecker]] for `(isStartOfFile, source)`.
+ */
+ def readArchive(
+ conf: Configuration,
+ file: PartitionedFile,
+ getParser: () => UnivocityParser,
+ getHeaderChecker: (Boolean, String) => CSVHeaderChecker,
+ requiredSchema: StructType): Iterator[InternalRow]
+
+ /**
+ * Shared driver used by the [[readArchive]] implementations: streams each non-skipped entry's
+ * `(parser, headerChecker, stream)` -- a fresh parser/header checker per entry -- through
+ * `parseEntry`. The header checker `source` (`CSV archive entry: <archive>!/<entryName>`) names
+ * the entry in error messages.
+ */
+ protected def streamArchiveEntries(
+ conf: Configuration,
+ file: PartitionedFile,
+ getParser: () => UnivocityParser,
+ getHeaderChecker: (Boolean, String) => CSVHeaderChecker)(
+ parseEntry: (UnivocityParser, CSVHeaderChecker, InputStream) => Iterator[InternalRow])
+ : Iterator[InternalRow] = {
+ ArchiveReader(file.toPath).readEntries(conf) { (entryName, in) =>
Review Comment:
`ignoreCorruptFiles` ends up archive-granular here. An archive is a single non-splittable `PartitionedFile`, so `FileScanRDD`'s per-file corrupt handling skips the **whole** archive on any throw (corrupt gzip/tar, an IO error mid-entry, or a FAILFAST malformed record), whereas a directory of loose files skips only the bad file and keeps the rest. The PR claims directory parity, but the corrupt test only covers a whole-corrupt archive — not a bad entry among good ones. Worth a test that asserts the actual behavior and a one-line note that corrupt handling is archive-granular.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ArchiveReader.scala:
##########
@@ -0,0 +1,235 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources
+
+import java.io.{Closeable, InputStream}
+import java.util.Locale
+import java.util.zip.GZIPInputStream
+
+import scala.util.control.NonFatal
+
+import org.apache.commons.compress.archivers.tar.{TarArchiveEntry, TarArchiveInputStream}
+import org.apache.commons.io.ByteOrderMark
+import org.apache.commons.io.input.{BOMInputStream, CloseShieldInputStream}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.hadoop.io.Text
+import org.apache.hadoop.util.LineReader
+
+import org.apache.spark.TaskContext
+
+/**
+ * Streaming reader for a single archive file. The archive is opened once and decompressed/unpacked
+ * as a stream -- entries are never materialized to local disk. [[readEntries]] hands each entry's
+ * bytes to a caller-supplied parse function as a bounded [[InputStream]] and concatenates the
+ * per-entry results into a single iterator, advancing to the next entry only once the current one
+ * is fully consumed. At most one entry is in flight at a time, so memory stays bounded regardless
+ * of archive size.
+ *
+ * This is format-agnostic: a data source whose per-file reader can consume an `InputStream` wires
+ * up archive support by calling [[readEntries]] from its read/inference paths and supplying a
+ * `parseEntry` that turns one entry stream into rows (or tokens). Formats that need random access
+ * within a file (e.g. Parquet/ORC footers) cannot use this streaming path.
+ *
+ * A concrete subclass implements [[readEntries]] for a specific archive format. Obtain the reader
+ * for a path via `ArchiveReader(path)`, which selects the implementation by file extension; new
+ * archive formats are added by writing another subclass rather than modifying existing ones.
+ */
+abstract class ArchiveReader(path: Path) {
+
+ /**
+ * Streams the archive entry by entry, applying `parseEntry` to each non-skipped entry's
+ * `(name, stream)` and concatenating the results into a single iterator. The next entry is opened
+ * only once the current entry's iterator is exhausted, so nothing is buffered to disk and at most
+ * one entry's bytes are read at a time. The archive stream is closed when the returned iterator
+ * is exhausted, when [[Closeable.close]] is called on it, and (defensively) on task completion.
+ */
+ def readEntries[T](
+ conf: Configuration)(
+ parseEntry: (String, InputStream) => Iterator[T]): Iterator[T]
+}
+
+object ArchiveReader {
+
+ /**
+ * Whether `path` names an archive this reader can stream. Dispatched purely on the file
+ * extension -- `.tar`, `.tar.gz`, or `.tgz` -- since the bytes are not inspected here.
+ */
+ def isArchivePath(path: Path): Boolean = {
+ val name = path.getName.toLowerCase(Locale.ROOT)
+ name.endsWith(".tar") || name.endsWith(".tar.gz") || name.endsWith(".tgz")
+ }
+
+ /**
+ * Returns the [[ArchiveReader]] implementation for `path`, selected by its file extension. Only
+ * paths for which [[isArchivePath]] is true are supported; new archive formats add a case here.
+ */
+ def apply(path: Path): ArchiveReader = new TarArchiveReader(path)
+
+ /**
+ * Splits one already-decompressed archive entry's bytes into lines. The reusable, format-agnostic
+ * line source for archive entries; the entry stream is not closed here (the reader owns the
+ * underlying stream).
+ *
+ * @param in bytes of one archive entry.
+ * @param lineSeparatorInRead the explicit read line separator, or `None` to detect line breaks.
+ * @return an iterator over the entry's lines as [[Text]], without the trailing separator.
+ */
+ def lineIterator(in: InputStream, lineSeparatorInRead: Option[Array[Byte]]): Iterator[Text] = {
+ // A leading byte-order mark is stripped (LineReader does not strip it on its own) so the lines
+ // match the non-archive read path.
+ val bomInputStream = BOMInputStream.builder()
+ .setInputStream(in)
+ .setByteOrderMarks(
+ ByteOrderMark.UTF_8,
+ ByteOrderMark.UTF_16LE,
+ ByteOrderMark.UTF_16BE,
+ ByteOrderMark.UTF_32LE,
+ ByteOrderMark.UTF_32BE)
+ .setInclude(false)
+ .get()
+ val reader = lineSeparatorInRead match {
+ case Some(sep) => new LineReader(bomInputStream, sep)
+ case _ => new LineReader(bomInputStream)
+ }
+ new Iterator[Text] {
+ private val text = new Text()
+ private var finished = false
+ private var hasValue = false
+
+ override def hasNext: Boolean = {
+ if (!finished && !hasValue) {
+ finished = reader.readLine(text) == 0
+ hasValue = !finished
+ }
+ !finished
+ }
+
+ override def next(): Text = {
+ if (!hasNext) throw new NoSuchElementException
+ hasValue = false
+ text
+ }
+ }
+ }
+}
+
+/**
+ * [[ArchiveReader]] for tar archives: plain `.tar`, gzipped `.tar.gz`, and `.tgz`.
+ *
+ * Gzip handling: Hadoop's `CompressionCodecFactory` matches the trailing `.gz` extension and
+ * auto-decompresses `.tar.gz` via `CodecStreams`, so we just wrap that stream in
+ * `TarArchiveInputStream`. `.tgz` is not a registered Hadoop codec extension, so the gzip layer is
+ * unwrapped explicitly here.
+ */
+class TarArchiveReader(path: Path) extends ArchiveReader(path) {
+
+ // Paths Hadoop's codec factory won't auto-decompress: we apply the gzip layer here.
+ private def needsExplicitGunzip: Boolean =
+ path.getName.toLowerCase(Locale.ROOT).endsWith(".tgz")
+
+ /**
+ * Whether an entry is not a real data file and must be skipped: a directory, or an OS sidecar
+ * dotfile (basename starting with `.`, e.g. macOS `._x` resource forks or `.DS_Store`).
+ */
+ private def shouldSkipEntry(entry: TarArchiveEntry): Boolean = {
+ if (entry.isDirectory) return true
+ val name = entry.getName
+ val basename = name.substring(name.lastIndexOf('/') + 1)
+ basename.startsWith(".")
Review Comment:
`shouldSkipEntry` skips only `.`-prefixed entries, but Spark's loose-file listing filters both `.`- and `_`-prefixed names via `HadoopFSUtils.shouldFilterOutPathName` (`InMemoryFileIndex`). So an entry named `_SUCCESS` / `_committed_*` inside an archive is read as data, breaking the "parse like a directory of the same files" parity the suite asserts. Mirror the `_` filter (ideally reuse `shouldFilterOutPathName`). Good news: this lives in the shared `TarArchiveReader`, so the fix benefits every future format for free.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57135][SQL] Support reading CSV files inside tar archives [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #56193:
URL: https://github.com/apache/spark/pull/56193#discussion_r3375345452
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/ArchiveReaderSuite.scala:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources
+
+import java.io.{ByteArrayOutputStream, Closeable, File, FileOutputStream, InputStream, OutputStream}
+import java.nio.charset.StandardCharsets
+import java.util.Properties
+import java.util.zip.GZIPOutputStream
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.commons.compress.archivers.tar.{TarArchiveEntry, TarArchiveOutputStream}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.{SparkFunSuite, TaskContext, TaskContextImpl}
+
+/**
+ * Unit tests for the streaming [[ArchiveReader]] core: `isArchivePath` dispatch and `readEntries`
+ * (entry ordering, gzip handling, dir/dotfile skipping, lazy advance, the non-closing entry
+ * stream, and cleanup). Nothing here touches local disk -- entries are consumed as streams.
+ */
+class ArchiveReaderSuite extends SparkFunSuite {
+
+ private case class Entry(name: String, data: Array[Byte], isDir: Boolean = false)
+
+ private def writeTar(file: File, entries: Seq[Entry]): Unit =
+ writeTarTo(new FileOutputStream(file), entries)
+
+ /** Write a gzipped tar, used to verify the `.tar.gz` / `.tgz` archive paths. */
+ private def writeTarGz(file: File, entries: Seq[Entry]): Unit =
+ writeTarTo(new GZIPOutputStream(new FileOutputStream(file)), entries)
+
+ private def writeTarTo(rawOut: OutputStream, entries: Seq[Entry]): Unit = {
+ val out = new TarArchiveOutputStream(rawOut)
+ try {
+ entries.foreach { e =>
+ // TarArchiveEntry treats a trailing slash in the name as a directory marker.
+ val rawName = if (e.isDir && !e.name.endsWith("/")) e.name + "/" else e.name
+ val tarEntry = new TarArchiveEntry(rawName)
+ if (!e.isDir) tarEntry.setSize(e.data.length.toLong)
+ out.putArchiveEntry(tarEntry)
+ if (!e.isDir) out.write(e.data)
+ out.closeArchiveEntry()
+ }
+ out.finish()
+ } finally out.close()
+ }
+
+ private def textEntry(name: String, body: String): Entry =
+ Entry(name, body.getBytes(StandardCharsets.UTF_8))
+
+ private def readAll(in: InputStream): Array[Byte] = {
+ val out = new ByteArrayOutputStream()
+ val buf = new Array[Byte](4096)
+ var n = in.read(buf)
+ while (n >= 0) {
+ out.write(buf, 0, n)
+ n = in.read(buf)
+ }
+ out.toByteArray
+ }
+
+ /** Drains every entry into `(name, decodedText)` pairs through `ArchiveReader.readEntries`. */
+ private def collect(file: File): Seq[(String, String)] =
+ ArchiveReader(new Path(file.toURI)).readEntries(new Configuration()) { (name, in) =>
+ Iterator.single((name, new String(readAll(in), StandardCharsets.UTF_8)))
+ }.toList
+
+ // ----- isArchivePath ------------------------------------------------------
+
+ test("isArchivePath: positive cases") {
+ Seq(
+ "foo.tar", "FOO.TAR", "/a/b/c/x.tar", "weird.TaR",
+ "foo.tar.gz", "FOO.TAR.GZ", "mixed.Tar.Gz", "/a/b/c/x.tar.gz",
+ "foo.tgz", "FOO.TGZ", "/a/b/c/x.tgz"
+ ).foreach { p =>
+ assert(ArchiveReader.isArchivePath(new Path(p)), s"expected archive match for $p")
+ }
+ }
+
+ test("isArchivePath: negative cases") {
+ Seq("foo.csv", "foo.gz", "foo", "dir/", "foo.tarball", "data.zip",
+ "foo.tar.bz2", "foo.targz").foreach { p =>
+ assert(!ArchiveReader.isArchivePath(new Path(p)), s"expected non-match for $p")
+ }
+ }
+
+ // ----- readEntries --------------------------------------------------------
+
+ test("readEntries: empty tar yields empty iterator") {
+ withTempDir { dir =>
+ val tar = new File(dir, "empty.tar")
+ writeTar(tar, Seq.empty)
+ assert(collect(tar).isEmpty)
+ }
+ }
+
+ test("readEntries: single entry exposes its name and bytes") {
+ withTempDir { dir =>
+ val tar = new File(dir, "single.tar")
+ writeTar(tar, Seq(textEntry("only.csv", "hello\n")))
+ assert(collect(tar) == Seq("only.csv" -> "hello\n"))
+ }
+ }
+
+ test("readEntries: multiple entries chained in tar order") {
+ withTempDir { dir =>
+ val tar = new File(dir, "multi.tar")
+ writeTar(tar, Seq(textEntry("a.csv", "a"), textEntry("b.csv", "b"), textEntry("c.csv", "c")))
+ assert(collect(tar) == Seq("a.csv" -> "a", "b.csv" -> "b", "c.csv" -> "c"))
+ }
+ }
+
+ test("readEntries: gzipped tar (.tar.gz) via Hadoop codec factory") {
+ withTempDir { dir =>
+ val tarGz = new File(dir, "data.tar.gz")
+ writeTarGz(tarGz, Seq(textEntry("a.csv", "a"), textEntry("b.csv", "b")))
+ assert(collect(tarGz) == Seq("a.csv" -> "a", "b.csv" -> "b"))
+ }
+ }
+
+ test("readEntries: gzipped tar (.tgz) via explicit GZIPInputStream wrap") {
+ withTempDir { dir =>
+ val tgz = new File(dir, "data.tgz")
+ writeTarGz(tgz, Seq(textEntry("a.csv", "a"), textEntry("b.csv", "b")))
+ assert(collect(tgz) == Seq("a.csv" -> "a", "b.csv" -> "b"))
+ }
+ }
+
+ test("readEntries: directory entries are skipped") {
+ withTempDir { dir =>
+ val tar = new File(dir, "dirs.tar")
+ writeTar(tar, Seq(
+ Entry("subdir", Array.emptyByteArray, isDir = true),
+ textEntry("subdir/data.csv", "x")))
+ assert(collect(tar) == Seq("subdir/data.csv" -> "x"))
+ }
+ }
+
+ test("readEntries: dotfile entries (e.g. macOS ._foo) are skipped") {
+ withTempDir { dir =>
+ val tar = new File(dir, "dots.tar")
+ writeTar(tar, Seq(
+ textEntry("._real.csv", "junk"), // macOS AppleDouble sidecar
+ textEntry(".hidden", "ignored"), // bare dotfile
+ textEntry("real.csv", "kept"),
Review Comment:
The skip fix from the last round (`basename.startsWith(".")` -> `HadoopFSUtils.shouldFilterOutPathName(basename)`) added skipping of `_`-prefixed markers (`_SUCCESS`, `_committed_*`), but this test only exercises `.`-prefixed names, and no other archive test uses a `_`-prefixed entry. A regression back to `startsWith(".")` would pass the whole suite. Adding a `_SUCCESS` entry locks the new behavior in (the assertion stays `Seq("real.csv" -> "kept")` since `_SUCCESS` is skipped):
```suggestion
textEntry("_SUCCESS", "marker"), // _-prefixed marker (now also skipped)
textEntry("real.csv", "kept"),
```
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ArchiveReader.scala:
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources
+
+import java.io.{Closeable, InputStream}
+import java.util.Locale
+import java.util.zip.GZIPInputStream
+
+import scala.util.control.NonFatal
+
+import org.apache.commons.compress.archivers.tar.{TarArchiveEntry, TarArchiveInputStream}
+import org.apache.commons.io.ByteOrderMark
+import org.apache.commons.io.input.{BOMInputStream, CloseShieldInputStream}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.hadoop.io.Text
+import org.apache.hadoop.util.LineReader
+
+import org.apache.spark.TaskContext
+import org.apache.spark.util.HadoopFSUtils
+
+/**
+ * Streaming reader for a single archive file. The archive is opened once and decompressed/unpacked
+ * as a stream -- entries are never materialized to local disk. [[readEntries]] hands each entry's
+ * bytes to a caller-supplied parse function as a bounded [[InputStream]] and concatenates the
+ * per-entry results into a single iterator, advancing to the next entry only once the current one
+ * is fully consumed. At most one entry is in flight at a time, so memory stays bounded regardless
+ * of archive size.
+ *
+ * This is format-agnostic: a data source whose per-file reader can consume an `InputStream` wires
+ * up archive support by calling [[readEntries]] from its read/inference paths and supplying a
+ * `parseEntry` that turns one entry stream into rows (or tokens). Formats that need random access
+ * within a file (e.g. Parquet/ORC footers) cannot use this streaming path.
+ *
+ * A concrete subclass implements [[readEntries]] for a specific archive format. Obtain the reader
+ * for a path via `ArchiveReader(path)`, which selects the implementation by file extension; new
+ * archive formats are added by writing another subclass rather than modifying existing ones.
+ */
+abstract class ArchiveReader(path: Path) {
+
+ /**
+ * Streams the archive entry by entry, applying `parseEntry` to each non-skipped entry's
+ * `(name, stream)` and concatenating the results into a single iterator. The next entry is opened
+ * only once the current entry's iterator is exhausted, so nothing is buffered to disk and at most
+ * one entry's bytes are read at a time. The archive stream is closed when the returned iterator
+ * is exhausted, when [[Closeable.close]] is called on it, and (defensively) on task completion.
+ */
+ def readEntries[T](
+ conf: Configuration)(
+ parseEntry: (String, InputStream) => Iterator[T]): Iterator[T]
+}
+
+object ArchiveReader {
+
+ /**
+ * Whether `path` names an archive this reader can stream. Dispatched purely on the file
+ * extension -- `.tar`, `.tar.gz`, or `.tgz` -- since the bytes are not inspected here.
+ */
+ def isArchivePath(path: Path): Boolean = {
+ val name = path.getName.toLowerCase(Locale.ROOT)
+ name.endsWith(".tar") || name.endsWith(".tar.gz") || name.endsWith(".tgz")
+ }
+
+ /**
+ * Returns the [[ArchiveReader]] implementation for `path`, selected by its file extension. Only
+ * paths for which [[isArchivePath]] is true are supported; new archive formats add a case here.
+ */
+ def apply(path: Path): ArchiveReader = new TarArchiveReader(path)
+
+ /**
+ * Splits one already-decompressed archive entry's bytes into lines. The reusable, format-agnostic
+ * line source for archive entries; the entry stream is not closed here (the reader owns the
+ * underlying stream).
+ *
+ * @param in bytes of one archive entry.
+ * @param lineSeparatorInRead the explicit read line separator, or `None` to detect line breaks.
+ * @return an iterator over the entry's lines as [[Text]], without the trailing separator.
+ */
+ def lineIterator(in: InputStream, lineSeparatorInRead: Option[Array[Byte]]): Iterator[Text] = {
+ // A leading byte-order mark is stripped (LineReader does not strip it on its own) so the lines
+ // match the non-archive read path.
+ val bomInputStream = BOMInputStream.builder()
+ .setInputStream(in)
+ .setByteOrderMarks(
+ ByteOrderMark.UTF_8,
+ ByteOrderMark.UTF_16LE,
+ ByteOrderMark.UTF_16BE,
+ ByteOrderMark.UTF_32LE,
+ ByteOrderMark.UTF_32BE)
+ .setInclude(false)
+ .get()
+ val reader = lineSeparatorInRead match {
+ case Some(sep) => new LineReader(bomInputStream, sep)
+ case _ => new LineReader(bomInputStream)
+ }
+ new Iterator[Text] {
+ private val text = new Text()
+ private var finished = false
+ private var hasValue = false
+
+ override def hasNext: Boolean = {
+ if (!finished && !hasValue) {
+ finished = reader.readLine(text) == 0
+ hasValue = !finished
+ }
+ !finished
+ }
+
+ override def next(): Text = {
+ if (!hasNext) throw new NoSuchElementException
+ hasValue = false
+ text
+ }
+ }
+ }
+}
+
+/**
+ * [[ArchiveReader]] for tar archives: plain `.tar`, gzipped `.tar.gz`, and `.tgz`.
+ *
+ * Gzip handling: Hadoop's `CompressionCodecFactory` matches the trailing `.gz` extension and
+ * auto-decompresses `.tar.gz` via `CodecStreams`, so we just wrap that stream in
+ * `TarArchiveInputStream`. `.tgz` is not a registered Hadoop codec extension, so the gzip layer is
+ * unwrapped explicitly here.
+ */
+class TarArchiveReader(path: Path) extends ArchiveReader(path) {
+
+ // Paths Hadoop's codec factory won't auto-decompress: we apply the gzip layer here.
+ private def needsExplicitGunzip: Boolean =
+ path.getName.toLowerCase(Locale.ROOT).endsWith(".tgz")
+
+ /**
+ * Whether an entry is not a real data file and must be skipped: a directory, or a name Spark's
+ * own file listing would filter out. Reusing [[HadoopFSUtils.shouldFilterOutPathName]] (the
+ * `InMemoryFileIndex` filter) keeps archive reads in parity with reading the same entries as
+ * loose files: `.`-prefixed sidecars (macOS `._x`, `.DS_Store`) and `_`-prefixed markers
+ * (`_SUCCESS`, `_committed_*`) are skipped, while data files are kept.
+ */
+ private def shouldSkipEntry(entry: TarArchiveEntry): Boolean = {
+ if (entry.isDirectory) return true
+ val name = entry.getName
+ val basename = name.substring(name.lastIndexOf('/') + 1)
+ HadoopFSUtils.shouldFilterOutPathName(basename)
Review Comment:
Minor: the doc above says reusing `shouldFilterOutPathName` "keeps archive reads in parity with reading the same entries as loose files," but the filter is applied to the basename only, whereas `InMemoryFileIndex` also skips entire `_`/`.`-prefixed *directories* (it doesn't recurse into them). So an entry like `_temporary/part-0.csv` (basename `part-0.csv`) is read as data here but would be skipped as a loose file — e.g. a leftover `_temporary` from a failed write tarred up with the data. Narrow edge; either apply the filter per path component or soften the comment to say it filters entry *basenames* the way loose-file listing filters names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57135][SQL] Support reading CSV files inside tar archives [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #56193:
URL: https://github.com/apache/spark/pull/56193#issuecomment-4654391416
failed CDC tests unrelated, merging to master/4.x, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org