You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "akshatshenoi-db (via GitHub)" <gi...@apache.org> on 2026/08/06 01:30:30 UTC
[PR] [SPARK-58605][SQL] Verify parent-archive _metadata for archive reads [spark]
akshatshenoi-db opened a new pull request, #57805:
URL: https://github.com/apache/spark/pull/57805
### What changes were proposed in this pull request?
Adds tests asserting that reading an archive exposes the **parent archive file's** values in the `_metadata` column (`file_path`, `file_name`, `file_size`, `file_modification_time`, `file_block_start`, `file_block_length`).
An archive is read as a single non-splittable `PartitionedFile`, so the generic `FileFormat` metadata extractors already produce the parent archive's values (`file_block_start` = 0, `file_block_length` = `file_size` = the archive's size on disk). This PR is therefore **test-only**: no production change was needed. The tests pin the behavior so it cannot silently regress.
Inner-file `_metadata` is explicitly a **non-goal**: because the archive is a single split, the `_metadata` mechanism operates at `PartitionedFile` granularity and cannot attribute a row to its originating inner entry.
This follows the archive-reader series: SPARK-57135 / SPARK-57321 (CSV), SPARK-57419 (JSON), SPARK-57478 (text), SPARK-57479 (XML), SPARK-57481 (Avro), SPARK-58382 (binaryFile), SPARK-57705 (zip), SPARK-58246 (7z).
### Why are the changes needed?
The parent-archive `_metadata` contract was entirely untested. A future change to file splitting or to the metadata extractors could regress it unnoticed, and users reading archives rely on `_metadata` to identify the source archive file.
### Does this PR introduce any user-facing change?
No. Test-only; archive reading remains gated by `spark.sql.files.archive.reader.enabled` (default false).
### How was this patch tested?
A new shared test in `ArchiveReadSuiteBase`, which runs for every format x container suite (csv/json/xml/avro x tar/zip/7z), asserting every row of a multi-entry archive carries the same parent-archive values, with `file_block_start = 0` and `file_block_length = file_size = ` the archive's size.
Plus the parallel test in `TextArchiveReadBase` and `BinaryFileArchiveReadBase`, which do not extend `ArchiveReadSuiteBase` (their row shapes differ). For binaryFile with `wholeFile=false` the test additionally pins that `_metadata` stays parent-only even though the `path`/`length` **data** columns are sourced per entry.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58605][SQL] Verify parent-archive _metadata for archive reads [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #57805: [SPARK-58605][SQL] Verify parent-archive _metadata for archive reads
URL: https://github.com/apache/spark/pull/57805
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58605][SQL] Verify parent-archive _metadata for archive reads [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #57805:
URL: https://github.com/apache/spark/pull/57805#issuecomment-5203970660
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/cc7fac39d303e8f819edb20df165b66c53467177
- merged into branch-4.x https://github.com/apache/spark/commit/c392c9bd3483364fccb5d4659b1d214b15edc10d
- merged into branch-4.3 https://github.com/apache/spark/commit/ceeca31aaf6ecb7a2d582a223b53291e804fd4e8
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58605][SQL] Verify parent-archive _metadata for archive reads [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57805:
URL: https://github.com/apache/spark/pull/57805#issuecomment-5202494327
cc @cloud-fan who has more context on SupportsArchiveFormat
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org