You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "RussellSpitzer (via GitHub)" <gi...@apache.org> on 2025/11/12 21:34:36 UTC
[PR] Parquet Manifest - POC [iceberg]
RussellSpitzer opened a new pull request, #13769:
URL: https://github.com/apache/iceberg/pull/13769
A Proof of Concept pr to show what allowing writing to Parquet Manifests might look like as well as the illuminate the rough edges we will need to work around to get this into the project.
I'm collecting some thoughts on the work in the document below:
https://docs.google.com/document/d/13TeFu20jhkUFbb-FaaGQWFiXKO6kKyy_nX1QyFgoUoU/edit?usp=sharing
Implementation issues
### Empty Structs
Parquet does not support an empty struct in the schema but our current definition of the manifest has a required “partition” struct. For an unpartitioned table this means an empty struct is required in the schema.
POC Work - I work around this by just putting in a dummy placeholder struct instead.
### Public API from Manifest Reader Reliance on Manifest Metadata
Currently the Manifest Reader attempts to extract table’s Specs information from inside the avro metadata. This isn’t actually required by any of our code and there is basically never a time in production where we read a manifest and don’t already have the specs. The tests are a different case and require a lot of rewriting.
There is unfortunately one major exception to this,
Snapshot provides several apis like (addedFiles(FileIO io)) which reads manifests directly with a FileIO to produce their entries, this creates a manifest group directly but without setting specById. In this context we also do not have access to the table, we probably will need to change this interface if we want to remove reliance on file metadata.
POC Work - Originally I focused on just fixing all the test references after skimming and trying to determine if there were any real production uses of the API. Once I discovered the issue with BaseSnapshot I tried to determine exactly the scope, 30 of the above test failures are related to this issue. Many are test related (checking snapshot contents) but there are some public usages in cherry-pick snapshot.
### Refactoring of Parquet InternalReader Implementation
The current parquet InternalReader API returns a reader function before we are able to get arguments for custom classes. ManifestReader needs to be able to read into custom classes (ManifestEntry, DataFile, etc ..) so the custom classes need to be created prior to the reader function being generated.
POC Work - Bit of a sloppy fix here but we move the reader into the Parquet class itself so that we can modify the function before it is built.
### Use of Immutables in BaseFile
Parquet readers using a reusable container fail because BaseFile returns an immutable representation of the splitOffset list.
POC Work - Just return a mutable list for now
### Determining Manifest Type Based on Extension
In order to have different behaviors for different types but allow reading both avro and parquet manifests, we need to rely on extensions. Most likely not a problem in the wild, but many of our tests use manifest file names without a type extension.
POC Work - Just fixing tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet Manifest - POC [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #13769:
URL: https://github.com/apache/iceberg/pull/13769#discussion_r2263211900
##########
build.gradle:
##########
@@ -341,6 +341,7 @@ project(':iceberg-core') {
api project(':iceberg-api')
implementation project(':iceberg-common')
implementation project(path: ':iceberg-bundled-guava', configuration: 'shadow')
+ testRuntimeOnly project(':iceberg-parquet')
Review Comment:
Allows tests to use parquet while running (it's only loaded via reflection)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet Manifest - POC [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer closed pull request #13769: Parquet Manifest - POC
URL: https://github.com/apache/iceberg/pull/13769
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet Manifest - POC [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on PR #13769:
URL: https://github.com/apache/iceberg/pull/13769#issuecomment-3527164579
Added non-stale lable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet Manifest - POC [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #13769: Parquet Manifest - POC
URL: https://github.com/apache/iceberg/pull/13769
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet Manifest - POC [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on PR #13769:
URL: https://github.com/apache/iceberg/pull/13769#issuecomment-3524082309
reopened the PR as @RussellSpitzer intends to resume this work upon his return from leave.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet Manifest - POC [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #13769:
URL: https://github.com/apache/iceberg/pull/13769#discussion_r2263219831
##########
core/src/main/java/org/apache/iceberg/ManifestReader.java:
##########
@@ -146,18 +153,17 @@ private <T extends ContentFile<T>> PartitionSpec readPartitionSpec(InputFile inp
}
private static <T extends ContentFile<T>> Map<String, String> readMetadata(InputFile inputFile) {
- Map<String, String> metadata;
+ Map<String, String> metadata = Collections.emptyMap();
+ FileFormat manifestFormat = FileFormat.fromFileName(inputFile.location());
+
try {
try (CloseableIterable<ManifestEntry<T>> headerReader =
- InternalData.read(FileFormat.AVRO, inputFile)
Review Comment:
This breaks a backwards compatibility, previously an input file didn't actually have to have an avro suffix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet Manifest - POC [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13769:
URL: https://github.com/apache/iceberg/pull/13769#issuecomment-3383626595
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Parquet Manifest - POC [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13769:
URL: https://github.com/apache/iceberg/pull/13769#issuecomment-3408704201
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org