You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "pan3793 (via GitHub)" <gi...@apache.org> on 2025/08/06 13:57:10 UTC
[PR] GH-3141: Add constructor to `ParquetFileReader` to allow passing in parquet footer and expose setRequestedSchema that accepts `List` [parquet-java]
pan3793 opened a new pull request, #3262:
URL: https://github.com/apache/parquet-java/pull/3262
<!--
Thanks for opening a pull request!
If you're new to Parquet-Java, information on how to contribute can be found here: https://parquet.apache.org/docs/contribution-guidelines/contributing
Please open a GitHub issue for this pull request: https://github.com/apache/parquet-java/issues/new/choose
and format pull request title as below:
GH-${GITHUB_ISSUE_ID}: ${SUMMARY}
or simply use the title below if it is a minor issue:
MINOR: ${SUMMARY}
-->
Co-authored-by: David Zhu <da...@alluxio.com>
### Rationale for this change
This PR takes over #3165 as the author is silent for a while.
The proposed API allows the caller to reuse InputStream to save HDFS NameNode RPCs, https://github.com/apache/spark/pull/50765 demonstrates how this PR benefits Spark.
### What changes are included in this PR?
- Add constructor to `ParquetFileReader` to allow passing in parquet footer
- Add new method `setRequestedSchema(List<ColumnDescriptor>)` to `ParquetFileReader`
- UT is modified to use the newly added API.
### Are these changes tested?
Yes, UT is tuned.
### Are there any user-facing changes?
It exposes new public APIs for `ParquetFileReader`.
<!-- Please uncomment the line below and replace ${GITHUB_ISSUE_ID} with the actual Github issue id. -->
Closes #3141
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3141: Add constructor to `ParquetFileReader` to allow passing in parquet footer and expose setRequestedSchema that accepts `List` [parquet-java]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #3262:
URL: https://github.com/apache/parquet-java/pull/3262#discussion_r2292706189
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -567,8 +567,23 @@ public static final ParquetMetadata readFooter(InputFile file, MetadataFilter fi
public static final ParquetMetadata readFooter(InputFile file, ParquetReadOptions options, SeekableInputStream f)
throws IOException {
+ return readFooter(file, options, f, /*closeStreamOnFailure*/ false);
+ }
+
+ public static final ParquetMetadata readFooter(
Review Comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3141: Add constructor to `ParquetFileReader` to allow passing in parquet footer and expose setRequestedSchema that accepts `List` [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #3262:
URL: https://github.com/apache/parquet-java/pull/3262#discussion_r2292567404
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -943,6 +965,51 @@ public ParquetFileReader(InputFile file, ParquetReadOptions options, SeekableInp
f.close();
throw e;
}
+
+ this.fileMetaData = footer.getFileMetaData();
Review Comment:
Can't we simply write this:
```java
public ParquetFileReader(InputFile file, ParquetReadOptions options, SeekableInputStream f) throws IOException {
this(file, readFooter(file, options, f, /*closeStreamOnFailure=*/true), options, f);
}
```
To keep the same behavior, we also need to modify readFooter to close the stream once failed:
```java
public static final ParquetMetadata readFooter(InputFile file, ParquetReadOptions options, SeekableInputStream f, boolean closeStreamOnFailure)
throws IOException {
ParquetMetadataConverter converter = new ParquetMetadataConverter(options);
try {
return readFooter(file, options, f, converter);
} catch (Exception e) {
// In case that readFooter throws an exception in the constructor, the new stream
// should be closed. Otherwise, there's no way to close this outside.
if (closeStreamOnFailure) {
f.close();
}
throw e;
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3141: Add constructor to `ParquetFileReader` to allow passing in parquet footer and expose setRequestedSchema that accepts `List` [parquet-java]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #3262:
URL: https://github.com/apache/parquet-java/pull/3262#discussion_r2292571566
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -943,6 +965,51 @@ public ParquetFileReader(InputFile file, ParquetReadOptions options, SeekableInp
f.close();
throw e;
}
+
+ this.fileMetaData = footer.getFileMetaData();
Review Comment:
`readFooter` is a public API, I don't think we should change the semantics silently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3141: Add constructor to `ParquetFileReader` to allow passing in parquet footer and expose setRequestedSchema that accepts `List` [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #3262:
URL: https://github.com/apache/parquet-java/pull/3262#discussion_r2290329226
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -943,6 +965,51 @@ public ParquetFileReader(InputFile file, ParquetReadOptions options, SeekableInp
f.close();
throw e;
}
+
+ this.fileMetaData = footer.getFileMetaData();
Review Comment:
Can we use a private delegate constructor?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3141: Add constructor to `ParquetFileReader` to allow passing in parquet footer and expose setRequestedSchema that accepts `List` [parquet-java]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #3262:
URL: https://github.com/apache/parquet-java/pull/3262#discussion_r2292571566
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -943,6 +965,51 @@ public ParquetFileReader(InputFile file, ParquetReadOptions options, SeekableInp
f.close();
throw e;
}
+
+ this.fileMetaData = footer.getFileMetaData();
Review Comment:
~`readFooter` is a public API, I don't think we should change the semantics silently~
oh, I see, you add a new parameter `closeStreamOnFailure `
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -943,6 +965,51 @@ public ParquetFileReader(InputFile file, ParquetReadOptions options, SeekableInp
f.close();
throw e;
}
+
+ this.fileMetaData = footer.getFileMetaData();
Review Comment:
~`readFooter` is a public API, I don't think we should change the semantics silently~
oh, I see, you add a new parameter `closeStreamOnFailure`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3141: Add constructor to `ParquetFileReader` to allow passing in parquet footer and expose setRequestedSchema that accepts `List` [parquet-java]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #3262:
URL: https://github.com/apache/parquet-java/pull/3262#discussion_r2290272734
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -943,6 +965,51 @@ public ParquetFileReader(InputFile file, ParquetReadOptions options, SeekableInp
f.close();
throw e;
}
+
+ this.fileMetaData = footer.getFileMetaData();
Review Comment:
@wgtmac I'm afraid we can not do that
<img width="1002" height="949" alt="image" src="https://github.com/user-attachments/assets/a500b806-526b-463b-9723-b1521a1bbbbb" />
I also tried to make `ParquetMetadata footer` nullable, and read from the file when `footer` is NULL, but this does not work either, because some places pass NULL explicitly.
To completely fix the code duplication issue of `ParquetFileReader`, I think we should introduce a `Builder` for it, but this is out of the scope of this PR. Given that this is a public API used widely by downstream projects, deprecating current APIs and suggesting migrating to new APIs is another burden ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3141: Add constructor to `ParquetFileReader` to allow passing in parquet footer and expose setRequestedSchema that accepts `List` [parquet-java]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #3262:
URL: https://github.com/apache/parquet-java/pull/3262#discussion_r2290272734
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -943,6 +965,51 @@ public ParquetFileReader(InputFile file, ParquetReadOptions options, SeekableInp
f.close();
throw e;
}
+
+ this.fileMetaData = footer.getFileMetaData();
Review Comment:
@wgtmac I'm afraid we can not do that
<img width="1002" height="949" alt="image" src="https://github.com/user-attachments/assets/a500b806-526b-463b-9723-b1521a1bbbbb" />
I also tried to make `ParquetMetadata footer` nullable, and read from the file when `footer` is NULL, but this does not work either, because some places pass NULL explicitly.
To completely fix the code duplication issue of ParquetFileReader, I think we should introduce a `Builder` for it, but this is out of the scope of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3141: Add constructor to `ParquetFileReader` to allow passing in parquet footer and expose setRequestedSchema that accepts `List` [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #3262:
URL: https://github.com/apache/parquet-java/pull/3262#discussion_r2259152621
##########
parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestDataPageChecksums.java:
##########
@@ -688,8 +690,13 @@ private int getDataOffset(Page page) {
*/
private ParquetFileReader getParquetFileReader(Path path, Configuration conf, List<ColumnDescriptor> columns)
throws IOException {
- ParquetMetadata footer = ParquetFileReader.readFooter(conf, path);
- return new ParquetFileReader(conf, footer.getFileMetaData(), path, footer.getBlocks(), columns);
+ HadoopInputFile inputFile = HadoopInputFile.fromPath(path, conf);
+ SeekableInputStream inputStream = inputFile.newStream();
+ ParquetReadOptions readOptions = HadoopReadOptions.builder(conf).build();
+ ParquetMetadata footer = ParquetFileReader.readFooter(inputFile, readOptions, inputStream);
+ ParquetFileReader reader = ParquetFileReader.open(inputFile, footer, readOptions, inputStream);
+ reader.setRequestedSchema(columns);
+ return reader;
Review Comment:
It would be better to explicitly add a new test case rather than silently changing old ones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3141: Add constructor to `ParquetFileReader` to allow passing in parquet footer and expose setRequestedSchema that accepts `List` [parquet-java]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #3262:
URL: https://github.com/apache/parquet-java/pull/3262#discussion_r2259188854
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1054,13 +1121,17 @@ public List<BlockMetaData> getRowGroups() {
return blocks;
}
- public void setRequestedSchema(MessageType projection) {
+ public void setRequestedSchema(List<ColumnDescriptor> columns) {
Review Comment:
the diff here is not smart, this actually adds a new method
`public void setRequestedSchema(List<ColumnDescriptor> columns)`
there are many ctors of `ParquetFileReader` were marked as deprecated but without providing the equivalent replacement, for example,
```
@Deprecated
public ParquetFileReader(
Configuration configuration,
FileMetaData fileMetaData,
Path filePath,
List<BlockMetaData> blocks,
List<ColumnDescriptor> columns)
```
this API is proposed to restore the ability to set read columns via `List<ColumnDescriptor>`, instead of `MessageType`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3141: Add constructor to `ParquetFileReader` to allow passing in parquet footer and expose setRequestedSchema that accepts `List` [parquet-java]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #3262:
URL: https://github.com/apache/parquet-java/pull/3262#discussion_r2290352850
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -943,6 +965,51 @@ public ParquetFileReader(InputFile file, ParquetReadOptions options, SeekableInp
f.close();
throw e;
}
+
+ this.fileMetaData = footer.getFileMetaData();
Review Comment:
Seems infeasible before Java 23 ([JEP 482](https://openjdk.org/jeps/482))
<img width="968" height="593" alt="image" src="https://github.com/user-attachments/assets/f3dcc7f6-5647-4c3b-a4ad-d8565b613b89" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3141: Add constructor to `ParquetFileReader` to allow passing in parquet footer and expose setRequestedSchema that accepts `List` [parquet-java]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #3262:
URL: https://github.com/apache/parquet-java/pull/3262#issuecomment-3160291614
cc @wgtmac @wangyum
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3141: Add constructor to `ParquetFileReader` to allow passing in parquet footer and expose setRequestedSchema that accepts `List` [parquet-java]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #3262:
URL: https://github.com/apache/parquet-java/pull/3262#discussion_r2292705266
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -567,8 +567,23 @@ public static final ParquetMetadata readFooter(InputFile file, MetadataFilter fi
public static final ParquetMetadata readFooter(InputFile file, ParquetReadOptions options, SeekableInputStream f)
throws IOException {
+ return readFooter(file, options, f, /*closeStreamOnFailure*/ false);
+ }
+
+ public static final ParquetMetadata readFooter(
Review Comment:
```suggestion
private static final ParquetMetadata readFooter(
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3141: Add constructor to `ParquetFileReader` to allow passing in parquet footer and expose setRequestedSchema that accepts `List` [parquet-java]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #3262:
URL: https://github.com/apache/parquet-java/pull/3262#discussion_r2259194441
##########
parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestDataPageChecksums.java:
##########
@@ -688,8 +690,13 @@ private int getDataOffset(Page page) {
*/
private ParquetFileReader getParquetFileReader(Path path, Configuration conf, List<ColumnDescriptor> columns)
throws IOException {
- ParquetMetadata footer = ParquetFileReader.readFooter(conf, path);
- return new ParquetFileReader(conf, footer.getFileMetaData(), path, footer.getBlocks(), columns);
+ HadoopInputFile inputFile = HadoopInputFile.fromPath(path, conf);
+ SeekableInputStream inputStream = inputFile.newStream();
+ ParquetReadOptions readOptions = HadoopReadOptions.builder(conf).build();
+ ParquetMetadata footer = ParquetFileReader.readFooter(inputFile, readOptions, inputStream);
+ ParquetFileReader reader = ParquetFileReader.open(inputFile, footer, readOptions, inputStream);
+ reader.setRequestedSchema(columns);
+ return reader;
Review Comment:
I understand that, but it seems there is no dedicated test suite for `ParquetFileReader` constructor. The replaced API is deprecated, and the replacement does not reduce the test coverage because the deprecated API is used many times in other test suites.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3141: Add constructor to `ParquetFileReader` to allow passing in parquet footer and expose setRequestedSchema that accepts `List` [parquet-java]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #3262:
URL: https://github.com/apache/parquet-java/pull/3262#discussion_r2290352850
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -943,6 +965,51 @@ public ParquetFileReader(InputFile file, ParquetReadOptions options, SeekableInp
f.close();
throw e;
}
+
+ this.fileMetaData = footer.getFileMetaData();
Review Comment:
Seems infeasible before Java 23 ([JEP 482](https://openjdk.org/jeps/482))
<img width="1072" height="469" alt="image" src="https://github.com/user-attachments/assets/36a1a958-ecaf-4276-b72a-9753401e22ec" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3141: Add constructor to `ParquetFileReader` to allow passing in parquet footer and expose setRequestedSchema that accepts `List` [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #3262:
URL: https://github.com/apache/parquet-java/pull/3262#discussion_r2289890153
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -943,6 +965,51 @@ public ParquetFileReader(InputFile file, ParquetReadOptions options, SeekableInp
f.close();
throw e;
}
+
+ this.fileMetaData = footer.getFileMetaData();
Review Comment:
Ah, I found these lines duplicate a lot with the below ctor. Can we create a new method instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3141: Add constructor to `ParquetFileReader` to allow passing in parquet footer and expose setRequestedSchema that accepts `List` [parquet-java]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #3262:
URL: https://github.com/apache/parquet-java/pull/3262#discussion_r2290352850
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -943,6 +965,51 @@ public ParquetFileReader(InputFile file, ParquetReadOptions options, SeekableInp
f.close();
throw e;
}
+
+ this.fileMetaData = footer.getFileMetaData();
Review Comment:
Seems infeasible before Java 23 ([JEP 448](https://openjdk.org/jeps/448))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3141: Add constructor to `ParquetFileReader` to allow passing in parquet footer and expose setRequestedSchema that accepts `List` [parquet-java]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #3262:
URL: https://github.com/apache/parquet-java/pull/3262#discussion_r2290272734
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -943,6 +965,51 @@ public ParquetFileReader(InputFile file, ParquetReadOptions options, SeekableInp
f.close();
throw e;
}
+
+ this.fileMetaData = footer.getFileMetaData();
Review Comment:
@wgtmac I'm afraid we can not do that
<img width="1002" height="949" alt="image" src="https://github.com/user-attachments/assets/a500b806-526b-463b-9723-b1521a1bbbbb" />
I also tried to make `ParquetMetadata footer` nullable, and read from the file when `footer` is NULL, but this does not work either, because some places pass NULL explicitly.
To completely fix the code duplication issue of `ParquetFileReader`, I think we should introduce a `Builder` for it, but this is out of the scope of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3141: Add constructor to `ParquetFileReader` to allow passing in parquet footer and expose setRequestedSchema that accepts `List` [parquet-java]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #3262:
URL: https://github.com/apache/parquet-java/pull/3262#discussion_r2290352850
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -943,6 +965,51 @@ public ParquetFileReader(InputFile file, ParquetReadOptions options, SeekableInp
f.close();
throw e;
}
+
+ this.fileMetaData = footer.getFileMetaData();
Review Comment:
Seems infeasible before Java 23 ([JEP 482](https://openjdk.org/jeps/482))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3141: Add constructor to `ParquetFileReader` to allow passing in parquet footer and expose setRequestedSchema that accepts `List` [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #3262:
URL: https://github.com/apache/parquet-java/pull/3262#discussion_r2259148285
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1054,13 +1121,17 @@ public List<BlockMetaData> getRowGroups() {
return blocks;
}
- public void setRequestedSchema(MessageType projection) {
+ public void setRequestedSchema(List<ColumnDescriptor> columns) {
Review Comment:
Why do we need this change? Does it exist in the original PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3141: Add constructor to `ParquetFileReader` to allow passing in parquet footer and expose setRequestedSchema that accepts `List` [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #3262:
URL: https://github.com/apache/parquet-java/pull/3262#issuecomment-3213123710
Thanks @pan3793 and @wangyum!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3141: Add constructor to `ParquetFileReader` to allow passing in parquet footer and expose setRequestedSchema that accepts `List` [parquet-java]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #3262:
URL: https://github.com/apache/parquet-java/pull/3262#discussion_r2292584561
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -943,6 +965,51 @@ public ParquetFileReader(InputFile file, ParquetReadOptions options, SeekableInp
f.close();
throw e;
}
+
+ this.fileMetaData = footer.getFileMetaData();
Review Comment:
@wgtmac thank you for the suggestion, it's feasible, updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3141: Add constructor to `ParquetFileReader` to allow passing in parquet footer and expose setRequestedSchema that accepts `List` [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #3262:
URL: https://github.com/apache/parquet-java/pull/3262#discussion_r2292692859
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -567,8 +567,23 @@ public static final ParquetMetadata readFooter(InputFile file, MetadataFilter fi
public static final ParquetMetadata readFooter(InputFile file, ParquetReadOptions options, SeekableInputStream f)
throws IOException {
+ return readFooter(file, options, f, /*closeStreamOnFailure*/ false);
+ }
+
+ public static final ParquetMetadata readFooter(
Review Comment:
nit: can we make it private?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3141: Add constructor to `ParquetFileReader` to allow passing in parquet footer and expose setRequestedSchema that accepts `List` [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #3262:
URL: https://github.com/apache/parquet-java/pull/3262#discussion_r2292691719
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -943,6 +965,51 @@ public ParquetFileReader(InputFile file, ParquetReadOptions options, SeekableInp
f.close();
throw e;
}
+
+ this.fileMetaData = footer.getFileMetaData();
Review Comment:
Thanks a lot for your effort!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3141: Add constructor to `ParquetFileReader` to allow passing in parquet footer and expose setRequestedSchema that accepts `List` [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac merged PR #3262:
URL: https://github.com/apache/parquet-java/pull/3262
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3141: Add constructor to `ParquetFileReader` to allow passing in parquet footer and expose setRequestedSchema that accepts `List` [parquet-java]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #3262:
URL: https://github.com/apache/parquet-java/pull/3262#discussion_r2290352850
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -943,6 +965,51 @@ public ParquetFileReader(InputFile file, ParquetReadOptions options, SeekableInp
f.close();
throw e;
}
+
+ this.fileMetaData = footer.getFileMetaData();
Review Comment:
Seems infeasible before Java 22 ([JEP 447](https://openjdk.org/jeps/447))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org