You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "ConeyLiu (via GitHub)" <gi...@apache.org> on 2025/01/24 08:50:58 UTC
[PR] [MINOR]: Fail fast when footer size larger than Int.MaxValue [parquet-java]
ConeyLiu opened a new pull request, #3136:
URL: https://github.com/apache/parquet-java/pull/3136
<!--
Thanks for opening a pull request!
If you're new to Parquet-Java, information on how to contribute can be found here: https://parquet.apache.org/docs/contribution-guidelines/contributing
Please open a GitHub issue for this pull request: https://github.com/apache/parquet-java/issues/new/choose
and format pull request title as below:
GH-${GITHUB_ISSUE_ID}: ${SUMMARY}
or simply use the title below if it is a minor issue:
MINOR: ${SUMMARY}
-->
### Rationale for this change
The footer size is limited to unsigned int for some other language implementations, such as Rust. However, it is limited to Int.MaxValue on the Java side due to some limitations(eg, lack of unsigned int supports, the biggest byte array/buffer is limited to Int.MaxValue). So this PR reads the footer size as long and fails fast if it is larger than Int.MaxValue.
### What changes are included in this PR?
### Are these changes tested?
Existed UTs.
### Are there any user-facing changes?
No
<!-- Please uncomment the line below and replace ${GITHUB_ISSUE_ID} with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] [MINOR]: Fail fast when footer size larger than Int.MaxValue [parquet-java]
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on code in PR #3136:
URL: https://github.com/apache/parquet-java/pull/3136#discussion_r1928349167
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -589,7 +589,12 @@ private static final ParquetMetadata readFooter(
long fileMetadataLengthIndex = fileLen - magic.length - FOOTER_LENGTH_SIZE;
LOG.debug("reading footer index at {}", fileMetadataLengthIndex);
f.seek(fileMetadataLengthIndex);
- int fileMetadataLength = readIntLittleEndian(f);
+ long readFileMetadataLength = readIntLittleEndian(f) & 0xFFFFFFFFL;
Review Comment:
read as a unsigned int
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] MINOR: Reader fails fast when footer size is larger than INT_MAX [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac merged PR #3136:
URL: https://github.com/apache/parquet-java/pull/3136
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] [MINOR]: Fail fast when footer size larger than Int.MaxValue [parquet-java]
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on PR #3136:
URL: https://github.com/apache/parquet-java/pull/3136#issuecomment-2611997502
@dylanburati I have tried to support the footer with an unsigned int size in Java implementation. However, due to some limitations, it can not be done. We could write a footer(without encrypted) size larger than Int.MaxValue bytes, while we can not read it due to some limitations in thrift.
```java
java.io.IOException: can not read class org.apache.parquet.format.FileMetaData: MaxMessageSize reached
at org.apache.parquet.format.Util.read(Util.java:393)
at org.apache.parquet.format.Util.readFileMetaData(Util.java:156)
at org.apache.parquet.format.converter.ParquetMetadataConverter$3.visit(ParquetMetadataConverter.java:1545)
at org.apache.parquet.format.converter.ParquetMetadataConverter$3.visit(ParquetMetadataConverter.java:1542)
at org.apache.parquet.format.converter.ParquetMetadataConverter$NoFilter.accept(ParquetMetadataConverter.java:1242)
at org.apache.parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:1542)
at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:637)
at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:571)
Caused by: org.apache.thrift.transport.TTransportException: MaxMessageSize reached
at org.apache.thrift.transport.TEndpointTransport.checkReadBytesAvailable(TEndpointTransport.java:94)
at org.apache.thrift.protocol.TCompactProtocol.checkStringReadLength(TCompactProtocol.java:726)
at org.apache.thrift.protocol.TCompactProtocol.readString(TCompactProtocol.java:675)
at org.apache.parquet.format.InterningProtocol.readString(InterningProtocol.java:269)
at org.apache.parquet.format.KeyValue$KeyValueStandardScheme.read(KeyValue.java:415)
at org.apache.parquet.format.KeyValue$KeyValueStandardScheme.read(KeyValue.java:392)
at org.apache.parquet.format.KeyValue.read(KeyValue.java:327)
at org.apache.parquet.format.FileMetaData$FileMetaDataStandardScheme.read(FileMetaData.java:1338)
```
Since it is a rare condition. Here we just fail fast with an exception.
cc @wgtmac
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] MINOR: Reader fails fast when footer size is larger than INT_MAX [parquet-java]
Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on PR #3136:
URL: https://github.com/apache/parquet-java/pull/3136#issuecomment-2700855278
Thanks @wgtmac
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org