You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/01/30 06:19:30 UTC
[GitHub] [arrow] liyafan82 opened a new pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader
liyafan82 opened a new pull request #12296:
URL: https://github.com/apache/arrow/pull/12296
When a malformed Arrow file containing an extremely large footer size (much larger than the file size) is fed to the ArrowFileReader, our implementation fails detect the problem, due to integer arithmetic overflow.
This will lead to extremely large memory allocation and eventually causing an OutOfMemoryError.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #12296:
URL: https://github.com/apache/arrow/pull/12296#discussion_r800131518
##########
File path: java/vector/src/main/java/org/apache/arrow/vector/ipc/ArrowMagic.java
##########
@@ -27,9 +27,9 @@
class ArrowMagic {
private ArrowMagic(){}
- private static final byte[] MAGIC = "ARROW1".getBytes(StandardCharsets.UTF_8);
+ static final byte[] MAGIC = "ARROW1".getBytes(StandardCharsets.UTF_8);
- public static final int MAGIC_LENGTH = MAGIC.length;
+ static final int MAGIC_LENGTH = MAGIC.length;
Review comment:
why make this package private? This seems like an unnecessary API breakage?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] liyafan82 commented on pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader
Posted by GitBox <gi...@apache.org>.
liyafan82 commented on pull request #12296:
URL: https://github.com/apache/arrow/pull/12296#issuecomment-1033465626
> LGTM, just one question about the API change. Did this come from the fuzzer? Have you opened up a seperate PR to add the fuzzing file to our corpus to prevent regressions?
@emkornfield Thanks for your feedback. This bug was discovered by the previous (C++) fuzzing files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] liyafan82 closed pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader
Posted by GitBox <gi...@apache.org>.
liyafan82 closed pull request #12296:
URL: https://github.com/apache/arrow/pull/12296
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12296:
URL: https://github.com/apache/arrow/pull/12296#issuecomment-1044125189
Benchmark runs are scheduled for baseline = c54d7b9c3edad0167402d9e2892aaeaf3c6e39c0 and contender = f49547b1504d9a514fd847307a2a6715959dc5e1. f49547b1504d9a514fd847307a2a6715959dc5e1 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/a1dd2d63922e4648a19b0790597c2a36...98cfc1f372c648eaaa0001542894cc58/)
[Finished :arrow_down:0.13% :arrow_up:0.0%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/d1e3e80da4f34834a7b00dcf47d80138...20a96a33dad64bf59e6701949b7bfcf1/)
[Failed :arrow_down:0.72% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/e5baa4ab715d407cbe73631ae6bbaeca...64f4025d0fda4bf3960310434303d75f/)
[Finished :arrow_down:0.21% :arrow_up:0.09%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/bfa12b0e32234f70bfc52b0db8a483da...4fcea6f04602422787e27290f8aaf720/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12296:
URL: https://github.com/apache/arrow/pull/12296#issuecomment-1025079815
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #12296:
URL: https://github.com/apache/arrow/pull/12296#issuecomment-1043999033
LGTM, please go ahead and merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12296:
URL: https://github.com/apache/arrow/pull/12296#issuecomment-1044125189
Benchmark runs are scheduled for baseline = c54d7b9c3edad0167402d9e2892aaeaf3c6e39c0 and contender = f49547b1504d9a514fd847307a2a6715959dc5e1. f49547b1504d9a514fd847307a2a6715959dc5e1 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/a1dd2d63922e4648a19b0790597c2a36...98cfc1f372c648eaaa0001542894cc58/)
[Scheduled] [test-mac-arm](https://conbench.ursa.dev/compare/runs/d1e3e80da4f34834a7b00dcf47d80138...20a96a33dad64bf59e6701949b7bfcf1/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/e5baa4ab715d407cbe73631ae6bbaeca...64f4025d0fda4bf3960310434303d75f/)
[Finished :arrow_down:0.21% :arrow_up:0.09%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/bfa12b0e32234f70bfc52b0db8a483da...4fcea6f04602422787e27290f8aaf720/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] liyafan82 commented on a change in pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader
Posted by GitBox <gi...@apache.org>.
liyafan82 commented on a change in pull request #12296:
URL: https://github.com/apache/arrow/pull/12296#discussion_r802368490
##########
File path: java/vector/src/main/java/org/apache/arrow/vector/ipc/ArrowMagic.java
##########
@@ -27,9 +27,9 @@
class ArrowMagic {
private ArrowMagic(){}
- private static final byte[] MAGIC = "ARROW1".getBytes(StandardCharsets.UTF_8);
+ static final byte[] MAGIC = "ARROW1".getBytes(StandardCharsets.UTF_8);
- public static final int MAGIC_LENGTH = MAGIC.length;
+ static final int MAGIC_LENGTH = MAGIC.length;
Review comment:
Sorry. I have reverted this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] liyafan82 commented on pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader
Posted by GitBox <gi...@apache.org>.
liyafan82 commented on pull request #12296:
URL: https://github.com/apache/arrow/pull/12296#issuecomment-1043995135
@emkornfield Thanks for your comments. If you have no more comment, may I merge it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12296:
URL: https://github.com/apache/arrow/pull/12296#issuecomment-1044125189
Benchmark runs are scheduled for baseline = c54d7b9c3edad0167402d9e2892aaeaf3c6e39c0 and contender = f49547b1504d9a514fd847307a2a6715959dc5e1. f49547b1504d9a514fd847307a2a6715959dc5e1 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/a1dd2d63922e4648a19b0790597c2a36...98cfc1f372c648eaaa0001542894cc58/)
[Scheduled] [test-mac-arm](https://conbench.ursa.dev/compare/runs/d1e3e80da4f34834a7b00dcf47d80138...20a96a33dad64bf59e6701949b7bfcf1/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/e5baa4ab715d407cbe73631ae6bbaeca...64f4025d0fda4bf3960310434303d75f/)
[Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/bfa12b0e32234f70bfc52b0db8a483da...4fcea6f04602422787e27290f8aaf720/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader
Posted by GitBox <gi...@apache.org>.
ursabot commented on pull request #12296:
URL: https://github.com/apache/arrow/pull/12296#issuecomment-1044125189
Benchmark runs are scheduled for baseline = c54d7b9c3edad0167402d9e2892aaeaf3c6e39c0 and contender = f49547b1504d9a514fd847307a2a6715959dc5e1. f49547b1504d9a514fd847307a2a6715959dc5e1 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Scheduled] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/a1dd2d63922e4648a19b0790597c2a36...98cfc1f372c648eaaa0001542894cc58/)
[Scheduled] [test-mac-arm](https://conbench.ursa.dev/compare/runs/d1e3e80da4f34834a7b00dcf47d80138...20a96a33dad64bf59e6701949b7bfcf1/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/e5baa4ab715d407cbe73631ae6bbaeca...64f4025d0fda4bf3960310434303d75f/)
[Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/bfa12b0e32234f70bfc52b0db8a483da...4fcea6f04602422787e27290f8aaf720/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] liyafan82 edited a comment on pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader
Posted by GitBox <gi...@apache.org>.
liyafan82 edited a comment on pull request #12296:
URL: https://github.com/apache/arrow/pull/12296#issuecomment-1033465626
> LGTM, just one question about the API change. Did this come from the fuzzer? Have you opened up a seperate PR to add the fuzzing file to our corpus to prevent regressions?
@emkornfield Thanks for your feedback. This bug was discovered by the previous (C++) fuzzing files, from a closed PR (https://github.com/apache/arrow/pull/10789/files)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12296:
URL: https://github.com/apache/arrow/pull/12296#issuecomment-1044125189
Benchmark runs are scheduled for baseline = c54d7b9c3edad0167402d9e2892aaeaf3c6e39c0 and contender = f49547b1504d9a514fd847307a2a6715959dc5e1. f49547b1504d9a514fd847307a2a6715959dc5e1 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/a1dd2d63922e4648a19b0790597c2a36...98cfc1f372c648eaaa0001542894cc58/)
[Finished :arrow_down:0.13% :arrow_up:0.0%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/d1e3e80da4f34834a7b00dcf47d80138...20a96a33dad64bf59e6701949b7bfcf1/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/e5baa4ab715d407cbe73631ae6bbaeca...64f4025d0fda4bf3960310434303d75f/)
[Finished :arrow_down:0.21% :arrow_up:0.09%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/bfa12b0e32234f70bfc52b0db8a483da...4fcea6f04602422787e27290f8aaf720/)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #12296:
URL: https://github.com/apache/arrow/pull/12296#issuecomment-1030760342
LGTM, just one question about the API change. Did this come from the fuzzer? Have you opened up a seperate PR to add the fuzzing file to our corpus to prevent regressions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org