You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/01/30 06:19:30 UTC

[GitHub] [arrow] liyafan82 opened a new pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader

liyafan82 opened a new pull request #12296:
URL: https://github.com/apache/arrow/pull/12296


   When a malformed Arrow file containing an extremely large footer size (much larger than the file size) is fed to the ArrowFileReader, our implementation fails detect the problem, due to integer arithmetic overflow.
   
   This will lead to extremely large memory allocation and eventually causing an OutOfMemoryError.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] emkornfield commented on a change in pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader

Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #12296:
URL: https://github.com/apache/arrow/pull/12296#discussion_r800131518



##########
File path: java/vector/src/main/java/org/apache/arrow/vector/ipc/ArrowMagic.java
##########
@@ -27,9 +27,9 @@
 class ArrowMagic {
   private ArrowMagic(){}
 
-  private static final byte[] MAGIC = "ARROW1".getBytes(StandardCharsets.UTF_8);
+  static final byte[] MAGIC = "ARROW1".getBytes(StandardCharsets.UTF_8);
 
-  public static final int MAGIC_LENGTH = MAGIC.length;
+  static final int MAGIC_LENGTH = MAGIC.length;

Review comment:
       why make this package private?  This seems like an unnecessary API breakage?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] liyafan82 commented on pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader

Posted by GitBox <gi...@apache.org>.
liyafan82 commented on pull request #12296:
URL: https://github.com/apache/arrow/pull/12296#issuecomment-1033465626


   > LGTM, just one question about the API change. Did this come from the fuzzer? Have you opened up a seperate PR to add the fuzzing file to our corpus to prevent regressions?
   
   @emkornfield Thanks for your feedback. This bug was discovered by the previous (C++) fuzzing files. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] liyafan82 closed pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader

Posted by GitBox <gi...@apache.org>.
liyafan82 closed pull request #12296:
URL: https://github.com/apache/arrow/pull/12296


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] ursabot edited a comment on pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader

Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12296:
URL: https://github.com/apache/arrow/pull/12296#issuecomment-1044125189


   Benchmark runs are scheduled for baseline = c54d7b9c3edad0167402d9e2892aaeaf3c6e39c0 and contender = f49547b1504d9a514fd847307a2a6715959dc5e1. f49547b1504d9a514fd847307a2a6715959dc5e1 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
   Conbench compare runs links:
   [Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/a1dd2d63922e4648a19b0790597c2a36...98cfc1f372c648eaaa0001542894cc58/)
   [Finished :arrow_down:0.13% :arrow_up:0.0%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/d1e3e80da4f34834a7b00dcf47d80138...20a96a33dad64bf59e6701949b7bfcf1/)
   [Failed :arrow_down:0.72% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/e5baa4ab715d407cbe73631ae6bbaeca...64f4025d0fda4bf3960310434303d75f/)
   [Finished :arrow_down:0.21% :arrow_up:0.09%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/bfa12b0e32234f70bfc52b0db8a483da...4fcea6f04602422787e27290f8aaf720/)
   Supported benchmarks:
   ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
   test-mac-arm: Supported benchmark langs: C++, Python, R
   ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
   ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] github-actions[bot] commented on pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12296:
URL: https://github.com/apache/arrow/pull/12296#issuecomment-1025079815






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] emkornfield commented on pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader

Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #12296:
URL: https://github.com/apache/arrow/pull/12296#issuecomment-1043999033


   LGTM, please go ahead and merge.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] ursabot edited a comment on pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader

Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12296:
URL: https://github.com/apache/arrow/pull/12296#issuecomment-1044125189


   Benchmark runs are scheduled for baseline = c54d7b9c3edad0167402d9e2892aaeaf3c6e39c0 and contender = f49547b1504d9a514fd847307a2a6715959dc5e1. f49547b1504d9a514fd847307a2a6715959dc5e1 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
   Conbench compare runs links:
   [Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/a1dd2d63922e4648a19b0790597c2a36...98cfc1f372c648eaaa0001542894cc58/)
   [Scheduled] [test-mac-arm](https://conbench.ursa.dev/compare/runs/d1e3e80da4f34834a7b00dcf47d80138...20a96a33dad64bf59e6701949b7bfcf1/)
   [Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/e5baa4ab715d407cbe73631ae6bbaeca...64f4025d0fda4bf3960310434303d75f/)
   [Finished :arrow_down:0.21% :arrow_up:0.09%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/bfa12b0e32234f70bfc52b0db8a483da...4fcea6f04602422787e27290f8aaf720/)
   Supported benchmarks:
   ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
   test-mac-arm: Supported benchmark langs: C++, Python, R
   ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
   ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] liyafan82 commented on a change in pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader

Posted by GitBox <gi...@apache.org>.
liyafan82 commented on a change in pull request #12296:
URL: https://github.com/apache/arrow/pull/12296#discussion_r802368490



##########
File path: java/vector/src/main/java/org/apache/arrow/vector/ipc/ArrowMagic.java
##########
@@ -27,9 +27,9 @@
 class ArrowMagic {
   private ArrowMagic(){}
 
-  private static final byte[] MAGIC = "ARROW1".getBytes(StandardCharsets.UTF_8);
+  static final byte[] MAGIC = "ARROW1".getBytes(StandardCharsets.UTF_8);
 
-  public static final int MAGIC_LENGTH = MAGIC.length;
+  static final int MAGIC_LENGTH = MAGIC.length;

Review comment:
       Sorry. I have reverted this change. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] liyafan82 commented on pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader

Posted by GitBox <gi...@apache.org>.
liyafan82 commented on pull request #12296:
URL: https://github.com/apache/arrow/pull/12296#issuecomment-1043995135


   @emkornfield Thanks for your comments. If you have no more comment, may I merge it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] ursabot edited a comment on pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader

Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12296:
URL: https://github.com/apache/arrow/pull/12296#issuecomment-1044125189


   Benchmark runs are scheduled for baseline = c54d7b9c3edad0167402d9e2892aaeaf3c6e39c0 and contender = f49547b1504d9a514fd847307a2a6715959dc5e1. f49547b1504d9a514fd847307a2a6715959dc5e1 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
   Conbench compare runs links:
   [Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/a1dd2d63922e4648a19b0790597c2a36...98cfc1f372c648eaaa0001542894cc58/)
   [Scheduled] [test-mac-arm](https://conbench.ursa.dev/compare/runs/d1e3e80da4f34834a7b00dcf47d80138...20a96a33dad64bf59e6701949b7bfcf1/)
   [Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/e5baa4ab715d407cbe73631ae6bbaeca...64f4025d0fda4bf3960310434303d75f/)
   [Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/bfa12b0e32234f70bfc52b0db8a483da...4fcea6f04602422787e27290f8aaf720/)
   Supported benchmarks:
   ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
   test-mac-arm: Supported benchmark langs: C++, Python, R
   ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
   ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] ursabot commented on pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader

Posted by GitBox <gi...@apache.org>.
ursabot commented on pull request #12296:
URL: https://github.com/apache/arrow/pull/12296#issuecomment-1044125189


   Benchmark runs are scheduled for baseline = c54d7b9c3edad0167402d9e2892aaeaf3c6e39c0 and contender = f49547b1504d9a514fd847307a2a6715959dc5e1. f49547b1504d9a514fd847307a2a6715959dc5e1 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
   Conbench compare runs links:
   [Scheduled] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/a1dd2d63922e4648a19b0790597c2a36...98cfc1f372c648eaaa0001542894cc58/)
   [Scheduled] [test-mac-arm](https://conbench.ursa.dev/compare/runs/d1e3e80da4f34834a7b00dcf47d80138...20a96a33dad64bf59e6701949b7bfcf1/)
   [Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/e5baa4ab715d407cbe73631ae6bbaeca...64f4025d0fda4bf3960310434303d75f/)
   [Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/bfa12b0e32234f70bfc52b0db8a483da...4fcea6f04602422787e27290f8aaf720/)
   Supported benchmarks:
   ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
   test-mac-arm: Supported benchmark langs: C++, Python, R
   ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
   ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] liyafan82 edited a comment on pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader

Posted by GitBox <gi...@apache.org>.
liyafan82 edited a comment on pull request #12296:
URL: https://github.com/apache/arrow/pull/12296#issuecomment-1033465626


   > LGTM, just one question about the API change. Did this come from the fuzzer? Have you opened up a seperate PR to add the fuzzing file to our corpus to prevent regressions?
   
   @emkornfield Thanks for your feedback. This bug was discovered by the previous (C++) fuzzing files, from a closed PR (https://github.com/apache/arrow/pull/10789/files)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] ursabot edited a comment on pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader

Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #12296:
URL: https://github.com/apache/arrow/pull/12296#issuecomment-1044125189


   Benchmark runs are scheduled for baseline = c54d7b9c3edad0167402d9e2892aaeaf3c6e39c0 and contender = f49547b1504d9a514fd847307a2a6715959dc5e1. f49547b1504d9a514fd847307a2a6715959dc5e1 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
   Conbench compare runs links:
   [Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/a1dd2d63922e4648a19b0790597c2a36...98cfc1f372c648eaaa0001542894cc58/)
   [Finished :arrow_down:0.13% :arrow_up:0.0%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/d1e3e80da4f34834a7b00dcf47d80138...20a96a33dad64bf59e6701949b7bfcf1/)
   [Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/e5baa4ab715d407cbe73631ae6bbaeca...64f4025d0fda4bf3960310434303d75f/)
   [Finished :arrow_down:0.21% :arrow_up:0.09%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/bfa12b0e32234f70bfc52b0db8a483da...4fcea6f04602422787e27290f8aaf720/)
   Supported benchmarks:
   ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
   test-mac-arm: Supported benchmark langs: C++, Python, R
   ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
   ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] emkornfield commented on pull request #12296: ARROW-15502: [Java] Detect exceptional footer size in Arrow file reader

Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #12296:
URL: https://github.com/apache/arrow/pull/12296#issuecomment-1030760342


   LGTM, just one question about the API change.  Did this come from the fuzzer?  Have you opened up a seperate PR to add the fuzzing file to our corpus to prevent regressions?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org