You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@orc.apache.org by GitBox <gi...@apache.org> on 2022/12/17 12:01:36 UTC
[GitHub] [orc] cxzl25 opened a new pull request, #1340: ORC-1332: Avoid NegativeArraySizeException when using searchArgument
cxzl25 opened a new pull request, #1340:
URL: https://github.com/apache/orc/pull/1340
### What changes were proposed in this pull request?
searchArgument takes effect only when rowIndexStride is greater than 0.
### Why are the changes needed?
When `orc.row.index.stride` is set to a negative number, using searchArgument will throw NegativeArraySizeException.
```java
Caused by: java.lang.NegativeArraySizeException
at org.apache.orc.impl.RecordReaderImpl$SargApplier.pickRowGroups(RecordReaderImpl.java:1164)
at org.apache.orc.impl.RecordReaderImpl.pickRowGroups(RecordReaderImpl.java:1273)
at org.apache.orc.impl.RecordReaderImpl.readStripe(RecordReaderImpl.java:1293)
at org.apache.orc.impl.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:1345)
at org.apache.orc.impl.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1388)
at org.apache.orc.impl.RecordReaderImpl.<init>(RecordReaderImpl.java:367)
```
### How was this patch tested?
add UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] deshanxiao commented on pull request #1340: ORC-1332: Avoid NegativeArraySizeException when using searchArgument
Posted by GitBox <gi...@apache.org>.
deshanxiao commented on PR #1340:
URL: https://github.com/apache/orc/pull/1340#issuecomment-1357027270
CC @dongjoon-hyun @guiyanakuang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] williamhyun closed pull request #1340: ORC-1332: Avoid NegativeArraySizeException when using searchArgument
Posted by GitBox <gi...@apache.org>.
williamhyun closed pull request #1340: ORC-1332: Avoid NegativeArraySizeException when using searchArgument
URL: https://github.com/apache/orc/pull/1340
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] dongjoon-hyun commented on pull request #1340: ORC-1332: Avoid NegativeArraySizeException when using searchArgument
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #1340:
URL: https://github.com/apache/orc/pull/1340#issuecomment-1373986421
I backported this to branch-1.7 too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] cxzl25 commented on a diff in pull request #1340: ORC-1332: Avoid NegativeArraySizeException when using searchArgument
Posted by GitBox <gi...@apache.org>.
cxzl25 commented on code in PR #1340:
URL: https://github.com/apache/orc/pull/1340#discussion_r1051515495
##########
java/core/src/java/org/apache/orc/impl/RecordReaderImpl.java:
##########
@@ -237,7 +237,7 @@ protected RecordReaderImpl(ReaderImpl fileReader,
this.fileIncluded = evolution.getFileIncluded();
SearchArgument sarg = options.getSearchArgument();
boolean[] rowIndexCols = new boolean[evolution.getFileIncluded().length];
- if (sarg != null && rowIndexStride != 0) {
+ if (sarg != null && rowIndexStride > 0) {
Review Comment:
https://github.com/apache/orc/blob/becf5a58107e37f939d5c3d56cb169c182af2ea1/java/core/src/java/org/apache/orc/impl/WriterImpl.java#L186-L188
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] deshanxiao commented on pull request #1340: ORC-1332: Avoid NegativeArraySizeException when using searchArgument
Posted by GitBox <gi...@apache.org>.
deshanxiao commented on PR #1340:
URL: https://github.com/apache/orc/pull/1340#issuecomment-1357026499
Other LGTM.
Thank you @cxzl25 This is a good fix. The main problem is the the inconsistency of reading and writing implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] deshanxiao commented on a diff in pull request #1340: ORC-1332: Avoid NegativeArraySizeException when using searchArgument
Posted by GitBox <gi...@apache.org>.
deshanxiao commented on code in PR #1340:
URL: https://github.com/apache/orc/pull/1340#discussion_r1051755164
##########
java/core/src/test/org/apache/orc/impl/TestRecordReaderImpl.java:
##########
@@ -2571,4 +2573,48 @@ private void testSmallCompressionSizeOrc(int compressionSize) throws IOException
}
}
}
+
+ @Test
+ public void testRowIndexStrideNegativeFilter() throws Exception {
+ Path testFilePath = new Path(workDir, "rowIndexStrideNegative.orc");
+ Configuration conf = new Configuration();
+ FileSystem fs = FileSystem.get(conf);
+ fs.delete(testFilePath, true);
+
+ TypeDescription schema =
+ TypeDescription.fromString("struct<str:string>");
Review Comment:
Use 4-space indentation please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [orc] guiyanakuang commented on pull request #1340: ORC-1332: Avoid NegativeArraySizeException when using searchArgument
Posted by GitBox <gi...@apache.org>.
guiyanakuang commented on PR #1340:
URL: https://github.com/apache/orc/pull/1340#issuecomment-1357098325
Thank you for making a PR.
I think it would be better to also make the writer reset the negative rowIndexStride to 0. This provides a wider range of compatibility. Allowing a negative rowIndexStride to be written to the file footer, then older versions of the ORC reader will still not be read correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org