You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by GitBox <gi...@apache.org> on 2022/06/09 03:51:48 UTC
[GitHub] [parquet-mr] LuciferYang opened a new pull request, #972: PARQUET-2154: `ParquetFileReader` should close its input stream when `filterRowGroups` throw Exception in constructor
LuciferYang opened a new pull request, #972:
URL: https://github.com/apache/parquet-mr/pull/972
During the construction of `ParquetFileReader`, if `filterRowGroups(footer.getBlocks())` method throws an exception, it will cause resource leak because when the Exception thrown, the open stream `this.f = file.newStream()` looks unable to be closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] LuciferYang commented on pull request #972: PARQUET-2154: `ParquetFileReader` should close its input stream when `filterRowGroups` throw Exception in constructor
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #972:
URL: https://github.com/apache/parquet-mr/pull/972#issuecomment-1152089288
Thanks @sunchao @HyukjinKwon @ggershinsky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] ggershinsky commented on pull request #972: PARQUET-2154: `ParquetFileReader` should close its input stream when `filterRowGroups` throw Exception in constructor
Posted by GitBox <gi...@apache.org>.
ggershinsky commented on PR #972:
URL: https://github.com/apache/parquet-mr/pull/972#issuecomment-1150670630
yep, I've approved the CI run. Once complete, will merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] ggershinsky commented on pull request #972: PARQUET-2154: `ParquetFileReader` should close its input stream when `filterRowGroups` throw Exception in constructor
Posted by GitBox <gi...@apache.org>.
ggershinsky commented on PR #972:
URL: https://github.com/apache/parquet-mr/pull/972#issuecomment-1151344885
can you re-run the CI? (eg via re-opening the PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] LuciferYang commented on a diff in pull request #972: PARQUET-2154: `ParquetFileReader` should close its input stream when `filterRowGroups` throw Exception in constructor
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #972:
URL: https://github.com/apache/parquet-mr/pull/972#discussion_r893047755
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -715,7 +715,14 @@ public ParquetFileReader(
.withDecryption(fileDecryptor.getDecryptionProperties())
.build();
}
- this.blocks = filterRowGroups(blocks);
+ try {
+ this.blocks = filterRowGroups(blocks);
+ } catch (Exception e) {
Review Comment:
There will be more than IOE here, for example, if push down filters for repeated primitive types will throw `IllegalArgumentException` as follows:
```
Caused by: java.lang.IllegalArgumentException: FilterPredicates do not currently support repeated columns. Column keywords is repeated.
at org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumn(SchemaCompatibilityValidator.java:176)
at org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumnFilterPredicate(SchemaCompatibilityValidator.java:149)
at org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:89)
at org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:56)
at org.apache.parquet.filter2.predicate.Operators$NotEq.accept(Operators.java:192)
at org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validate(SchemaCompatibilityValidator.java:61)
at org.apache.parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:95)
at org.apache.parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:45)
at org.apache.parquet.filter2.compat.FilterCompat$FilterPredicateCompat.accept(FilterCompat.java:149)
at org.apache.parquet.filter2.compat.RowGroupFilter.filterRowGroups(RowGroupFilter.java:72)
at org.apache.parquet.hadoop.ParquetFileReader.filterRowGroups(ParquetFileReader.java:870)
at org.apache.parquet.hadoop.ParquetFileReader.<init>(ParquetFileReader.java:789)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] LuciferYang commented on pull request #972: PARQUET-2154: `ParquetFileReader` should close its input stream when `filterRowGroups` throw Exception in constructor
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #972:
URL: https://github.com/apache/parquet-mr/pull/972#issuecomment-1151743122
@ggershinsky yeah～ all passed now～ thanks～
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] LuciferYang closed pull request #972: PARQUET-2154: `ParquetFileReader` should close its input stream when `filterRowGroups` throw Exception in constructor
Posted by GitBox <gi...@apache.org>.
LuciferYang closed pull request #972: PARQUET-2154: `ParquetFileReader` should close its input stream when `filterRowGroups` throw Exception in constructor
URL: https://github.com/apache/parquet-mr/pull/972
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] LuciferYang commented on a diff in pull request #972: PARQUET-2154: `ParquetFileReader` should close its input stream when `filterRowGroups` throw Exception in constructor
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #972:
URL: https://github.com/apache/parquet-mr/pull/972#discussion_r893047755
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -715,7 +715,14 @@ public ParquetFileReader(
.withDecryption(fileDecryptor.getDecryptionProperties())
.build();
}
- this.blocks = filterRowGroups(blocks);
+ try {
+ this.blocks = filterRowGroups(blocks);
+ } catch (Exception e) {
Review Comment:
There will be more than IOE here, for example if push down filters for repeated primitive types will throw `IllegalArgumentException` as follows:
```
Caused by: java.lang.IllegalArgumentException: FilterPredicates do not currently support repeated columns. Column keywords is repeated.
at org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumn(SchemaCompatibilityValidator.java:176)
at org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumnFilterPredicate(SchemaCompatibilityValidator.java:149)
at org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:89)
at org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:56)
at org.apache.parquet.filter2.predicate.Operators$NotEq.accept(Operators.java:192)
at org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validate(SchemaCompatibilityValidator.java:61)
at org.apache.parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:95)
at org.apache.parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:45)
at org.apache.parquet.filter2.compat.FilterCompat$FilterPredicateCompat.accept(FilterCompat.java:149)
at org.apache.parquet.filter2.compat.RowGroupFilter.filterRowGroups(RowGroupFilter.java:72)
at org.apache.parquet.hadoop.ParquetFileReader.filterRowGroups(ParquetFileReader.java:870)
at org.apache.parquet.hadoop.ParquetFileReader.<init>(ParquetFileReader.java:789)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] LuciferYang commented on pull request #972: PARQUET-2154: `ParquetFileReader` should close its input stream when `filterRowGroups` throw Exception in constructor
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #972:
URL: https://github.com/apache/parquet-mr/pull/972#issuecomment-1151345710
> can you re-run the CI? (eg via re-opening the PR)
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] ggershinsky merged pull request #972: PARQUET-2154: `ParquetFileReader` should close its input stream when `filterRowGroups` throw Exception in constructor
Posted by GitBox <gi...@apache.org>.
ggershinsky merged PR #972:
URL: https://github.com/apache/parquet-mr/pull/972
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [parquet-mr] LuciferYang commented on pull request #972: PARQUET-2154: `ParquetFileReader` should close its input stream when `filterRowGroups` throw Exception in constructor
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #972:
URL: https://github.com/apache/parquet-mr/pull/972#issuecomment-1151319153
@ggershinsky I found Travis CI failed, but `Raw log` is null
<img width="1102" alt="image" src="https://user-images.githubusercontent.com/1475305/172891856-e8acc55a-99cd-4023-9e31-e0fac5b9d1e4.png">
What should I do now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org