You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/11/24 12:34:13 UTC
[GitHub] [spark] LuciferYang opened a new pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
LuciferYang opened a new pull request #30484:
URL: https://github.com/apache/spark/pull/30484
### What changes were proposed in this pull request?
There are two places in Spark to call the `SpecificParquetRecordReaderBase.initialize(InputSplit, taskAttemptContext)` method, one is in ParquetFileFormat and the other one is in ParquetPartitionReaderFactory, the "inputSplit.rowGroupOffsets" passed in to the `initialize` method both `null`, it seems that the branch of "rowgroupoffsets! = null" is useless.
So this is pr remove this branch.
### Why are the changes needed?
Remove a unreachable branch to simplify code.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Pass the Jenkins or GitHub Action
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-732991017
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #30484:
URL: https://github.com/apache/spark/pull/30484#discussion_r548350431
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -93,48 +93,17 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
throws IOException, InterruptedException {
Configuration configuration = taskAttemptContext.getConfiguration();
ParquetInputSplit split = (ParquetInputSplit)inputSplit;
+ Preconditions.checkState(split.getRowGroupOffsets() == null,
Review comment:
Why don't you just simply use `assert`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #30484:
URL: https://github.com/apache/spark/pull/30484#discussion_r548368844
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -93,48 +93,17 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
throws IOException, InterruptedException {
Configuration configuration = taskAttemptContext.getConfiguration();
ParquetInputSplit split = (ParquetInputSplit)inputSplit;
+ Preconditions.checkState(split.getRowGroupOffsets() == null,
Review comment:
https://github.com/apache/spark/blob/61881bb6988aa0320b4bacfabbc0ee6f05f287cb/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetPartitionReaderFactory.scala#L126-L134
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-733637332
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #30484:
URL: https://github.com/apache/spark/pull/30484#discussion_r548374525
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -93,48 +93,17 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
throws IOException, InterruptedException {
Configuration configuration = taskAttemptContext.getConfiguration();
ParquetInputSplit split = (ParquetInputSplit)inputSplit;
+ Preconditions.checkState(split.getRowGroupOffsets() == null,
+ "rowGroupOffsets in ParquetInputSplit should be null.");
this.file = split.getPath();
- long[] rowGroupOffsets = split.getRowGroupOffsets();
-
- ParquetMetadata footer;
- List<BlockMetaData> blocks;
-
- // if task.side.metadata is set, rowGroupOffsets is null
- if (rowGroupOffsets == null) {
Review comment:
After rethinking @LuciferYang, actually this is aimed to be ported to Apache Parquet, see the comments above:
```
* TODO: move this to the parquet-mr project. There are performance benefits of doing it
* this way, albeit at a higher cost to implement. This base class is reusable.
```
It's being tracked at [PARQUET-131](https://issues.apache.org/jira/browse/PARQUET-131). Shall we just simply add a comment that this code is not used in Apache Spark for now?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #30484: [SPARK-33532][SQL] Add comments to a unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #30484:
URL: https://github.com/apache/spark/pull/30484
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-734041648
@HyukjinKwon Is it appropriate for us to clean up this code branch now, or do it together when upgrade Apache Parquet version
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-749102020
**[Test build #133175 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133175/testReport)** for PR 30484 at commit [`d210aa4`](https://github.com/apache/spark/commit/d210aa4cb25359b6cadf6766fee2c47499a6c0aa).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-749548530
The 2nd key change from SPARK-14535, this pr remove `buildInternalScan` from `FileFormat`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #30484:
URL: https://github.com/apache/spark/pull/30484#discussion_r548367035
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -93,48 +93,17 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
throws IOException, InterruptedException {
Configuration configuration = taskAttemptContext.getConfiguration();
ParquetInputSplit split = (ParquetInputSplit)inputSplit;
+ Preconditions.checkState(split.getRowGroupOffsets() == null,
Review comment:
For the current spark code, I think `split.getRowGroupOffsets()` should always be null
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-750303471
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133284/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-751555143
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38010/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-732991017
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #30484: [SPARK-33532][SQL] Add comments to a unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-751563129
Thx ~ @HyukjinKwon @srowen
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-750014968
**[Test build #133284 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133284/testReport)** for PR 30484 at commit [`2a57f30`](https://github.com/apache/spark/commit/2a57f3094ae33306b1e7cfcd1919ec4dab98da03).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-749544350
The first key change from SPARK-13883, ParquetRelation.buildReader method always use `null` to construct `ParquetInputSplit` and pass it to UnsafeRowParquetRecordReader.init method.
https://github.com/apache/spark/pull/11709/files#diff-ae45608e2a6173960880aceaeedac157bc9fb30d0a7a6259c152cf33ce6d9fa7R350-R365

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #30484:
URL: https://github.com/apache/spark/pull/30484#discussion_r548368844
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -93,48 +93,17 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
throws IOException, InterruptedException {
Configuration configuration = taskAttemptContext.getConfiguration();
ParquetInputSplit split = (ParquetInputSplit)inputSplit;
+ Preconditions.checkState(split.getRowGroupOffsets() == null,
Review comment:
https://github.com/apache/spark/blob/61881bb6988aa0320b4bacfabbc0ee6f05f287cb/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetPartitionReaderFactory.scala#L117-L134
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30484: [SPARK-33532][SQL] Add comments to a unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-751595025
**[Test build #133419 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133419/testReport)** for PR 30484 at commit [`bd793cf`](https://github.com/apache/spark/commit/bd793cffab3e7df5ce87240fc25dab3ff47b1f79).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-750303471
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133284/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #30484: [SPARK-33532][SQL] Add comments to a unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #30484:
URL: https://github.com/apache/spark/pull/30484#discussion_r549198012
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -93,48 +93,17 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
throws IOException, InterruptedException {
Configuration configuration = taskAttemptContext.getConfiguration();
ParquetInputSplit split = (ParquetInputSplit)inputSplit;
+ Preconditions.checkState(split.getRowGroupOffsets() == null,
+ "rowGroupOffsets in ParquetInputSplit should be null.");
this.file = split.getPath();
- long[] rowGroupOffsets = split.getRowGroupOffsets();
-
- ParquetMetadata footer;
- List<BlockMetaData> blocks;
-
- // if task.side.metadata is set, rowGroupOffsets is null
- if (rowGroupOffsets == null) {
Review comment:
done ~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #30484:
URL: https://github.com/apache/spark/pull/30484#discussion_r548367035
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -93,48 +93,17 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
throws IOException, InterruptedException {
Configuration configuration = taskAttemptContext.getConfiguration();
ParquetInputSplit split = (ParquetInputSplit)inputSplit;
+ Preconditions.checkState(split.getRowGroupOffsets() == null,
Review comment:
For the current spark code, I think `split.getRowGroupOffsets()` should always be null because after SPARK-13883 and SPARK-13989 `ParquetInputSplit` is created manually in `ParquetRelation(ParquetFileFormat) ` and set `rowGroupOffsets` to null explicitly
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30484: [SPARK-33532][SQL] Add comments to a unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-751548572
**[Test build #133418 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133418/testReport)** for PR 30484 at commit [`afa1bf0`](https://github.com/apache/spark/commit/afa1bf0aa13cc63e94831a6f0e7ed773328ba461).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30484: [SPARK-33532][SQL] Add comments to a unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-751599773
**[Test build #133418 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133418/testReport)** for PR 30484 at commit [`afa1bf0`](https://github.com/apache/spark/commit/afa1bf0aa13cc63e94831a6f0e7ed773328ba461).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30484: [SPARK-33532][SQL] Add comments to a unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-751603760
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #30484:
URL: https://github.com/apache/spark/pull/30484#discussion_r548398249
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -93,48 +93,17 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
throws IOException, InterruptedException {
Configuration configuration = taskAttemptContext.getConfiguration();
ParquetInputSplit split = (ParquetInputSplit)inputSplit;
+ Preconditions.checkState(split.getRowGroupOffsets() == null,
+ "rowGroupOffsets in ParquetInputSplit should be null.");
this.file = split.getPath();
- long[] rowGroupOffsets = split.getRowGroupOffsets();
-
- ParquetMetadata footer;
- List<BlockMetaData> blocks;
-
- // if task.side.metadata is set, rowGroupOffsets is null
- if (rowGroupOffsets == null) {
Review comment:
Add a comment and revert the code change?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-749372549
@LuciferYang I am very sorry but do you mind pointing out which commit added that codes and removed the usages? It would be much easier to review with that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-733572799
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-733510794
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #30484:
URL: https://github.com/apache/spark/pull/30484#discussion_r548368655
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -93,48 +93,17 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
throws IOException, InterruptedException {
Configuration configuration = taskAttemptContext.getConfiguration();
ParquetInputSplit split = (ParquetInputSplit)inputSplit;
+ Preconditions.checkState(split.getRowGroupOffsets() == null,
Review comment:
https://github.com/apache/spark/blob/61881bb6988aa0320b4bacfabbc0ee6f05f287cb/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala#L255-L267
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-751548572
**[Test build #133418 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133418/testReport)** for PR 30484 at commit [`afa1bf0`](https://github.com/apache/spark/commit/afa1bf0aa13cc63e94831a6f0e7ed773328ba461).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-750058973
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37881/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-733509871
**[Test build #131724 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131724/testReport)** for PR 30484 at commit [`6cbd321`](https://github.com/apache/spark/commit/6cbd32172abf384c5d8a05d09b256fcc78e6e33f).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-750034367
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37881/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-732960198
**[Test build #131659 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131659/testReport)** for PR 30484 at commit [`83b85d4`](https://github.com/apache/spark/commit/83b85d4c1101a39e0b85b41d54862badf4247ad5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-733467380
**[Test build #131733 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131733/testReport)** for PR 30484 at commit [`a0859e7`](https://github.com/apache/spark/commit/a0859e75e68d0fd248e39a648cdd715c8356a132).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-733467380
**[Test build #131733 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131733/testReport)** for PR 30484 at commit [`a0859e7`](https://github.com/apache/spark/commit/a0859e75e68d0fd248e39a648cdd715c8356a132).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-749155896
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37774/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-749233626
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133175/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-749155896
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37774/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #30484:
URL: https://github.com/apache/spark/pull/30484#discussion_r547818852
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -94,47 +92,14 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
Configuration configuration = taskAttemptContext.getConfiguration();
ParquetInputSplit split = (ParquetInputSplit)inputSplit;
this.file = split.getPath();
- long[] rowGroupOffsets = split.getRowGroupOffsets();
-
- ParquetMetadata footer;
- List<BlockMetaData> blocks;
-
- // if task.side.metadata is set, rowGroupOffsets is null
- if (rowGroupOffsets == null) {
Review comment:
2a57f30 add a assert use `Preconditions.checkState`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-749542175
The branch to be deleted in current pr is from SPARK-11787,.
In SPARK-11787`SqlNewHadoopPartition` create `UnsafeRowParquetRecordReader` when `spark.parquet.enableUnsafeRowRecordReader` is true and data format is `org.apache.parquet.hadoop.ParquetInputFormat`
The key point is the `ParquetInputSplit` pass to `UnsafeRowParquetRecordReader.init` method, if `rowGroupOffsets` in `ParquetInputSplit` is not null will enter the code branch which I want to delete in this pr.
`ParquetInputSplit` produce by `SqlNewHadoopRDD#getPartitions` use `ParquetInputFormat.getSplits` method.
https://github.com/apache/parquet-mr/blob/32c46643845ea8a705c35d4ec8fc654cc8ff816d/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetInputFormat.java#L287-L306
The `else branch` will create a `ParquetInputSplit` and `rowGroupOffsets` not null, further calls are as follows:
```
ParquetInputFormat#getSplits(JobContext)
-> ParquetInputFormat#getSplits(Configuration,List<Footer>)
-> ClientSideMetadataSplitStrategy#getSplits
-> ClientSideMetadataSplitStrategy#generateSplits
-> ClientSideMetadataSplitStrategy.SplitInfo#getParquetInputSplit
```
https://github.com/apache/parquet-mr/blob/32c46643845ea8a705c35d4ec8fc654cc8ff816d/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetInputFormat.java#L635-L649
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30484: [SPARK-33532][SQL] Add comments to a unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-751548590
**[Test build #133419 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133419/testReport)** for PR 30484 at commit [`bd793cf`](https://github.com/apache/spark/commit/bd793cffab3e7df5ce87240fc25dab3ff47b1f79).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-749567216
At least for testing purposes here, you could add a check that it isn't null. Then at least tests would tell us whether that is true in tests. We may or may not leave in the check as an assert or something.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-749542175
The branch to be deleted in current pr is from SPARK-11787,.
In SPARK-11787`SqlNewHadoopPartition` create `UnsafeRowParquetRecordReader` when `spark.parquet.enableUnsafeRowRecordReader` is true and data format is `org.apache.parquet.hadoop.ParquetInputFormat`
The key point is the `ParquetInputSplit` pass to `UnsafeRowParquetRecordReader.init` method, if `rowGroupOffsets` in `ParquetInputSplit` is not null will enter the code branch which I want to delete in this pr.
`ParquetInputSplit` produce by `SqlNewHadoopRDD#getPartitions` use `ParquetInputFormat.getSplits` method.
https://github.com/apache/parquet-mr/blob/32c46643845ea8a705c35d4ec8fc654cc8ff816d/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetInputFormat.java#L287-L306

The `else branch` will create a `ParquetInputSplit` and `rowGroupOffsets` not null, further calls are as follows:
```
ParquetInputFormat#getSplits(JobContext)
-> ParquetInputFormat#getSplits(Configuration,List<Footer>)
-> ClientSideMetadataSplitStrategy#getSplits
-> ClientSideMetadataSplitStrategy#generateSplits
-> ClientSideMetadataSplitStrategy.SplitInfo#getParquetInputSplit
```
https://github.com/apache/parquet-mr/blob/32c46643845ea8a705c35d4ec8fc654cc8ff816d/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetInputFormat.java#L635-L649

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-733433369
**[Test build #131724 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131724/testReport)** for PR 30484 at commit [`6cbd321`](https://github.com/apache/spark/commit/6cbd32172abf384c5d8a05d09b256fcc78e6e33f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #30484:
URL: https://github.com/apache/spark/pull/30484#discussion_r548808635
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -93,48 +93,17 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
throws IOException, InterruptedException {
Configuration configuration = taskAttemptContext.getConfiguration();
ParquetInputSplit split = (ParquetInputSplit)inputSplit;
+ Preconditions.checkState(split.getRowGroupOffsets() == null,
+ "rowGroupOffsets in ParquetInputSplit should be null.");
this.file = split.getPath();
- long[] rowGroupOffsets = split.getRowGroupOffsets();
-
- ParquetMetadata footer;
- List<BlockMetaData> blocks;
-
- // if task.side.metadata is set, rowGroupOffsets is null
- if (rowGroupOffsets == null) {
Review comment:
But PARQUET-131 seems to have made no progress in the last three years ... , anyone really do it ?
I'm just worried about whether this TODO will really be completed :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #30484:
URL: https://github.com/apache/spark/pull/30484#discussion_r547605001
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -94,47 +92,14 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
Configuration configuration = taskAttemptContext.getConfiguration();
ParquetInputSplit split = (ParquetInputSplit)inputSplit;
this.file = split.getPath();
- long[] rowGroupOffsets = split.getRowGroupOffsets();
-
- ParquetMetadata footer;
- List<BlockMetaData> blocks;
-
- // if task.side.metadata is set, rowGroupOffsets is null
- if (rowGroupOffsets == null) {
Review comment:
OK~ Will do it later. a little busy at the moment
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #30484:
URL: https://github.com/apache/spark/pull/30484#discussion_r548355931
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -93,48 +93,17 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
throws IOException, InterruptedException {
Configuration configuration = taskAttemptContext.getConfiguration();
ParquetInputSplit split = (ParquetInputSplit)inputSplit;
+ Preconditions.checkState(split.getRowGroupOffsets() == null,
Review comment:
`assert` in Java code depends on `-ea` flag to enable, we can't guarantee users execute spark apps with this flag, or do we just want to ensure this `assert` works in UTs?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-750300523
**[Test build #133284 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133284/testReport)** for PR 30484 at commit [`2a57f30`](https://github.com/apache/spark/commit/2a57f3094ae33306b1e7cfcd1919ec4dab98da03).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #30484: [SPARK-33532][SQL] Add comments to a unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-751579950
> I manually tested via ./dev/lint-scala.
>
> Merged to master.
@HyukjinKwon thx ~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #30484:
URL: https://github.com/apache/spark/pull/30484#discussion_r548808635
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -93,48 +93,17 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
throws IOException, InterruptedException {
Configuration configuration = taskAttemptContext.getConfiguration();
ParquetInputSplit split = (ParquetInputSplit)inputSplit;
+ Preconditions.checkState(split.getRowGroupOffsets() == null,
+ "rowGroupOffsets in ParquetInputSplit should be null.");
this.file = split.getPath();
- long[] rowGroupOffsets = split.getRowGroupOffsets();
-
- ParquetMetadata footer;
- List<BlockMetaData> blocks;
-
- // if task.side.metadata is set, rowGroupOffsets is null
- if (rowGroupOffsets == null) {
Review comment:
@HyukjinKwon But PARQUET-131 seems to have made no progress in the last three years ... , anyone really do it ?
I'm just worried about whether this TODO will really be completed, haha :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-751548590
**[Test build #133419 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133419/testReport)** for PR 30484 at commit [`bd793cf`](https://github.com/apache/spark/commit/bd793cffab3e7df5ce87240fc25dab3ff47b1f79).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-733230956
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-749542175
The branch to be deleted in current pr is from SPARK-11787,.
In SPARK-11787`SqlNewHadoopPartition` create `UnsafeRowParquetRecordReader` when `spark.parquet.enableUnsafeRowRecordReader` is true and data format is `org.apache.parquet.hadoop.ParquetInputFormat`
The key point is the `ParquetInputSplit` pass to `UnsafeRowParquetRecordReader.init` method, if `rowGroupOffsets` in `ParquetInputSplit` is not null will enter the code branch which I want to delete in this pr.
`ParquetInputSplit` produce by `SqlNewHadoopRDD#getPartitions` use `ParquetInputFormat.getSplits` method.
https://github.com/apache/parquet-mr/blob/32c46643845ea8a705c35d4ec8fc654cc8ff816d/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetInputFormat.java#L287-L306
The `else branch` will create a `ParquetInputSplit` and `rowGroupOffsets` not null, further calls are as follows:
```
ParquetInputFormat#getSplits(JobContext)
-> ParquetInputFormat#getSplits(Configuration,List<Footer>)
-> ClientSideMetadataSplitStrategy#getSplits
-> ClientSideMetadataSplitStrategy#generateSplits
-> ClientSideMetadataSplitStrategy.SplitInfo#getParquetInputSplit
```
https://github.com/apache/parquet-mr/blob/32c46643845ea8a705c35d4ec8fc654cc8ff816d/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetInputFormat.java#L635-L649
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30484: [SPARK-33532][SQL] Add comments to a unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-751562239
Thanks @LuciferYang. LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-751555826
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38009/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-750014968
**[Test build #133284 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133284/testReport)** for PR 30484 at commit [`2a57f30`](https://github.com/apache/spark/commit/2a57f3094ae33306b1e7cfcd1919ec4dab98da03).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30484: [SPARK-33532][SQL] Add comments to a unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-751559186
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38010/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-733571617
**[Test build #131733 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131733/testReport)** for PR 30484 at commit [`a0859e7`](https://github.com/apache/spark/commit/a0859e75e68d0fd248e39a648cdd715c8356a132).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-733510794
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-733230956
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30484: [SPARK-33532][SQL] Add comments to a unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-751559200
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38010/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-749220625
**[Test build #133175 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133175/testReport)** for PR 30484 at commit [`d210aa4`](https://github.com/apache/spark/commit/d210aa4cb25359b6cadf6766fee2c47499a6c0aa).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30484: [SPARK-33532][SQL] Add comments to a unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-751603760
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-733433369
**[Test build #131724 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131724/testReport)** for PR 30484 at commit [`6cbd321`](https://github.com/apache/spark/commit/6cbd32172abf384c5d8a05d09b256fcc78e6e33f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #30484:
URL: https://github.com/apache/spark/pull/30484#discussion_r548363983
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -93,48 +93,17 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
throws IOException, InterruptedException {
Configuration configuration = taskAttemptContext.getConfiguration();
ParquetInputSplit split = (ParquetInputSplit)inputSplit;
+ Preconditions.checkState(split.getRowGroupOffsets() == null,
Review comment:
Either is reasonable here. If it should never be true in any execution of Spark, we assert it. If user input or state can cause it to not be true, we need something besides an assert. Most of the point is verifying this at test time, yes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-749233626
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133175/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-750059047
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37881/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30484: [SPARK-33532][SQL] Add comments to a unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-751565687
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38009/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-750059047
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37881/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-733567809
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-749102020
**[Test build #133175 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133175/testReport)** for PR 30484 at commit [`d210aa4`](https://github.com/apache/spark/commit/d210aa4cb25359b6cadf6766fee2c47499a6c0aa).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #30484:
URL: https://github.com/apache/spark/pull/30484#discussion_r547339218
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -94,47 +92,14 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
Configuration configuration = taskAttemptContext.getConfiguration();
ParquetInputSplit split = (ParquetInputSplit)inputSplit;
this.file = split.getPath();
- long[] rowGroupOffsets = split.getRowGroupOffsets();
-
- ParquetMetadata footer;
- List<BlockMetaData> blocks;
-
- // if task.side.metadata is set, rowGroupOffsets is null
- if (rowGroupOffsets == null) {
Review comment:
Yeah, let's add an assert. That sounds like a good idea.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-732982229
**[Test build #131659 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131659/testReport)** for PR 30484 at commit [`83b85d4`](https://github.com/apache/spark/commit/83b85d4c1101a39e0b85b41d54862badf4247ad5).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-749380105
> @LuciferYang I am very sorry but do you mind pointing out which commit added that codes and removed the usages? It would be much easier to review with that.
@HyukjinKwon OK, let me investigate ~ I think it's a very interesting thing ~ haha ~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #30484:
URL: https://github.com/apache/spark/pull/30484#discussion_r548861926
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -93,48 +93,17 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
throws IOException, InterruptedException {
Configuration configuration = taskAttemptContext.getConfiguration();
ParquetInputSplit split = (ParquetInputSplit)inputSplit;
+ Preconditions.checkState(split.getRowGroupOffsets() == null,
+ "rowGroupOffsets in ParquetInputSplit should be null.");
this.file = split.getPath();
- long[] rowGroupOffsets = split.getRowGroupOffsets();
-
- ParquetMetadata footer;
- List<BlockMetaData> blocks;
-
- // if task.side.metadata is set, rowGroupOffsets is null
- if (rowGroupOffsets == null) {
Review comment:
Yeah, just simply adding a comment because this is planned to move to Parquet. It's stuck but I think we should port it in long run anyway.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #30484:
URL: https://github.com/apache/spark/pull/30484#discussion_r548808635
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -93,48 +93,17 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
throws IOException, InterruptedException {
Configuration configuration = taskAttemptContext.getConfiguration();
ParquetInputSplit split = (ParquetInputSplit)inputSplit;
+ Preconditions.checkState(split.getRowGroupOffsets() == null,
+ "rowGroupOffsets in ParquetInputSplit should be null.");
this.file = split.getPath();
- long[] rowGroupOffsets = split.getRowGroupOffsets();
-
- ParquetMetadata footer;
- List<BlockMetaData> blocks;
-
- // if task.side.metadata is set, rowGroupOffsets is null
- if (rowGroupOffsets == null) {
Review comment:
@HyukjinKwon But PARQUET-131 seems to have made no progress in the last three years ... , anyone really do it ?
I'm just worried about whether this TODO will really be completed :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #30484:
URL: https://github.com/apache/spark/pull/30484#discussion_r548355931
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -93,48 +93,17 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
throws IOException, InterruptedException {
Configuration configuration = taskAttemptContext.getConfiguration();
ParquetInputSplit split = (ParquetInputSplit)inputSplit;
+ Preconditions.checkState(split.getRowGroupOffsets() == null,
Review comment:
The assert of Java code depends on `-ea` flag to activate, we can't guarantee users execute spark apps with this flag, or do we just want to ensure this `assert` works in UTs?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30484: [SPARK-33532][SQL] Add comments to a unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-751559200
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-733572799
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #30484:
URL: https://github.com/apache/spark/pull/30484#discussion_r548355931
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -93,48 +93,17 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
throws IOException, InterruptedException {
Configuration configuration = taskAttemptContext.getConfiguration();
ParquetInputSplit split = (ParquetInputSplit)inputSplit;
+ Preconditions.checkState(split.getRowGroupOffsets() == null,
Review comment:
`assert` in Java code depends on `-ea` flag to activate, we can't guarantee users execute spark apps with this flag, or do we just want to ensure this `assert` works in UTs?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-732960198
**[Test build #131659 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131659/testReport)** for PR 30484 at commit [`83b85d4`](https://github.com/apache/spark/commit/83b85d4c1101a39e0b85b41d54862badf4247ad5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30484: [SPARK-33532][SQL] Add comments to a unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-751561603
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38009/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang edited a comment on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-749548530
Then SPARK-13989 remove `VectorizedParquetRecordReader` create logical from `SqlNewHadoopRDD`, SPARK-14535 remove `buildInternalScan` from `FileFormat` and SPARK-14596 remove `SqlNewHadoopRDD` from spark code.
After these changes, `VectorizedParquetRecordReader` only create by `parquet.DefaultSource#buildReader` method and the `rowGroupOffsets` in `ParquetInputSplit` always `null`.
So I think `rowGroupOffsets != null` branch in `SpecificParquetRecordReaderBase.init` method is unreachable now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #30484:
URL: https://github.com/apache/spark/pull/30484#discussion_r549185376
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java
##########
@@ -93,48 +93,17 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont
throws IOException, InterruptedException {
Configuration configuration = taskAttemptContext.getConfiguration();
ParquetInputSplit split = (ParquetInputSplit)inputSplit;
+ Preconditions.checkState(split.getRowGroupOffsets() == null,
+ "rowGroupOffsets in ParquetInputSplit should be null.");
this.file = split.getPath();
- long[] rowGroupOffsets = split.getRowGroupOffsets();
-
- ParquetMetadata footer;
- List<BlockMetaData> blocks;
-
- // if task.side.metadata is set, rowGroupOffsets is null
- if (rowGroupOffsets == null) {
Review comment:
OK ~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30484: [SPARK-33532][SQL] Add comments to a unreachable branch in SpecificParquetRecordReaderBase.initialize method
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30484:
URL: https://github.com/apache/spark/pull/30484#issuecomment-751576803
I manually tested via `./dev/lint-scala`.
Merged to master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org