You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/01/11 08:42:08 UTC
[GitHub] [spark] LuciferYang opened a new pull request #35163: [SPARK-37864][SQL] Support vectorized read boolean values use RLE encoding with Parquet DataPage V2
LuciferYang opened a new pull request #35163:
URL: https://github.com/apache/spark/pull/35163
### What changes were proposed in this pull request?
Parquet v2 data page write Boolean Values use RLE encoding, when read v2 boolean type values it will throw exceptions as follows now:
```java
Caused by: java.lang.UnsupportedOperationException: Unsupported encoding: RLE
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.getValuesReader(VectorizedColumnReader.java:305) ~[classes/:?]
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.initDataReader(VectorizedColumnReader.java:277) ~[classes/:?]
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readPageV2(VectorizedColumnReader.java:344) ~[classes/:?]
at
```
This PR extends the `readBooleans` and `skipBooleans` of `VectorizedRleValuesReader` to ensure that the above scenario can pass.
### Why are the changes needed?
Support Parquet v2 data page RLE encoding for the vectorized read path
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Add new test case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35163: [WIP][SPARK-37864][SQL] Support vectorized read boolean values use RLE encoding with Parquet DataPage V2
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35163:
URL: https://github.com/apache/spark/pull/35163#discussion_r781895356
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetEncodingSuite.scala
##########
@@ -182,4 +182,40 @@ class ParquetEncodingSuite extends ParquetCompatibilityTest with SharedSparkSess
}
}
}
+
+ test("parquet v2 pages - rle encoding for boolean value columns") {
Review comment:
Without this pr, this case will throw exception:
```
Caused by: java.lang.UnsupportedOperationException: Unsupported encoding: RLE
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.getValuesReader(VectorizedColumnReader.java:304) ~[classes/:?]
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.initDataReader(VectorizedColumnReader.java:276) ~[classes/:?]
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readPageV2(VectorizedColumnReader.java:343) ~[classes/:?]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #35163: [SPARK-37864][SQL] Support vectorized read boolean values use RLE encoding with Parquet DataPage V2
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #35163:
URL: https://github.com/apache/spark/pull/35163#issuecomment-1012334222
Merged to master, thanks @LuciferYang !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #35163: [SPARK-37864][SQL] Support vectorized read boolean values use RLE encoding with Parquet DataPage V2
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #35163:
URL: https://github.com/apache/spark/pull/35163#issuecomment-1012702635
thanks @sunchao
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35163: [SPARK-37864][SQL] Support vectorized read boolean values use RLE encoding with Parquet DataPage V2
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35163:
URL: https://github.com/apache/spark/pull/35163#discussion_r783667686
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -407,7 +426,7 @@ public void skipIntegers(int total) {
@Override
public void skipBooleans(int total) {
- throw new UnsupportedOperationException("only skipIntegers is valid");
+ skipIntegers(total);
Review comment:
OK ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35163: [SPARK-37864][SQL] Support vectorized read boolean values use RLE encoding with Parquet DataPage V2
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35163:
URL: https://github.com/apache/spark/pull/35163#discussion_r783678463
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -407,7 +426,7 @@ public void skipIntegers(int total) {
@Override
public void skipBooleans(int total) {
- throw new UnsupportedOperationException("only skipIntegers is valid");
+ skipIntegers(total);
Review comment:
@sunchao 7a9d4fd fix this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao closed pull request #35163: [SPARK-37864][SQL] Support vectorized read boolean values use RLE encoding with Parquet DataPage V2
Posted by GitBox <gi...@apache.org>.
sunchao closed pull request #35163:
URL: https://github.com/apache/spark/pull/35163
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #35163: [WIP][SPARK-37864][SQL] Support vectorized read boolean values use RLE encoding with Parquet DataPage V2
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #35163:
URL: https://github.com/apache/spark/pull/35163#discussion_r781895356
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetEncodingSuite.scala
##########
@@ -182,4 +182,40 @@ class ParquetEncodingSuite extends ParquetCompatibilityTest with SharedSparkSess
}
}
}
+
+ test("parquet v2 pages - rle encoding for boolean value columns") {
Review comment:
Before this pr, this case will throw exception:
```
Caused by: java.lang.UnsupportedOperationException: Unsupported encoding: RLE
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.getValuesReader(VectorizedColumnReader.java:304) ~[classes/:?]
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.initDataReader(VectorizedColumnReader.java:276) ~[classes/:?]
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readPageV2(VectorizedColumnReader.java:343) ~[classes/:?]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #35163: [SPARK-37864][SQL] Support vectorized read boolean values use RLE encoding with Parquet DataPage V2
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #35163:
URL: https://github.com/apache/spark/pull/35163#discussion_r783657432
##########
File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
##########
@@ -407,7 +426,7 @@ public void skipIntegers(int total) {
@Override
public void skipBooleans(int total) {
- throw new UnsupportedOperationException("only skipIntegers is valid");
+ skipIntegers(total);
Review comment:
nit: maybe create another method like `skipN` and have `skipIntegers` and `skipBooleans` share it? it's a bit confusing when first reading this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org