Re: [PR] Spark, Arrow, Parquet: Add vectorized read support for parquet RLE encoded data pages [iceberg]

["jbewing (via GitHub)" <gi...@apache.org>]

jbewing commented on code in PR #15410:
URL: https://github.com/apache/iceberg/pull/15410#discussion_r2876114511


##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/data/vectorized/parquet/TestParquetVectorizedReads.java:
##########
@@ -430,20 +439,30 @@ public void testSupportedReadsForParquetV2() throws Exception {
   }
 
   @Test
-  public void testUnsupportedReadsForParquetV2() throws Exception {
-    // Some types use delta encoding and which are not supported for vectorized reads
-    Schema schema = new Schema(SUPPORTED_PRIMITIVES.fields());
-    File dataFile = File.createTempFile("junit", null, temp.toFile());
-    assertThat(dataFile.delete()).as("Delete should succeed").isTrue();
-    Iterable<Record> data =
-        generateData(schema, 30000, 0L, RandomData.DEFAULT_NULL_PERCENTAGE, IDENTITY);
-    try (FileAppender<Record> writer = getParquetV2Writer(schema, dataFile)) {
-      writer.addAll(data);
-    }
-    assertThatThrownBy(() -> assertRecordsMatch(schema, 30000, data, dataFile, false, BATCH_SIZE))
+  public void testRLEEncodingOnlySupportsBooleanDataPage() {
+    MessageType schema =
+        new MessageType(
+            "test",
+            primitive(PrimitiveTypeName.INT32, Type.Repetition.OPTIONAL).id(1).named("int_col"));
+    ColumnDescriptor intColumnDesc = schema.getColumnDescription(new String[] {"int_col"});
+    ByteBufferInputStream stream = ByteBufferInputStream.wrap(ByteBuffer.allocate(0));
+
+    String expectedMessage =
+        "Cannot support vectorized reads for column "
+            + intColumnDesc
+            + " with encoding "
+            + Encoding.RLE
+            + ". Disable vectorized reads to read this table/file";
+
+    assertThatThrownBy(
+            () ->
+                new VectorizedPageIterator(intColumnDesc, "parquet-mr", false) {
+                  {
+                    initDataReader(Encoding.RLE, stream, 0);
+                  }
+                })
         .isInstanceOf(UnsupportedOperationException.class)
-        .hasMessageStartingWith("Cannot support vectorized reads for column")
-        .hasMessageEndingWith("Disable vectorized reads to read this table/file");
+        .hasMessage(expectedMessage);

Review Comment:
   I don't think I fully understand why it's brittle here. Isn't the exception message implicitly part of our API interface and, therefore, there should be friction (in the form of a failing test) to changing it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org