You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2026/07/03 18:42:04 UTC
[PR] Simplify ColumnReader SPI: drop iterator API, replace type checks with getValueType() [pinot]
Jackie-Jiang opened a new pull request, #18918:
URL: https://github.com/apache/pinot/pull/18918
## Summary
Cleans up the `ColumnReader` SPI (`pinot-spi`), used for column-major segment building, by removing a
redundant second read API and simplifying type introspection.
`ColumnReader` previously exposed two overlapping ways to read a column:
- an iterator-style API — `hasNext()`, `next()`, `isNextNull()`, `skipNext()`, `rewind()`, and the
`nextXxx()` / `nextXxxMV()` type-specific methods, and
- random access by document id — `getXxx(docId)` / `getValue(docId)`.
Every production consumer can be expressed with random access alone, so this removes the iterator API
entirely. Both column-major consumers — `SegmentColumnarIndexCreator.indexColumn(String, ColumnReader)`
and `ColumnarSegmentPreIndexStatsContainer.collectColumn(...)` — now read via a
`for (docId = 0; docId < getTotalDocs(); docId++)` loop over `getValue(docId)`, preserving the previous
null semantics.
It also replaces the boolean type-check methods (`isSingleValue()` + `isInt()`…`isBytes()`) with a single
`@Nullable PinotDataType getValueType()` that encodes both value type and cardinality (e.g. `INT` vs
`INT_ARRAY`), returning `null` when the column has no directly-readable accessor (BOOLEAN / TIMESTAMP /
JSON — read those via `getValue(docId)`). A static `ColumnReader.toValueType(DataType, boolean)` helper
centralizes the mapping for implementations backed by a `DataType`, and `DataTypeColumnTransformer.isNoOp()`
collapses to `getValueType() == destType`.
All implementations (`PinotSegmentColumnReaderImpl`, `DefaultValueColumnReader`, `ArrowColumnReader`,
`BatchedArrowColumnReader`) and their tests are migrated; the interface members are reordered and the
Javadoc rewritten for clarity.
## Backward incompatibility
`ColumnReader` is a `pinot-spi` interface first shipped in 1.5.0. Removing methods from it is
binary-incompatible for any external plugin that implements or calls `ColumnReader`. It is a
recently-added, internal-facing SPI for the still-evolving column-major segment build, so the expected
blast radius is nil, but the PR is labeled `backward-incompat` accordingly.
Release note: the `ColumnReader` SPI no longer exposes the sequential/iterator read API
(`hasNext`/`next`/`nextXxx`/`nextXxxMV`/`rewind`/`skipNext`/`isNextNull`) or the boolean type-check methods
(`isSingleValue`/`isInt`/…/`isBytes`). Read column values by document id via `getValue(docId)` /
`getXxx(docId)`, and use `getValueType()` to obtain the directly-readable value type and cardinality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Simplify ColumnReader SPI: drop iterator API, replace type checks with getValueType() [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #18918:
URL: https://github.com/apache/pinot/pull/18918#discussion_r3522381176
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReader.java:
##########
@@ -23,300 +23,110 @@
import java.io.Serializable;
import java.math.BigDecimal;
import javax.annotation.Nullable;
-
-
-/**
- * The <code>ColumnReader</code> interface is used to read column data from various data sources
- * for columnar segment building. Unlike RecordReader which reads row-by-row, ColumnReader provides
- * column-wise access to data, enabling efficient columnar segment creation.
- *
- * <p>This interface provides 3 patterns optimised for different use cases
- * (Some implementations may not support all patterns):
- * <ul>
- * <li>Sequential iteration over all values in a column using hasNext(), next() and rewind()</li>
- * <li>Sequential iteration with type-specific methods (isInt(), isLong(), nextInt(), nextLong(), etc.)
- * and null handling (isNextNull(), skipNext()) and supports hasNext() and rewind()</li>
- * <li>Random access by document ID using getInt(docId), getLong(docId), etc. and isNull(docId) for null checks</li>
- * </ul>
- *
- * <p>Implementations should handle data type conversions and efficient column-wise data access patterns.
- *
- * <h2>Usage Patterns</h2>
- *
- * <p>There are three primary patterns for reading data from a ColumnReader:
- *
- * <h3>Pattern 1: Sequential Iteration with Generic next() and Null Checks</h3>
- * <p>This pattern uses the generic {@link #next()} method which returns Object and may return null.
- * Suitable when you need to handle arbitrary data types or when null handling is done on the return value.
- *
- * <pre>{@code
- * // Read all values in the column
- * while (columnReader.hasNext()) {
- * Object value;
- * try {
- * value = columnReader.next();
- * } catch (Exception e) {
- * // Handle exception / log
- * continue;
- * }
- * if (value != null) {
- * // Process non-null value
- * processValue(value);
- * } else {
- * // Handle null value
- * handleNullValue();
- * }
- * }
- *
- * // Rewind to read the column again
- * columnReader.rewind();
- *
- * // Second pass through the data
- * while (columnReader.hasNext()) {
- * Object value = columnReader.next();
- * if (value != null) {
- * processValueAgain(value);
- * }
- * }
- * }</pre>
- *
- * <h3>Pattern 2: Sequential Iteration with Type-Specific Methods and Explicit Null Checks</h3>
- * <p>This pattern uses {@link #isNextNull()} to check for nulls before calling type-specific methods
- * like {@link #nextInt()}, {@link #nextLong()}, etc. Use {@link #skipNext()} to advance past null values.
- * This is the preferred pattern when you know the column data type and want to avoid boxing overhead.
- * Before using this pattern, check the column data type using methods like {@link #isInt()}, {@link #isLong()}, etc.
- * If the data type does not match, fall back to Pattern 1 with {@link #next()}.
- * <pre>{@code
- * // Read all int values in the column, handling nulls
- * if (columnReader.isInt()) {
- * while (columnReader.hasNext()) {
- * if (columnReader.isNextNull()) {
- * // Skip the null value
- * columnReader.skipNext();
- * handleNullValue();
- * } else {
- * // Read the primitive int value (no boxing)
- * int value = columnReader.nextInt();
- * processIntValue(value);
- * }
- * }
- * } else {
- * // Fallback to Pattern 1 if not INT type
- * }
- *
- * }</pre>
- *
- * <h3>Pattern 3: Random Access by Document ID</h3>
- * <p>This pattern uses {@link #getTotalDocs()} to get the total number of documents, then uses
- * document ID-based accessors like {@link #getInt(int)}, {@link #getLong(int)}, etc. to read
- * specific values. Use {@link #isNull(int)} to check if a value is null before reading.
- * This pattern is useful when you need random access or want to process documents in a specific order.
- *
- * <pre>{@code
- *
- * // Random access example - read specific document IDs
- * int[] docIdsToRead = {5, 10, 15, 20};
- * for (int docId : docIdsToRead) {
- * if (!columnReader.isNull(docId)) {
- * int value = columnReader.getInt(docId);
- * processSpecificDoc(docId, value);
- * }
- * }
- *
- * // Read in reverse order
- * // Get the total number of documents
- * int totalDocs = columnReader.getTotalDocs();
- * for (int docId = totalDocs - 1; docId >= 0; docId--) {
- * if (!columnReader.isNull(docId)) {
- * int value = columnReader.getInt(docId);
- * processReverseOrder(docId, value);
- * }
- * }
- * }</pre>
- *
- * <h3>Choosing the Right Pattern</h3>
- * <ul>
- * <li><b>Pattern 1</b>: Use when dealing with generic Object types or when you don't know
- * the column type at compile time. Less efficient due to boxing.</li>
- * <li><b>Pattern 2</b>: Use when you know the column type and want efficient sequential iteration
- * with primitive types. Preferred for most columnar segment building scenarios.</li>
- * <li><b>Pattern 3</b>: Use when you need random access, want to process documents in a specific
- * order, or need to access the same document multiple times.</li>
- * </ul>
- *
- */
+import org.apache.pinot.spi.data.FieldSpec.DataType;
+import org.apache.pinot.spi.utils.PinotDataType;
+
+
+/// Reads the values of a single column by document ID, for columnar segment building. Where a `RecordReader` exposes a
+/// data source row-by-row, a `ColumnReader` exposes one column at a time: a caller reads every value of a column before
+/// moving to the next one. Building a segment one fully-materialized column at a time is cheaper than transposing row
+/// records when the source is already columnar (e.g. Arrow, Parquet).
+///
+/// Documents are addressed by a 0-based id, from 0 (inclusive) to [#getTotalDocs()] (exclusive). To read a column:
+/// 1. Call [#getValueType()] once. A non-`null` result names the type that can be read directly through the matching
+/// type-specific accessor; `null` means the column must be read through the boxed [#getValue(int)].
+/// 2. For each document id, call [#isNull(int)] first — type-specific accessors cannot represent null and must not be
+/// called for a null value.
+/// 3. Read each non-null value with the accessor chosen in step 1, e.g. [#getInt(int)] / [#getIntMV(int)] when
+/// [#getValueType()] is [PinotDataType#INT] / [PinotDataType#INT_ARRAY], or [#getValue(int)] otherwise.
+///
+/// Implementations perform any conversion between the source encoding and the returned Pinot type, are not required to
+/// be thread-safe, and must be released with [#close()] when no longer needed.
+///
+/// ```
+/// PinotDataType valueType = columnReader.getValueType();
+/// for (int docId = 0; docId < columnReader.getTotalDocs(); docId++) {
+/// if (columnReader.isNull(docId)) {
+/// continue;
+/// }
+/// if (valueType == PinotDataType.INT) {
+/// consumeInt(columnReader.getInt(docId));
+/// } else {
+/// consumeObject(columnReader.getValue(docId));
+/// }
+/// }
+/// ```
public interface ColumnReader extends Closeable, Serializable {
- /**
- * Return <code>true</code> if more values remain to be read in this column.
- * <p>This method should not throw exception. Caller is not responsible for handling exceptions from this method.
- */
- boolean hasNext();
+ /// Returns the name of the column read by this reader.
+ String getColumnName();
- /**
- * Get the next value in the column.
- * <p>This method should be called only if {@link #hasNext()} returns <code>true</code>. Caller is responsible for
- * handling exceptions from this method and skip the value if user wants to continue reading the remaining values.
- *
- * @return Next column value, or null if the value is null
- * @throws IOException If an I/O error occurs while reading
- */
+ /// Returns the [PinotDataType] whose type-specific accessors can read this column directly, or `null` if the column
+ /// has no such accessor and must be read through [#getValue(int)].
+ ///
+ /// The returned type encodes cardinality: a single-value column returns a scalar type read through the scalar
+ /// accessor (e.g. [PinotDataType#INT] via [#getInt(int)]), while a multi-value column returns the array type read
+ /// through the multi-value accessor (e.g. [PinotDataType#INT_ARRAY] via [#getIntMV(int)]). Columns whose stored type
+ /// has no dedicated accessor — such as BOOLEAN, TIMESTAMP, or JSON — return `null`.
Review Comment:
Resolved by reworking the SPI contract: `getValueType()` returns `T` only when `getValue()` returns a `T`-typed value (the logical type). So `BOOLEAN` → `null` (read via `getValue()` as a `Boolean`), `TIMESTAMP` → `null` (a `Timestamp`), and `JSON` → `STRING`. The Arrow reader now derives the value type from the column's logical type and returns `Boolean` from `getValue()` (matching the row-major `ArrowRecordExtractor`) instead of mapping `Bool` → `INT`.
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/columntransformer/DataTypeColumnTransformerTest.java:
##########
@@ -211,7 +153,7 @@ public void testIsNoOpForMatchingIntMVTypes() {
.build();
FieldSpec fieldSpec = schema.getFieldSpecFor(COLUMN_NAME);
TableConfig tableConfig = new TableConfigBuilder(TableType.OFFLINE).setTableName("testTable").build();
- ColumnReader reader = mockColumnReader().multiValue().asInt().build();
+ ColumnReader reader = mockReader(PinotDataType.INT_ARRAY);
DataTypeColumnTransformer transformer = new DataTypeColumnTransformer(tableConfig, fieldSpec, reader);
Review Comment:
Added `FLOAT_ARRAY` and `DOUBLE_ARRAY` multi-value `isNoOp` cases to `DataTypeColumnTransformerTest`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Simplify ColumnReader SPI: drop iterator API, replace type checks with getValueType() [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #18918:
URL: https://github.com/apache/pinot/pull/18918#discussion_r3521495532
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReaderImplTest.java:
##########
@@ -185,30 +167,30 @@ private Schema getSchema(IndexType indexType) {
}
}
- /**
- * Data provider for single-value accessor methods.
- * @return test parameters: column name, random access getter, sequential getter,
- * value converter function (required for float), index type
- */
+ /// Data provider for single-value accessor methods.
+ ///
+ /// @return test parameters: column name, random access getter, value converter function (required for float), index
+ /// type
@DataProvider(name = "singleValueAccessorProvider")
public Object[][] singleValueAccessorProvider() {
// Base column configurations
- Object[][] baseConfigs = new Object[][] {
- {INT_COL_1, (SingleValueGetter) PinotSegmentColumnReaderImpl::getInt,
- (SingleValueSequentialGetter) PinotSegmentColumnReaderImpl::nextInt, null},
- {LONG_COL, (SingleValueGetter) PinotSegmentColumnReaderImpl::getLong,
- (SingleValueSequentialGetter) PinotSegmentColumnReaderImpl::nextLong, null},
- {FLOAT_COL, (SingleValueGetter) PinotSegmentColumnReaderImpl::getFloat,
- (SingleValueSequentialGetter) PinotSegmentColumnReaderImpl::nextFloat,
- (Function<Object, Object>) val -> val instanceof Double ? ((Double) val).floatValue() : val},
- {DOUBLE_COL, (SingleValueGetter) PinotSegmentColumnReaderImpl::getDouble,
- (SingleValueSequentialGetter) PinotSegmentColumnReaderImpl::nextDouble, null},
- {BIG_DECIMAL_COL, (SingleValueGetter) PinotSegmentColumnReaderImpl::getBigDecimal,
- (SingleValueSequentialGetter) PinotSegmentColumnReaderImpl::nextBigDecimal, null},
- {STRING_COL_1, (SingleValueGetter) PinotSegmentColumnReaderImpl::getString,
- (SingleValueSequentialGetter) PinotSegmentColumnReaderImpl::nextString, null},
- {BYTES_COL, (SingleValueGetter) PinotSegmentColumnReaderImpl::getBytes,
- (SingleValueSequentialGetter) PinotSegmentColumnReaderImpl::nextBytes, null},
+ Object[][] baseConfigs = new Object[][]{
+ {INT_COL_1, (SingleValueGetter) PinotSegmentColumnReaderImpl::getInt, null}, {
+ LONG_COL,
+ (SingleValueGetter) PinotSegmentColumnReaderImpl::getLong, null
+ }, {
+ FLOAT_COL, (SingleValueGetter) PinotSegmentColumnReaderImpl::getFloat, (Function<Object, Object>) val ->
+ val instanceof Double ? ((Double) val).floatValue() : val
+ }, {DOUBLE_COL, (SingleValueGetter) PinotSegmentColumnReaderImpl::getDouble, null}, {
+ BIG_DECIMAL_COL,
+ (SingleValueGetter) PinotSegmentColumnReaderImpl::getBigDecimal, null
+ }, {
+ STRING_COL_1,
+ (SingleValueGetter) PinotSegmentColumnReaderImpl::getString, null
+ }, {
+ BYTES_COL,
+ (SingleValueGetter) PinotSegmentColumnReaderImpl::getBytes, null
+ },
};
Review Comment:
The `baseConfigs` array initializer formatting/indentation here is inconsistent with nearby test style and is likely to trip Checkstyle (indentation + array init alignment). Reformat each entry onto its own line with consistent indentation.
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/readers/DefaultValueColumnReaderTest.java:
##########
@@ -308,20 +231,20 @@ public void testMultiValueFloatColumn() {
FieldSpec fieldSpec = new DimensionFieldSpec(COLUMN_NAME, FieldSpec.DataType.FLOAT, false);
DefaultValueColumnReader reader = new DefaultValueColumnReader(COLUMN_NAME, NUM_DOCS, fieldSpec);
- // Test sequential access with nextFloatMV
+ // Test random access
float expectedValue = ((Number) fieldSpec.getDefaultNullValue()).floatValue();
float[] expectedArray = new float[]{expectedValue};
- for (int i = 0; i < NUM_DOCS; i++) {
- MultiValueResult<float[]> mvResult = reader.nextFloatMV();
- Assert.assertFalse(mvResult.hasNulls());
- Assert.assertTrue(Arrays.equals(mvResult.getValues(), expectedArray));
+ for (int docId = 0; docId < reader.getTotalDocs(); docId++) {
+ MultiValueResult<float[]> mvResult = reader.getFloatMV(docId);
+ assertFalse(mvResult.hasNulls());
+ assertEquals(mvResult.getValues(), expectedArray);
}
- // Test random access and array reuse
- reader.rewind();
+ // Test array reuse
+ // Test array reuse
Review Comment:
Duplicate comment line (`// Test array reuse`) can be collapsed to a single line to avoid noise in the test.
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/readers/DefaultValueColumnReaderTest.java:
##########
@@ -19,24 +19,26 @@
package org.apache.pinot.segment.local.segment.readers;
import java.math.BigDecimal;
-import java.util.Arrays;
import org.apache.pinot.spi.data.DimensionFieldSpec;
import org.apache.pinot.spi.data.FieldSpec;
import org.apache.pinot.spi.data.MetricFieldSpec;
import org.apache.pinot.spi.data.readers.MultiValueResult;
-import org.testng.Assert;
+import org.apache.pinot.spi.utils.PinotDataType;
import org.testng.annotations.Test;
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertSame;
+import static org.testng.Assert.assertTrue;
+
/// Comprehensive tests for [DefaultValueColumnReader].
///
/// This test validates:
/// - Single-value fields for all data types (INT, LONG, FLOAT, DOUBLE, BIG_DECIMAL, STRING, BYTES)
/// - Multi-value fields for all data types
-/// - Sequential access methods (next*, hasNext, skipNext, isNextNull)
/// - Random access methods (get*, isNull)
/// - Type indicator methods (isInt, isLong, isBigDecimal, etc.)
Review Comment:
The class-level test description still claims it validates the removed `isInt()/isLong()/...` type-indicator methods. This should be updated to `getValueType()` to reflect the new SPI.
##########
pinot-plugins/pinot-input-format/pinot-arrow/src/main/java/org/apache/pinot/plugin/inputformat/arrow/ArrowColumnReader.java:
##########
@@ -35,60 +35,47 @@
import org.apache.arrow.vector.complex.ListVector;
import org.apache.pinot.spi.data.readers.ColumnReader;
import org.apache.pinot.spi.data.readers.MultiValueResult;
-
-
-/**
- * Column reader for Apache Arrow {@link FieldVector}.
- *
- * <p>Wraps a single Arrow {@link FieldVector} and exposes sequential and random-access
- * read patterns conforming to the {@link ColumnReader} contract. The vector is owned by
- * the enclosing {@link ArrowColumnReaderFactory} which is responsible for its lifecycle;
- * closing this reader is a no-op for the underlying vector.
- *
- * <p>Supported Arrow types map to Pinot's stored types as follows:
- * <ul>
- * <li>{@code IntVector} / {@code BitVector} → {@code INT} (BitVector promoted to 0/1)</li>
- * <li>{@code BigIntVector} → {@code LONG}</li>
- * <li>{@code Float4Vector} → {@code FLOAT}</li>
- * <li>{@code Float8Vector} → {@code DOUBLE}</li>
- * <li>{@code DecimalVector} → {@code BIG_DECIMAL}</li>
- * <li>{@code VarCharVector} → {@code STRING}</li>
- * <li>{@code VarBinaryVector} → {@code BYTES}</li>
- * <li>{@code ListVector} of the above → multi-value variant</li>
- * </ul>
- *
- * <p>The list above applies to the typed primitive accessors only ({@link #getInt},
- * {@link #getString}, {@link #getIntMV}, ...). Complex types (Map, Struct, Union, ...)
- * and temporal types are still readable via the generic {@link #getValue(int)} /
- * {@link #next()} accessors, which delegate to {@link ArrowToPinotTypeConverter} and
- * return the same canonical JDK types as the row-major path ({@link
- * ArrowRecordExtractor}) — e.g. {@code Map<String, Object>} for Struct / Map,
- * {@code Object[]} for List variants, {@code LocalDate} / {@code LocalTime} /
- * {@code Timestamp} for temporal types.
- *
- * <p><b>BitVector caveat:</b> Arrow's {@link FieldVector#getObject} returns a {@code Boolean}
- * for a {@link BitVector}, but Pinot stores booleans as {@code INT} (0/1). The generic
- * {@link #getValue(int)} therefore special-cases {@code BitVector} to return an {@code Integer}
- * 0/1, keeping it consistent with {@link #getInt(int)} and the advertised INT mapping above
- * rather than surfacing the shared converter's {@code Boolean}.
- *
- * <p>{@code @SuppressWarnings("serial")}: {@link ColumnReader} is {@link java.io.Serializable} by SPI
- * contract, but this reader holds non-serializable Arrow state and is never serialized — it lives
- * only for the duration of a columnar segment build.
- *
- * <p>This class is not thread-safe.
- */
-@SuppressWarnings("serial")
+import org.apache.pinot.spi.utils.PinotDataType;
+
+
+/// Column reader for an Apache Arrow [FieldVector].
+///
+/// Wraps a single Arrow [FieldVector] and exposes random-access reads conforming to the [ColumnReader] contract. The
+/// vector is owned by the enclosing [ArrowColumnReaderFactory], which is responsible for its lifecycle; closing this
+/// reader is a no-op for the underlying vector.
+///
+/// Supported Arrow types map to Pinot's stored types as follows:
+/// - `IntVector` / `BitVector` → `INT` (BitVector promoted to 0/1)
+/// - `BigIntVector` → `LONG`
+/// - `Float4Vector` → `FLOAT`
+/// - `Float8Vector` → `DOUBLE`
+/// - `DecimalVector` → `BIG_DECIMAL`
+/// - `VarCharVector` → `STRING`
+/// - `VarBinaryVector` → `BYTES`
+/// - `ListVector` of the above → multi-value variant
+///
+/// The list above applies to the typed primitive accessors only ([#getInt(int)], [#getString(int)], [#getIntMV(int)],
+/// ...). Complex types (Map, Struct, Union, ...) and temporal types are still readable via the generic
+/// [#getValue(int)] accessor, which delegates to [ArrowToPinotTypeConverter] and returns the same canonical JDK types
+/// as the row-major path ([ArrowRecordExtractor]) — e.g. `Map<String, Object>` for Struct / Map, `Object[]` for List
+/// variants, `LocalDate` / `LocalTime` / `Timestamp` for temporal types.
+///
+/// **BitVector caveat:** Arrow's [FieldVector#getObject(int)] returns a `Boolean` for a `BitVector`, but Pinot stores
+/// booleans as `INT` (0/1). The generic [#getValue(int)] therefore special-cases `BitVector` to return an `Integer`
+/// 0/1, keeping it consistent with [#getInt(int)] and the advertised INT mapping above rather than surfacing the
+/// shared converter's `Boolean`.
+///
+/// `@SuppressWarnings("serial")`: [ColumnReader] is [java.io.Serializable] by SPI contract, but this reader holds
+/// non-serializable Arrow state and is never serialized — it lives only for the duration of a columnar segment build.
+///
+/// This class is not thread-safe.
public class ArrowColumnReader implements ColumnReader {
Review Comment:
The Javadoc mentions `@SuppressWarnings("serial")`, but the annotation was removed. Either add the annotation back or drop the mention; adding it back keeps the intent clear and avoids serializable warnings.
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReaderImplTest.java:
##########
@@ -273,166 +242,87 @@ public void testSingleValueAccessors(String columnName, SingleValueGetter random
if (expectedValue == null) {
// Null values are replaced with default during segment creation (non-nullable)
if (actualValue instanceof byte[]) {
- Assert.assertEquals(actualValue, defaultValue, "Null should be replaced with default at docId " + docId);
+ assertEquals(actualValue, defaultValue, "Null should be replaced with default at docId " + docId);
} else if (actualValue instanceof Double) {
- Assert.assertEquals((Double) actualValue, (Double) defaultValue, 0.0001,
+ assertEquals((Double) actualValue, (Double) defaultValue, 0.0001,
"Null should be replaced with default at docId " + docId);
} else if (actualValue instanceof Float) {
- Assert.assertEquals((Float) actualValue, (Float) defaultValue, 0.0001f,
+ assertEquals((Float) actualValue, (Float) defaultValue, 0.0001f,
"Null should be replaced with default at docId " + docId);
} else if (actualValue instanceof BigDecimal) {
- Assert.assertEquals(((BigDecimal) actualValue).compareTo((BigDecimal) defaultValue), 0,
+ assertEquals(((BigDecimal) actualValue).compareTo((BigDecimal) defaultValue), 0,
"Null should be replaced with default at docId " + docId);
} else {
- Assert.assertEquals(actualValue, defaultValue, "Null should be replaced with default at docId " + docId);
+ assertEquals(actualValue, defaultValue, "Null should be replaced with default at docId " + docId);
}
} else {
// Convert expected value if converter is provided
Object convertedExpectedValue = valueConverter != null ? valueConverter.apply(expectedValue) : expectedValue;
if (actualValue instanceof Double) {
- Assert.assertEquals((Double) actualValue, (Double) convertedExpectedValue, 0.0001,
+ assertEquals((Double) actualValue, (Double) convertedExpectedValue, 0.0001,
"Value mismatch at docId " + docId);
} else if (actualValue instanceof Float) {
- Assert.assertEquals((Float) actualValue, (Float) convertedExpectedValue, 0.0001f,
+ assertEquals((Float) actualValue, (Float) convertedExpectedValue, 0.0001f,
"Value mismatch at docId " + docId);
} else if (actualValue instanceof BigDecimal) {
- Assert.assertEquals(((BigDecimal) actualValue).compareTo((BigDecimal) convertedExpectedValue), 0,
+ assertEquals(((BigDecimal) actualValue).compareTo((BigDecimal) convertedExpectedValue), 0,
"Value mismatch at docId " + docId);
} else {
- Assert.assertEquals(actualValue, convertedExpectedValue, "Value mismatch at docId " + docId);
- }
- }
- }
-
- // Test sequential iteration with type-specific methods (Pattern 2) - only for columns with sequential getters
- if (sequentialGetter != null) {
- reader.rewind();
- int docId = 0;
- while (reader.hasNext()) {
- GenericRow expectedRow = _testData.get(docId);
- Object expectedValue = expectedRow.getValue(columnName);
-
- if (indexType == IndexType.NULLABLE && expectedValue == null) {
- Assert.assertTrue(reader.isNextNull(),
- "isNextNull() should return true for null value at docId " + docId);
- reader.skipNext();
- } else if (reader.isNextNull()) {
- reader.skipNext();
- // For non-nullable segments, null is replaced with default
- Assert.assertNull(expectedValue, "Expected null value at docId " + docId);
- } else {
- Object actualValue = sequentialGetter.get(reader);
-
- if (expectedValue == null) {
- // Null values are replaced with default during segment creation (non-nullable)
- if (actualValue instanceof byte[]) {
- Assert.assertEquals(actualValue, defaultValue,
- "Null should be replaced with default at docId " + docId + " (sequential)");
- } else if (actualValue instanceof Double) {
- Assert.assertEquals((Double) actualValue, (Double) defaultValue, 0.0001,
- "Null should be replaced with default at docId " + docId + " (sequential)");
- } else if (actualValue instanceof Float) {
- Assert.assertEquals((Float) actualValue, (Float) defaultValue, 0.0001f,
- "Null should be replaced with default at docId " + docId + " (sequential)");
- } else if (actualValue instanceof BigDecimal) {
- Assert.assertEquals(((BigDecimal) actualValue).compareTo((BigDecimal) defaultValue), 0,
- "Null should be replaced with default at docId " + docId + " (sequential)");
- } else {
- Assert.assertEquals(actualValue, defaultValue,
- "Null should be replaced with default at docId " + docId + " (sequential)");
- }
- } else {
- // Convert expected value if converter is provided
- Object convertedValue = valueConverter != null ? valueConverter.apply(expectedValue) : expectedValue;
- if (actualValue instanceof Double) {
- Assert.assertEquals((Double) actualValue, (Double) convertedValue, 0.0001,
- "Value mismatch at docId " + docId + " (sequential)");
- } else if (actualValue instanceof Float) {
- Assert.assertEquals((Float) actualValue, (Float) convertedValue, 0.0001f,
- "Value mismatch at docId " + docId + " (sequential)");
- } else if (actualValue instanceof BigDecimal) {
- Assert.assertEquals(((BigDecimal) actualValue).compareTo((BigDecimal) convertedValue), 0,
- "Value mismatch at docId " + docId + " (sequential)");
- } else {
- Assert.assertEquals(actualValue, convertedValue,
- "Value mismatch at docId " + docId + " (sequential)");
- }
- }
+ assertEquals(actualValue, convertedExpectedValue, "Value mismatch at docId " + docId);
}
- docId++;
}
- Assert.assertEquals(docId, totalDocs, "Should have iterated through all documents");
}
} finally {
reader.close();
}
}
-
- /**
- * Data provider for multi-value accessor methods.
- * @return test parameters: column name, random access getter function,
- * sequential getter function, array converter function, index type
- */
+ /// Data provider for multi-value accessor methods.
+ ///
+ /// @return test parameters: column name, random access getter function, array converter function, index type
@DataProvider(name = "multiValueAccessorProvider")
public Object[][] multiValueAccessorProvider() {
// Base column configurations
// For primitive MV types, we need to extract .getValues() from MultiValueResult
// and verify that hasNulls() is false (nulls are removed by NullValueTransformer for MV primitive types)
- Object[][] baseConfigs = new Object[][] {
- {MV_INT_COL, (MultiValueGetter) (reader, docId) -> {
+ Object[][] baseConfigs = new Object[][]{
+ {
+ MV_INT_COL, (MultiValueGetter) (reader, docId) -> {
MultiValueResult<int[]> result = reader.getIntMV(docId);
- Assert.assertFalse(result.hasNulls(), "Multi-value primitive types should not have nulls");
+ assertFalse(result.hasNulls(), "Multi-value primitive types should not have nulls");
Review Comment:
The `multiValueAccessorProvider()` `baseConfigs` initializer has inconsistent indentation/bracing (entries start mid-line, mismatched indent levels inside lambdas). This is hard to read and is likely to violate Checkstyle indentation rules; please reformat this block similar to other data providers in this module (one entry per line, consistent 2-space indentation).
##########
pinot-plugins/pinot-input-format/pinot-arrow/src/main/java/org/apache/pinot/plugin/inputformat/arrow/BatchedArrowColumnReader.java:
##########
@@ -27,38 +27,30 @@
import org.apache.arrow.vector.types.pojo.Field;
import org.apache.pinot.spi.data.readers.ColumnReader;
import org.apache.pinot.spi.data.readers.MultiValueResult;
-
-
-/**
- * A {@link ColumnReader} over one column of a {@link BatchedArrowFileSource}. Reads are served by
- * delegating to a per-batch {@link ArrowColumnReader} over the current record batch's column vector;
- * the source loads at most one batch at a time, so peak resident memory is one batch (see
- * {@link BatchedArrowFileSource}).
- *
- * <p>Random access ({@code getXxx(docId)}) seeks to the owning batch and reads it; sequential access
- * ({@code next} / typed {@code nextXxx}) is built on it. Type predicates are derived from the column's
- * (decoded) Arrow {@link Field}, so they need no batch load. The column-major segment build consumes
- * each column to completion sequentially (rewind + next), one column at a time.
- *
- * <p>The per-batch delegate is invalidated when the shared source moves to another batch (whether
- * this reader advanced it or a different one did); {@link #delegate(int)} documents why and rebuilds
- * it. Interleaving readers therefore stays correct (it just re-loads batches); the build never
- * interleaves.
- *
- * <p>This class is not thread-safe. {@code @SuppressWarnings("serial")}: {@link ColumnReader} is
- * {@link java.io.Serializable} by SPI contract, but this reader holds non-serializable Arrow state
- * and is never serialized.
- */
-@SuppressWarnings("serial")
+import org.apache.pinot.spi.utils.PinotDataType;
+
+
+/// A [ColumnReader] over one column of a [BatchedArrowFileSource]. Reads are served by delegating to a per-batch
+/// [ArrowColumnReader] over the current record batch's column vector; the source loads at most one batch at a time, so
+/// peak resident memory is one batch (see [BatchedArrowFileSource]).
+///
+/// Random access (`getXxx(docId)`) seeks to the owning batch and reads it. The value type is derived from the column's
+/// (decoded) Arrow [Field], so it needs no batch load. The column-major segment build consumes each column to
+/// completion (by ascending docId), one column at a time.
+///
+/// The per-batch delegate is invalidated when the shared source moves to another batch (whether this reader advanced
+/// it or a different one did); [#delegate(int)] documents why and rebuilds it. Interleaving readers therefore stays
+/// correct (it just re-loads batches); the build never interleaves.
+///
+/// This class is not thread-safe. `@SuppressWarnings("serial")`: [ColumnReader] is [java.io.Serializable] by SPI
+/// contract, but this reader holds non-serializable Arrow state and is never serialized.
public class BatchedArrowColumnReader implements ColumnReader {
Review Comment:
The class Javadoc says `@SuppressWarnings("serial")`, but the annotation is no longer present. Add it back (or remove the mention) so the documentation matches the code and serializable warnings stay suppressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Simplify ColumnReader SPI: drop iterator API, replace type checks with getValueType() [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #18918:
URL: https://github.com/apache/pinot/pull/18918#discussion_r3532298210
##########
pinot-plugins/pinot-input-format/pinot-arrow/src/main/java/org/apache/pinot/plugin/inputformat/arrow/ArrowToPinotTypeConverter.java:
##########
@@ -59,6 +61,40 @@ public final class ArrowToPinotTypeConverter {
private ArrowToPinotTypeConverter() {
}
+ /// Returns the [PinotDataType] that a column with the given Arrow element [Types.MinorType] can be read as through a
+ /// type-specific `ColumnReader` accessor, or `null` when it has no dedicated accessor and must be read through
+ /// `ColumnReader.getValue()`. `singleValue` selects the scalar vs `_ARRAY` variant.
+ ///
+ /// The `MinorType` already encodes every distinction the accessors care about, so a plain switch suffices: only
+ /// signed 32/64-bit ints (`INT` / `BIGINT`), single/double floats, 128-bit `DECIMAL`, `VARCHAR`, and `VARBINARY`
+ /// have accessors. Everything else — unsigned/narrow ints, half floats, `DECIMAL256`, `LARGEVARCHAR` /
+ /// `LARGEVARBINARY` / fixed-size binary, `BIT` (boolean), temporal, and complex types — is a distinct `MinorType`
+ /// that falls through to `null` and is read via `getValue()`.
Review Comment:
The doc comment refers to `ColumnReader.getValue()` / `getValue()` without parameters, but the SPI method is `getValue(int)`. Using the correct signature avoids confusion for implementers/callers reading this helper’s contract.
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/readers/InMemoryColumnReader.java:
##########
@@ -202,6 +84,11 @@ public boolean isNull(int docId) {
return _isNull[docId];
}
+ @Override
+ public Object getValue(int docId) {
+ return _values[docId];
+ }
Review Comment:
`ColumnReader#getValue(int)` is documented to return an `Object[]` for multi-value columns, but this mock currently returns the raw stored value (e.g. an `int[]`), which breaks the SPI contract and can make tests accidentally depend on implementation details. Consider boxing primitive arrays when `_isSingleValue` is false so MV `getValue()` matches the interface contract.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Simplify ColumnReader SPI: drop iterator API, replace type checks with getValueType() [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #18918:
URL: https://github.com/apache/pinot/pull/18918#discussion_r3532947072
##########
pinot-plugins/pinot-input-format/pinot-arrow/src/main/java/org/apache/pinot/plugin/inputformat/arrow/ArrowToPinotTypeConverter.java:
##########
@@ -59,6 +61,40 @@ public final class ArrowToPinotTypeConverter {
private ArrowToPinotTypeConverter() {
}
+ /// Returns the [PinotDataType] that a column with the given Arrow element [Types.MinorType] can be read as through a
+ /// type-specific `ColumnReader` accessor, or `null` when it has no dedicated accessor and must be read through
+ /// `ColumnReader.getValue()`. `singleValue` selects the scalar vs `_ARRAY` variant.
+ ///
+ /// The `MinorType` already encodes every distinction the accessors care about, so a plain switch suffices: only
+ /// signed 32/64-bit ints (`INT` / `BIGINT`), single/double floats, 128-bit `DECIMAL`, `VARCHAR`, and `VARBINARY`
+ /// have accessors. Everything else — unsigned/narrow ints, half floats, `DECIMAL256`, `LARGEVARCHAR` /
+ /// `LARGEVARBINARY` / fixed-size binary, `BIT` (boolean), temporal, and complex types — is a distinct `MinorType`
+ /// that falls through to `null` and is read via `getValue()`.
Review Comment:
Fixed — the doc now refers to `getValue(int)` in both places.
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/readers/InMemoryColumnReader.java:
##########
@@ -202,6 +84,11 @@ public boolean isNull(int docId) {
return _isNull[docId];
}
+ @Override
+ public Object getValue(int docId) {
+ return _values[docId];
+ }
Review Comment:
Fixed. `getValue(int)` now boxes primitive multi-value arrays (`int[]` / `long[]` / …) to `Object[]` per the SPI contract, matching what the real readers (e.g. `ArrowColumnReader`) return for list columns. Single values, `null`, and values that are already `Object[]` (`BigDecimal[]` / `String[]` / `byte[][]`) pass through unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Simplify ColumnReader SPI: drop iterator API, replace type checks with getValueType() [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #18918:
URL: https://github.com/apache/pinot/pull/18918#discussion_r3521745901
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReaderImplTest.java:
##########
@@ -185,30 +167,30 @@ private Schema getSchema(IndexType indexType) {
}
}
- /**
- * Data provider for single-value accessor methods.
- * @return test parameters: column name, random access getter, sequential getter,
- * value converter function (required for float), index type
- */
+ /// Data provider for single-value accessor methods.
+ ///
+ /// @return test parameters: column name, random access getter, value converter function (required for float), index
+ /// type
@DataProvider(name = "singleValueAccessorProvider")
public Object[][] singleValueAccessorProvider() {
// Base column configurations
- Object[][] baseConfigs = new Object[][] {
- {INT_COL_1, (SingleValueGetter) PinotSegmentColumnReaderImpl::getInt,
- (SingleValueSequentialGetter) PinotSegmentColumnReaderImpl::nextInt, null},
- {LONG_COL, (SingleValueGetter) PinotSegmentColumnReaderImpl::getLong,
- (SingleValueSequentialGetter) PinotSegmentColumnReaderImpl::nextLong, null},
- {FLOAT_COL, (SingleValueGetter) PinotSegmentColumnReaderImpl::getFloat,
- (SingleValueSequentialGetter) PinotSegmentColumnReaderImpl::nextFloat,
- (Function<Object, Object>) val -> val instanceof Double ? ((Double) val).floatValue() : val},
- {DOUBLE_COL, (SingleValueGetter) PinotSegmentColumnReaderImpl::getDouble,
- (SingleValueSequentialGetter) PinotSegmentColumnReaderImpl::nextDouble, null},
- {BIG_DECIMAL_COL, (SingleValueGetter) PinotSegmentColumnReaderImpl::getBigDecimal,
- (SingleValueSequentialGetter) PinotSegmentColumnReaderImpl::nextBigDecimal, null},
- {STRING_COL_1, (SingleValueGetter) PinotSegmentColumnReaderImpl::getString,
- (SingleValueSequentialGetter) PinotSegmentColumnReaderImpl::nextString, null},
- {BYTES_COL, (SingleValueGetter) PinotSegmentColumnReaderImpl::getBytes,
- (SingleValueSequentialGetter) PinotSegmentColumnReaderImpl::nextBytes, null},
+ Object[][] baseConfigs = new Object[][]{
+ {INT_COL_1, (SingleValueGetter) PinotSegmentColumnReaderImpl::getInt, null}, {
+ LONG_COL,
+ (SingleValueGetter) PinotSegmentColumnReaderImpl::getLong, null
+ }, {
+ FLOAT_COL, (SingleValueGetter) PinotSegmentColumnReaderImpl::getFloat, (Function<Object, Object>) val ->
+ val instanceof Double ? ((Double) val).floatValue() : val
+ }, {DOUBLE_COL, (SingleValueGetter) PinotSegmentColumnReaderImpl::getDouble, null}, {
+ BIG_DECIMAL_COL,
+ (SingleValueGetter) PinotSegmentColumnReaderImpl::getBigDecimal, null
+ }, {
+ STRING_COL_1,
+ (SingleValueGetter) PinotSegmentColumnReaderImpl::getString, null
+ }, {
+ BYTES_COL,
+ (SingleValueGetter) PinotSegmentColumnReaderImpl::getBytes, null
+ },
};
Review Comment:
Reformatting deferred to `spotless:apply` (pre-commit). The multi-line rows only carry per-row converter/getter lambdas; the simple rows stay one-line, at consistent 12-space indentation. Not hand-editing to avoid diverging from spotless's canonical output.
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReaderImplTest.java:
##########
@@ -273,166 +242,87 @@ public void testSingleValueAccessors(String columnName, SingleValueGetter random
if (expectedValue == null) {
// Null values are replaced with default during segment creation (non-nullable)
if (actualValue instanceof byte[]) {
- Assert.assertEquals(actualValue, defaultValue, "Null should be replaced with default at docId " + docId);
+ assertEquals(actualValue, defaultValue, "Null should be replaced with default at docId " + docId);
} else if (actualValue instanceof Double) {
- Assert.assertEquals((Double) actualValue, (Double) defaultValue, 0.0001,
+ assertEquals((Double) actualValue, (Double) defaultValue, 0.0001,
"Null should be replaced with default at docId " + docId);
} else if (actualValue instanceof Float) {
- Assert.assertEquals((Float) actualValue, (Float) defaultValue, 0.0001f,
+ assertEquals((Float) actualValue, (Float) defaultValue, 0.0001f,
"Null should be replaced with default at docId " + docId);
} else if (actualValue instanceof BigDecimal) {
- Assert.assertEquals(((BigDecimal) actualValue).compareTo((BigDecimal) defaultValue), 0,
+ assertEquals(((BigDecimal) actualValue).compareTo((BigDecimal) defaultValue), 0,
"Null should be replaced with default at docId " + docId);
} else {
- Assert.assertEquals(actualValue, defaultValue, "Null should be replaced with default at docId " + docId);
+ assertEquals(actualValue, defaultValue, "Null should be replaced with default at docId " + docId);
}
} else {
// Convert expected value if converter is provided
Object convertedExpectedValue = valueConverter != null ? valueConverter.apply(expectedValue) : expectedValue;
if (actualValue instanceof Double) {
- Assert.assertEquals((Double) actualValue, (Double) convertedExpectedValue, 0.0001,
+ assertEquals((Double) actualValue, (Double) convertedExpectedValue, 0.0001,
"Value mismatch at docId " + docId);
} else if (actualValue instanceof Float) {
- Assert.assertEquals((Float) actualValue, (Float) convertedExpectedValue, 0.0001f,
+ assertEquals((Float) actualValue, (Float) convertedExpectedValue, 0.0001f,
"Value mismatch at docId " + docId);
} else if (actualValue instanceof BigDecimal) {
- Assert.assertEquals(((BigDecimal) actualValue).compareTo((BigDecimal) convertedExpectedValue), 0,
+ assertEquals(((BigDecimal) actualValue).compareTo((BigDecimal) convertedExpectedValue), 0,
"Value mismatch at docId " + docId);
} else {
- Assert.assertEquals(actualValue, convertedExpectedValue, "Value mismatch at docId " + docId);
- }
- }
- }
-
- // Test sequential iteration with type-specific methods (Pattern 2) - only for columns with sequential getters
- if (sequentialGetter != null) {
- reader.rewind();
- int docId = 0;
- while (reader.hasNext()) {
- GenericRow expectedRow = _testData.get(docId);
- Object expectedValue = expectedRow.getValue(columnName);
-
- if (indexType == IndexType.NULLABLE && expectedValue == null) {
- Assert.assertTrue(reader.isNextNull(),
- "isNextNull() should return true for null value at docId " + docId);
- reader.skipNext();
- } else if (reader.isNextNull()) {
- reader.skipNext();
- // For non-nullable segments, null is replaced with default
- Assert.assertNull(expectedValue, "Expected null value at docId " + docId);
- } else {
- Object actualValue = sequentialGetter.get(reader);
-
- if (expectedValue == null) {
- // Null values are replaced with default during segment creation (non-nullable)
- if (actualValue instanceof byte[]) {
- Assert.assertEquals(actualValue, defaultValue,
- "Null should be replaced with default at docId " + docId + " (sequential)");
- } else if (actualValue instanceof Double) {
- Assert.assertEquals((Double) actualValue, (Double) defaultValue, 0.0001,
- "Null should be replaced with default at docId " + docId + " (sequential)");
- } else if (actualValue instanceof Float) {
- Assert.assertEquals((Float) actualValue, (Float) defaultValue, 0.0001f,
- "Null should be replaced with default at docId " + docId + " (sequential)");
- } else if (actualValue instanceof BigDecimal) {
- Assert.assertEquals(((BigDecimal) actualValue).compareTo((BigDecimal) defaultValue), 0,
- "Null should be replaced with default at docId " + docId + " (sequential)");
- } else {
- Assert.assertEquals(actualValue, defaultValue,
- "Null should be replaced with default at docId " + docId + " (sequential)");
- }
- } else {
- // Convert expected value if converter is provided
- Object convertedValue = valueConverter != null ? valueConverter.apply(expectedValue) : expectedValue;
- if (actualValue instanceof Double) {
- Assert.assertEquals((Double) actualValue, (Double) convertedValue, 0.0001,
- "Value mismatch at docId " + docId + " (sequential)");
- } else if (actualValue instanceof Float) {
- Assert.assertEquals((Float) actualValue, (Float) convertedValue, 0.0001f,
- "Value mismatch at docId " + docId + " (sequential)");
- } else if (actualValue instanceof BigDecimal) {
- Assert.assertEquals(((BigDecimal) actualValue).compareTo((BigDecimal) convertedValue), 0,
- "Value mismatch at docId " + docId + " (sequential)");
- } else {
- Assert.assertEquals(actualValue, convertedValue,
- "Value mismatch at docId " + docId + " (sequential)");
- }
- }
+ assertEquals(actualValue, convertedExpectedValue, "Value mismatch at docId " + docId);
}
- docId++;
}
- Assert.assertEquals(docId, totalDocs, "Should have iterated through all documents");
}
} finally {
reader.close();
}
}
-
- /**
- * Data provider for multi-value accessor methods.
- * @return test parameters: column name, random access getter function,
- * sequential getter function, array converter function, index type
- */
+ /// Data provider for multi-value accessor methods.
+ ///
+ /// @return test parameters: column name, random access getter function, array converter function, index type
@DataProvider(name = "multiValueAccessorProvider")
public Object[][] multiValueAccessorProvider() {
// Base column configurations
// For primitive MV types, we need to extract .getValues() from MultiValueResult
// and verify that hasNulls() is false (nulls are removed by NullValueTransformer for MV primitive types)
- Object[][] baseConfigs = new Object[][] {
- {MV_INT_COL, (MultiValueGetter) (reader, docId) -> {
+ Object[][] baseConfigs = new Object[][]{
+ {
+ MV_INT_COL, (MultiValueGetter) (reader, docId) -> {
MultiValueResult<int[]> result = reader.getIntMV(docId);
- Assert.assertFalse(result.hasNulls(), "Multi-value primitive types should not have nulls");
+ assertFalse(result.hasNulls(), "Multi-value primitive types should not have nulls");
Review Comment:
Reformatting deferred to `spotless:apply` (pre-commit). The multi-line rows only carry per-row converter/getter lambdas; the simple rows stay one-line, at consistent 12-space indentation. Not hand-editing to avoid diverging from spotless's canonical output.
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/readers/DefaultValueColumnReaderTest.java:
##########
@@ -19,24 +19,26 @@
package org.apache.pinot.segment.local.segment.readers;
import java.math.BigDecimal;
-import java.util.Arrays;
import org.apache.pinot.spi.data.DimensionFieldSpec;
import org.apache.pinot.spi.data.FieldSpec;
import org.apache.pinot.spi.data.MetricFieldSpec;
import org.apache.pinot.spi.data.readers.MultiValueResult;
-import org.testng.Assert;
+import org.apache.pinot.spi.utils.PinotDataType;
import org.testng.annotations.Test;
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertSame;
+import static org.testng.Assert.assertTrue;
+
/// Comprehensive tests for [DefaultValueColumnReader].
///
/// This test validates:
/// - Single-value fields for all data types (INT, LONG, FLOAT, DOUBLE, BIG_DECIMAL, STRING, BYTES)
/// - Multi-value fields for all data types
-/// - Sequential access methods (next*, hasNext, skipNext, isNextNull)
/// - Random access methods (get*, isNull)
/// - Type indicator methods (isInt, isLong, isBigDecimal, etc.)
Review Comment:
Fixed in 6471bb8 — updated to "Value type reporting (getValueType)".
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/readers/DefaultValueColumnReaderTest.java:
##########
@@ -308,20 +231,20 @@ public void testMultiValueFloatColumn() {
FieldSpec fieldSpec = new DimensionFieldSpec(COLUMN_NAME, FieldSpec.DataType.FLOAT, false);
DefaultValueColumnReader reader = new DefaultValueColumnReader(COLUMN_NAME, NUM_DOCS, fieldSpec);
- // Test sequential access with nextFloatMV
+ // Test random access
float expectedValue = ((Number) fieldSpec.getDefaultNullValue()).floatValue();
float[] expectedArray = new float[]{expectedValue};
- for (int i = 0; i < NUM_DOCS; i++) {
- MultiValueResult<float[]> mvResult = reader.nextFloatMV();
- Assert.assertFalse(mvResult.hasNulls());
- Assert.assertTrue(Arrays.equals(mvResult.getValues(), expectedArray));
+ for (int docId = 0; docId < reader.getTotalDocs(); docId++) {
+ MultiValueResult<float[]> mvResult = reader.getFloatMV(docId);
+ assertFalse(mvResult.hasNulls());
+ assertEquals(mvResult.getValues(), expectedArray);
}
- // Test random access and array reuse
- reader.rewind();
+ // Test array reuse
+ // Test array reuse
Review Comment:
Fixed in 6471bb8 — collapsed the duplicate comment.
##########
pinot-plugins/pinot-input-format/pinot-arrow/src/main/java/org/apache/pinot/plugin/inputformat/arrow/ArrowColumnReader.java:
##########
@@ -35,60 +35,47 @@
import org.apache.arrow.vector.complex.ListVector;
import org.apache.pinot.spi.data.readers.ColumnReader;
import org.apache.pinot.spi.data.readers.MultiValueResult;
-
-
-/**
- * Column reader for Apache Arrow {@link FieldVector}.
- *
- * <p>Wraps a single Arrow {@link FieldVector} and exposes sequential and random-access
- * read patterns conforming to the {@link ColumnReader} contract. The vector is owned by
- * the enclosing {@link ArrowColumnReaderFactory} which is responsible for its lifecycle;
- * closing this reader is a no-op for the underlying vector.
- *
- * <p>Supported Arrow types map to Pinot's stored types as follows:
- * <ul>
- * <li>{@code IntVector} / {@code BitVector} → {@code INT} (BitVector promoted to 0/1)</li>
- * <li>{@code BigIntVector} → {@code LONG}</li>
- * <li>{@code Float4Vector} → {@code FLOAT}</li>
- * <li>{@code Float8Vector} → {@code DOUBLE}</li>
- * <li>{@code DecimalVector} → {@code BIG_DECIMAL}</li>
- * <li>{@code VarCharVector} → {@code STRING}</li>
- * <li>{@code VarBinaryVector} → {@code BYTES}</li>
- * <li>{@code ListVector} of the above → multi-value variant</li>
- * </ul>
- *
- * <p>The list above applies to the typed primitive accessors only ({@link #getInt},
- * {@link #getString}, {@link #getIntMV}, ...). Complex types (Map, Struct, Union, ...)
- * and temporal types are still readable via the generic {@link #getValue(int)} /
- * {@link #next()} accessors, which delegate to {@link ArrowToPinotTypeConverter} and
- * return the same canonical JDK types as the row-major path ({@link
- * ArrowRecordExtractor}) — e.g. {@code Map<String, Object>} for Struct / Map,
- * {@code Object[]} for List variants, {@code LocalDate} / {@code LocalTime} /
- * {@code Timestamp} for temporal types.
- *
- * <p><b>BitVector caveat:</b> Arrow's {@link FieldVector#getObject} returns a {@code Boolean}
- * for a {@link BitVector}, but Pinot stores booleans as {@code INT} (0/1). The generic
- * {@link #getValue(int)} therefore special-cases {@code BitVector} to return an {@code Integer}
- * 0/1, keeping it consistent with {@link #getInt(int)} and the advertised INT mapping above
- * rather than surfacing the shared converter's {@code Boolean}.
- *
- * <p>{@code @SuppressWarnings("serial")}: {@link ColumnReader} is {@link java.io.Serializable} by SPI
- * contract, but this reader holds non-serializable Arrow state and is never serialized — it lives
- * only for the duration of a columnar segment build.
- *
- * <p>This class is not thread-safe.
- */
-@SuppressWarnings("serial")
+import org.apache.pinot.spi.utils.PinotDataType;
+
+
+/// Column reader for an Apache Arrow [FieldVector].
+///
+/// Wraps a single Arrow [FieldVector] and exposes random-access reads conforming to the [ColumnReader] contract. The
+/// vector is owned by the enclosing [ArrowColumnReaderFactory], which is responsible for its lifecycle; closing this
+/// reader is a no-op for the underlying vector.
+///
+/// Supported Arrow types map to Pinot's stored types as follows:
+/// - `IntVector` / `BitVector` → `INT` (BitVector promoted to 0/1)
+/// - `BigIntVector` → `LONG`
+/// - `Float4Vector` → `FLOAT`
+/// - `Float8Vector` → `DOUBLE`
+/// - `DecimalVector` → `BIG_DECIMAL`
+/// - `VarCharVector` → `STRING`
+/// - `VarBinaryVector` → `BYTES`
+/// - `ListVector` of the above → multi-value variant
+///
+/// The list above applies to the typed primitive accessors only ([#getInt(int)], [#getString(int)], [#getIntMV(int)],
+/// ...). Complex types (Map, Struct, Union, ...) and temporal types are still readable via the generic
+/// [#getValue(int)] accessor, which delegates to [ArrowToPinotTypeConverter] and returns the same canonical JDK types
+/// as the row-major path ([ArrowRecordExtractor]) — e.g. `Map<String, Object>` for Struct / Map, `Object[]` for List
+/// variants, `LocalDate` / `LocalTime` / `Timestamp` for temporal types.
+///
+/// **BitVector caveat:** Arrow's [FieldVector#getObject(int)] returns a `Boolean` for a `BitVector`, but Pinot stores
+/// booleans as `INT` (0/1). The generic [#getValue(int)] therefore special-cases `BitVector` to return an `Integer`
+/// 0/1, keeping it consistent with [#getInt(int)] and the advertised INT mapping above rather than surfacing the
+/// shared converter's `Boolean`.
+///
+/// `@SuppressWarnings("serial")`: [ColumnReader] is [java.io.Serializable] by SPI contract, but this reader holds
+/// non-serializable Arrow state and is never serialized — it lives only for the duration of a columnar segment build.
+///
+/// This class is not thread-safe.
public class ArrowColumnReader implements ColumnReader {
Review Comment:
The `@SuppressWarnings("serial")` is redundant here: the sibling `ColumnReader` implementations (`PinotSegmentColumnReaderImpl`, `DefaultValueColumnReader`, `InMemoryColumnReader`) are `Serializable` via the interface without it and compile clean, so the `serial` lint is not emitted. Removed the stale Javadoc reference in 6471bb8 rather than restoring a redundant annotation.
##########
pinot-plugins/pinot-input-format/pinot-arrow/src/main/java/org/apache/pinot/plugin/inputformat/arrow/BatchedArrowColumnReader.java:
##########
@@ -27,38 +27,30 @@
import org.apache.arrow.vector.types.pojo.Field;
import org.apache.pinot.spi.data.readers.ColumnReader;
import org.apache.pinot.spi.data.readers.MultiValueResult;
-
-
-/**
- * A {@link ColumnReader} over one column of a {@link BatchedArrowFileSource}. Reads are served by
- * delegating to a per-batch {@link ArrowColumnReader} over the current record batch's column vector;
- * the source loads at most one batch at a time, so peak resident memory is one batch (see
- * {@link BatchedArrowFileSource}).
- *
- * <p>Random access ({@code getXxx(docId)}) seeks to the owning batch and reads it; sequential access
- * ({@code next} / typed {@code nextXxx}) is built on it. Type predicates are derived from the column's
- * (decoded) Arrow {@link Field}, so they need no batch load. The column-major segment build consumes
- * each column to completion sequentially (rewind + next), one column at a time.
- *
- * <p>The per-batch delegate is invalidated when the shared source moves to another batch (whether
- * this reader advanced it or a different one did); {@link #delegate(int)} documents why and rebuilds
- * it. Interleaving readers therefore stays correct (it just re-loads batches); the build never
- * interleaves.
- *
- * <p>This class is not thread-safe. {@code @SuppressWarnings("serial")}: {@link ColumnReader} is
- * {@link java.io.Serializable} by SPI contract, but this reader holds non-serializable Arrow state
- * and is never serialized.
- */
-@SuppressWarnings("serial")
+import org.apache.pinot.spi.utils.PinotDataType;
+
+
+/// A [ColumnReader] over one column of a [BatchedArrowFileSource]. Reads are served by delegating to a per-batch
+/// [ArrowColumnReader] over the current record batch's column vector; the source loads at most one batch at a time, so
+/// peak resident memory is one batch (see [BatchedArrowFileSource]).
+///
+/// Random access (`getXxx(docId)`) seeks to the owning batch and reads it. The value type is derived from the column's
+/// (decoded) Arrow [Field], so it needs no batch load. The column-major segment build consumes each column to
+/// completion (by ascending docId), one column at a time.
+///
+/// The per-batch delegate is invalidated when the shared source moves to another batch (whether this reader advanced
+/// it or a different one did); [#delegate(int)] documents why and rebuilds it. Interleaving readers therefore stays
+/// correct (it just re-loads batches); the build never interleaves.
+///
+/// This class is not thread-safe. `@SuppressWarnings("serial")`: [ColumnReader] is [java.io.Serializable] by SPI
+/// contract, but this reader holds non-serializable Arrow state and is never serialized.
public class BatchedArrowColumnReader implements ColumnReader {
Review Comment:
The `@SuppressWarnings("serial")` is redundant here: the sibling `ColumnReader` implementations (`PinotSegmentColumnReaderImpl`, `DefaultValueColumnReader`, `InMemoryColumnReader`) are `Serializable` via the interface without it and compile clean, so the `serial` lint is not emitted. Removed the stale Javadoc reference in 6471bb8 rather than restoring a redundant annotation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Simplify ColumnReader SPI: drop iterator API, replace type checks with getValueType() [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18918:
URL: https://github.com/apache/pinot/pull/18918#issuecomment-4879009920
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/18918?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `66 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 57.01%. Comparing base ([`19897c5`](https://app.codecov.io/gh/apache/pinot/commit/19897c591be647bc627a71ba57dfc935d34b25a7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`1358cbc`](https://app.codecov.io/gh/apache/pinot/commit/1358cbc18d9ea43919d27957e62c9a829a77a981?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/18918?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...l/stats/ColumnarSegmentPreIndexStatsContainer.java](https://app.codecov.io/gh/apache/pinot/pull/18918?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fsegment%2Fcreator%2Fimpl%2Fstats%2FColumnarSegmentPreIndexStatsContainer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9zdGF0cy9Db2x1bW5hclNlZ21lbnRQcmVJbmRleFN0YXRzQ29udGFpbmVyLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18918?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://app.codecov.io/gh/apache/pinot/pull/18918?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fsegment%2Fcreator%2Fimpl%2FSegmentColumnarIndexCreator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | 0.00% | [21 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18918?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/pinot/spi/data/readers/ColumnReader.java](https://app.codecov.io/gh/apache/pinot/pull/18918?src=pr&el=tree&filepath=pinot-spi%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fspi%2Fdata%2Freaders%2FColumnReader.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9yZWFkZXJzL0NvbHVtblJlYWRlci5qYXZh) | 0.00% | [9 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18918?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../segment/readers/PinotSegmentColumnReaderImpl.java](https://app.codecov.io/gh/apache/pinot/pull/18918?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fsegment%2Freaders%2FPinotSegmentColumnReaderImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L3JlYWRlcnMvUGlub3RTZWdtZW50Q29sdW1uUmVhZGVySW1wbC5qYXZh) | 0.00% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18918?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ocal/segment/readers/DefaultValueColumnReader.java](https://app.codecov.io/gh/apache/pinot/pull/18918?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fsegment%2Freaders%2FDefaultValueColumnReader.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L3JlYWRlcnMvRGVmYXVsdFZhbHVlQ29sdW1uUmVhZGVyLmphdmE=) | 0.00% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18918?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...l/columntransformer/DataTypeColumnTransformer.java](https://app.codecov.io/gh/apache/pinot/pull/18918?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fcolumntransformer%2FDataTypeColumnTransformer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9jb2x1bW50cmFuc2Zvcm1lci9EYXRhVHlwZUNvbHVtblRyYW5zZm9ybWVyLmphdmE=) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18918?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (19897c5) and HEAD (1358cbc). Click for more details.
>
> <details><summary>HEAD has 16 uploads less than BASE</summary>
>
>| Flag | BASE (19897c5) | HEAD (1358cbc) |
>|------|------|------|
>|unittests2|1|0|
>|unittests|2|1|
>|java-21|5|1|
>|temurin|5|1|
>|integration|3|0|
>|integration1|1|0|
>|custom-integration1|1|0|
>|integration2|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18918 +/- ##
============================================
- Coverage 64.80% 57.01% -7.79%
+ Complexity 1347 1 -1346
============================================
Files 3392 2604 -788
Lines 211769 152882 -58887
Branches 33345 24897 -8448
============================================
- Hits 137232 87168 -50064
+ Misses 63451 58304 -5147
+ Partials 11086 7410 -3676
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/18918/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/18918/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/18918/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/18918/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/18918/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/18918/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.01% <0.00%> (-7.79%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/18918/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.01% <0.00%> (-7.79%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/18918/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.01% <0.00%> (-7.79%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/18918/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.01% <0.00%> (+0.06%)` | :arrow_up: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/18918/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/pinot/pull/18918?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Simplify ColumnReader SPI: drop iterator API, replace type checks with getValueType() [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #18918:
URL: https://github.com/apache/pinot/pull/18918#discussion_r3521769663
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/columntransformer/DataTypeColumnTransformerTest.java:
##########
@@ -211,7 +153,7 @@ public void testIsNoOpForMatchingIntMVTypes() {
.build();
FieldSpec fieldSpec = schema.getFieldSpecFor(COLUMN_NAME);
TableConfig tableConfig = new TableConfigBuilder(TableType.OFFLINE).setTableName("testTable").build();
- ColumnReader reader = mockColumnReader().multiValue().asInt().build();
+ ColumnReader reader = mockReader(PinotDataType.INT_ARRAY);
DataTypeColumnTransformer transformer = new DataTypeColumnTransformer(tableConfig, fieldSpec, reader);
Review Comment:
The MV isNoOp coverage only asserts INT/LONG/STRING/BYTES; it is missing explicit `FLOAT_ARRAY` and `DOUBLE_ARRAY` cases, even though the SPI-driven `getValueType() == destType` logic needs to work for those types too. Adding MV float/double isNoOp tests would help ensure coverage across all supported primitive MV types.
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReader.java:
##########
@@ -23,300 +23,110 @@
import java.io.Serializable;
import java.math.BigDecimal;
import javax.annotation.Nullable;
-
-
-/**
- * The <code>ColumnReader</code> interface is used to read column data from various data sources
- * for columnar segment building. Unlike RecordReader which reads row-by-row, ColumnReader provides
- * column-wise access to data, enabling efficient columnar segment creation.
- *
- * <p>This interface provides 3 patterns optimised for different use cases
- * (Some implementations may not support all patterns):
- * <ul>
- * <li>Sequential iteration over all values in a column using hasNext(), next() and rewind()</li>
- * <li>Sequential iteration with type-specific methods (isInt(), isLong(), nextInt(), nextLong(), etc.)
- * and null handling (isNextNull(), skipNext()) and supports hasNext() and rewind()</li>
- * <li>Random access by document ID using getInt(docId), getLong(docId), etc. and isNull(docId) for null checks</li>
- * </ul>
- *
- * <p>Implementations should handle data type conversions and efficient column-wise data access patterns.
- *
- * <h2>Usage Patterns</h2>
- *
- * <p>There are three primary patterns for reading data from a ColumnReader:
- *
- * <h3>Pattern 1: Sequential Iteration with Generic next() and Null Checks</h3>
- * <p>This pattern uses the generic {@link #next()} method which returns Object and may return null.
- * Suitable when you need to handle arbitrary data types or when null handling is done on the return value.
- *
- * <pre>{@code
- * // Read all values in the column
- * while (columnReader.hasNext()) {
- * Object value;
- * try {
- * value = columnReader.next();
- * } catch (Exception e) {
- * // Handle exception / log
- * continue;
- * }
- * if (value != null) {
- * // Process non-null value
- * processValue(value);
- * } else {
- * // Handle null value
- * handleNullValue();
- * }
- * }
- *
- * // Rewind to read the column again
- * columnReader.rewind();
- *
- * // Second pass through the data
- * while (columnReader.hasNext()) {
- * Object value = columnReader.next();
- * if (value != null) {
- * processValueAgain(value);
- * }
- * }
- * }</pre>
- *
- * <h3>Pattern 2: Sequential Iteration with Type-Specific Methods and Explicit Null Checks</h3>
- * <p>This pattern uses {@link #isNextNull()} to check for nulls before calling type-specific methods
- * like {@link #nextInt()}, {@link #nextLong()}, etc. Use {@link #skipNext()} to advance past null values.
- * This is the preferred pattern when you know the column data type and want to avoid boxing overhead.
- * Before using this pattern, check the column data type using methods like {@link #isInt()}, {@link #isLong()}, etc.
- * If the data type does not match, fall back to Pattern 1 with {@link #next()}.
- * <pre>{@code
- * // Read all int values in the column, handling nulls
- * if (columnReader.isInt()) {
- * while (columnReader.hasNext()) {
- * if (columnReader.isNextNull()) {
- * // Skip the null value
- * columnReader.skipNext();
- * handleNullValue();
- * } else {
- * // Read the primitive int value (no boxing)
- * int value = columnReader.nextInt();
- * processIntValue(value);
- * }
- * }
- * } else {
- * // Fallback to Pattern 1 if not INT type
- * }
- *
- * }</pre>
- *
- * <h3>Pattern 3: Random Access by Document ID</h3>
- * <p>This pattern uses {@link #getTotalDocs()} to get the total number of documents, then uses
- * document ID-based accessors like {@link #getInt(int)}, {@link #getLong(int)}, etc. to read
- * specific values. Use {@link #isNull(int)} to check if a value is null before reading.
- * This pattern is useful when you need random access or want to process documents in a specific order.
- *
- * <pre>{@code
- *
- * // Random access example - read specific document IDs
- * int[] docIdsToRead = {5, 10, 15, 20};
- * for (int docId : docIdsToRead) {
- * if (!columnReader.isNull(docId)) {
- * int value = columnReader.getInt(docId);
- * processSpecificDoc(docId, value);
- * }
- * }
- *
- * // Read in reverse order
- * // Get the total number of documents
- * int totalDocs = columnReader.getTotalDocs();
- * for (int docId = totalDocs - 1; docId >= 0; docId--) {
- * if (!columnReader.isNull(docId)) {
- * int value = columnReader.getInt(docId);
- * processReverseOrder(docId, value);
- * }
- * }
- * }</pre>
- *
- * <h3>Choosing the Right Pattern</h3>
- * <ul>
- * <li><b>Pattern 1</b>: Use when dealing with generic Object types or when you don't know
- * the column type at compile time. Less efficient due to boxing.</li>
- * <li><b>Pattern 2</b>: Use when you know the column type and want efficient sequential iteration
- * with primitive types. Preferred for most columnar segment building scenarios.</li>
- * <li><b>Pattern 3</b>: Use when you need random access, want to process documents in a specific
- * order, or need to access the same document multiple times.</li>
- * </ul>
- *
- */
+import org.apache.pinot.spi.data.FieldSpec.DataType;
+import org.apache.pinot.spi.utils.PinotDataType;
+
+
+/// Reads the values of a single column by document ID, for columnar segment building. Where a `RecordReader` exposes a
+/// data source row-by-row, a `ColumnReader` exposes one column at a time: a caller reads every value of a column before
+/// moving to the next one. Building a segment one fully-materialized column at a time is cheaper than transposing row
+/// records when the source is already columnar (e.g. Arrow, Parquet).
+///
+/// Documents are addressed by a 0-based id, from 0 (inclusive) to [#getTotalDocs()] (exclusive). To read a column:
+/// 1. Call [#getValueType()] once. A non-`null` result names the type that can be read directly through the matching
+/// type-specific accessor; `null` means the column must be read through the boxed [#getValue(int)].
+/// 2. For each document id, call [#isNull(int)] first — type-specific accessors cannot represent null and must not be
+/// called for a null value.
+/// 3. Read each non-null value with the accessor chosen in step 1, e.g. [#getInt(int)] / [#getIntMV(int)] when
+/// [#getValueType()] is [PinotDataType#INT] / [PinotDataType#INT_ARRAY], or [#getValue(int)] otherwise.
+///
+/// Implementations perform any conversion between the source encoding and the returned Pinot type, are not required to
+/// be thread-safe, and must be released with [#close()] when no longer needed.
+///
+/// ```
+/// PinotDataType valueType = columnReader.getValueType();
+/// for (int docId = 0; docId < columnReader.getTotalDocs(); docId++) {
+/// if (columnReader.isNull(docId)) {
+/// continue;
+/// }
+/// if (valueType == PinotDataType.INT) {
+/// consumeInt(columnReader.getInt(docId));
+/// } else {
+/// consumeObject(columnReader.getValue(docId));
+/// }
+/// }
+/// ```
public interface ColumnReader extends Closeable, Serializable {
- /**
- * Return <code>true</code> if more values remain to be read in this column.
- * <p>This method should not throw exception. Caller is not responsible for handling exceptions from this method.
- */
- boolean hasNext();
+ /// Returns the name of the column read by this reader.
+ String getColumnName();
- /**
- * Get the next value in the column.
- * <p>This method should be called only if {@link #hasNext()} returns <code>true</code>. Caller is responsible for
- * handling exceptions from this method and skip the value if user wants to continue reading the remaining values.
- *
- * @return Next column value, or null if the value is null
- * @throws IOException If an I/O error occurs while reading
- */
+ /// Returns the [PinotDataType] whose type-specific accessors can read this column directly, or `null` if the column
+ /// has no such accessor and must be read through [#getValue(int)].
+ ///
+ /// The returned type encodes cardinality: a single-value column returns a scalar type read through the scalar
+ /// accessor (e.g. [PinotDataType#INT] via [#getInt(int)]), while a multi-value column returns the array type read
+ /// through the multi-value accessor (e.g. [PinotDataType#INT_ARRAY] via [#getIntMV(int)]). Columns whose stored type
+ /// has no dedicated accessor — such as BOOLEAN, TIMESTAMP, or JSON — return `null`.
Review Comment:
The `getValueType()` Javadoc says BOOLEAN columns have no dedicated accessor and should return `null`, but Arrow-based implementations treat Arrow booleans (`BitVector` / `ArrowType.Bool`) as directly readable INT (0/1) and return `PinotDataType.INT`. This is a contract/documentation mismatch that could confuse implementers and consumers; please clarify the SPI Javadoc to match the actual behavior (or adjust Arrow readers to return `null` for booleans if that is the intended contract).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Simplify ColumnReader SPI: drop iterator API, replace type checks with getValueType() [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #18918:
URL: https://github.com/apache/pinot/pull/18918#discussion_r3522434688
##########
pinot-plugins/pinot-input-format/pinot-arrow/src/main/java/org/apache/pinot/plugin/inputformat/arrow/ArrowToPinotTypeConverter.java:
##########
@@ -59,6 +60,53 @@ public final class ArrowToPinotTypeConverter {
private ArrowToPinotTypeConverter() {
}
+ /// Returns the [PinotDataType] that the given Arrow `elementType` (a scalar element type, already unwrapped from any
+ /// list) can be read as through a type-specific `ColumnReader` accessor, or `null` when it has no dedicated accessor
+ /// and must be read through `ColumnReader.getValue()`. Decided from the logical Arrow type, not the physical vector:
+ /// logical `Bool`, temporal types (unless `extractRawTimeValues` surfaces a `Timestamp`'s raw epoch `long`), and
+ /// complex types have no dedicated accessor.
+ ///
+ /// @param elementType scalar Arrow type of the column (or of a list's elements)
+ /// @param singleValue `true` for a single-value column (scalar type), `false` for the `_ARRAY` variant
+ /// @param extractRawTimeValues when `true`, a `Timestamp` column is readable as `LONG` (its raw epoch value);
+ /// otherwise it has no dedicated accessor
+ @Nullable
+ public static PinotDataType toValueType(ArrowType elementType, boolean singleValue, boolean extractRawTimeValues) {
+ switch (elementType.getTypeID()) {
+ case Int:
+ switch (((ArrowType.Int) elementType).getBitWidth()) {
+ case 32:
+ return singleValue ? PinotDataType.INT : PinotDataType.INT_ARRAY;
+ case 64:
+ return singleValue ? PinotDataType.LONG : PinotDataType.LONG_ARRAY;
+ default:
+ return null;
+ }
+ case FloatingPoint:
+ switch (((ArrowType.FloatingPoint) elementType).getPrecision()) {
+ case SINGLE:
+ return singleValue ? PinotDataType.FLOAT : PinotDataType.FLOAT_ARRAY;
+ case DOUBLE:
+ return singleValue ? PinotDataType.DOUBLE : PinotDataType.DOUBLE_ARRAY;
+ default:
+ return null;
+ }
+ case Decimal:
+ return singleValue ? PinotDataType.BIG_DECIMAL : PinotDataType.BIG_DECIMAL_ARRAY;
+ case Utf8:
+ case LargeUtf8:
+ return singleValue ? PinotDataType.STRING : PinotDataType.STRING_ARRAY;
+ case Binary:
+ case LargeBinary:
+ case FixedSizeBinary:
+ return singleValue ? PinotDataType.BYTES : PinotDataType.BYTES_ARRAY;
Review Comment:
`toValueType()` reports `STRING`/`BYTES` for `LargeUtf8`/`LargeBinary`/`FixedSizeBinary`, but the Arrow `ColumnReader` typed accessors in this module only handle `VarCharVector`/`VarBinaryVector` (not the Large*/FixedSize vector classes). This can violate the `getValueType()` contract and cause callers to pick `getString()`/`getBytes()` and then fail with a type mismatch. Consider returning `null` for these large/fixed logical types unless/until the typed accessors are extended to support them.
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReader.java:
##########
@@ -22,301 +22,180 @@
import java.io.IOException;
import java.io.Serializable;
import java.math.BigDecimal;
+import java.sql.Timestamp;
import javax.annotation.Nullable;
-
-
-/**
- * The <code>ColumnReader</code> interface is used to read column data from various data sources
- * for columnar segment building. Unlike RecordReader which reads row-by-row, ColumnReader provides
- * column-wise access to data, enabling efficient columnar segment creation.
- *
- * <p>This interface provides 3 patterns optimised for different use cases
- * (Some implementations may not support all patterns):
- * <ul>
- * <li>Sequential iteration over all values in a column using hasNext(), next() and rewind()</li>
- * <li>Sequential iteration with type-specific methods (isInt(), isLong(), nextInt(), nextLong(), etc.)
- * and null handling (isNextNull(), skipNext()) and supports hasNext() and rewind()</li>
- * <li>Random access by document ID using getInt(docId), getLong(docId), etc. and isNull(docId) for null checks</li>
- * </ul>
- *
- * <p>Implementations should handle data type conversions and efficient column-wise data access patterns.
- *
- * <h2>Usage Patterns</h2>
- *
- * <p>There are three primary patterns for reading data from a ColumnReader:
- *
- * <h3>Pattern 1: Sequential Iteration with Generic next() and Null Checks</h3>
- * <p>This pattern uses the generic {@link #next()} method which returns Object and may return null.
- * Suitable when you need to handle arbitrary data types or when null handling is done on the return value.
- *
- * <pre>{@code
- * // Read all values in the column
- * while (columnReader.hasNext()) {
- * Object value;
- * try {
- * value = columnReader.next();
- * } catch (Exception e) {
- * // Handle exception / log
- * continue;
- * }
- * if (value != null) {
- * // Process non-null value
- * processValue(value);
- * } else {
- * // Handle null value
- * handleNullValue();
- * }
- * }
- *
- * // Rewind to read the column again
- * columnReader.rewind();
- *
- * // Second pass through the data
- * while (columnReader.hasNext()) {
- * Object value = columnReader.next();
- * if (value != null) {
- * processValueAgain(value);
- * }
- * }
- * }</pre>
- *
- * <h3>Pattern 2: Sequential Iteration with Type-Specific Methods and Explicit Null Checks</h3>
- * <p>This pattern uses {@link #isNextNull()} to check for nulls before calling type-specific methods
- * like {@link #nextInt()}, {@link #nextLong()}, etc. Use {@link #skipNext()} to advance past null values.
- * This is the preferred pattern when you know the column data type and want to avoid boxing overhead.
- * Before using this pattern, check the column data type using methods like {@link #isInt()}, {@link #isLong()}, etc.
- * If the data type does not match, fall back to Pattern 1 with {@link #next()}.
- * <pre>{@code
- * // Read all int values in the column, handling nulls
- * if (columnReader.isInt()) {
- * while (columnReader.hasNext()) {
- * if (columnReader.isNextNull()) {
- * // Skip the null value
- * columnReader.skipNext();
- * handleNullValue();
- * } else {
- * // Read the primitive int value (no boxing)
- * int value = columnReader.nextInt();
- * processIntValue(value);
- * }
- * }
- * } else {
- * // Fallback to Pattern 1 if not INT type
- * }
- *
- * }</pre>
- *
- * <h3>Pattern 3: Random Access by Document ID</h3>
- * <p>This pattern uses {@link #getTotalDocs()} to get the total number of documents, then uses
- * document ID-based accessors like {@link #getInt(int)}, {@link #getLong(int)}, etc. to read
- * specific values. Use {@link #isNull(int)} to check if a value is null before reading.
- * This pattern is useful when you need random access or want to process documents in a specific order.
- *
- * <pre>{@code
- *
- * // Random access example - read specific document IDs
- * int[] docIdsToRead = {5, 10, 15, 20};
- * for (int docId : docIdsToRead) {
- * if (!columnReader.isNull(docId)) {
- * int value = columnReader.getInt(docId);
- * processSpecificDoc(docId, value);
- * }
- * }
- *
- * // Read in reverse order
- * // Get the total number of documents
- * int totalDocs = columnReader.getTotalDocs();
- * for (int docId = totalDocs - 1; docId >= 0; docId--) {
- * if (!columnReader.isNull(docId)) {
- * int value = columnReader.getInt(docId);
- * processReverseOrder(docId, value);
- * }
- * }
- * }</pre>
- *
- * <h3>Choosing the Right Pattern</h3>
- * <ul>
- * <li><b>Pattern 1</b>: Use when dealing with generic Object types or when you don't know
- * the column type at compile time. Less efficient due to boxing.</li>
- * <li><b>Pattern 2</b>: Use when you know the column type and want efficient sequential iteration
- * with primitive types. Preferred for most columnar segment building scenarios.</li>
- * <li><b>Pattern 3</b>: Use when you need random access, want to process documents in a specific
- * order, or need to access the same document multiple times.</li>
- * </ul>
- *
- */
+import org.apache.pinot.spi.data.FieldSpec.DataType;
+import org.apache.pinot.spi.utils.BooleanUtils;
+import org.apache.pinot.spi.utils.PinotDataType;
+
+
+/// Reads the values of a single column by document ID, for columnar segment building. Where a `RecordReader` exposes a
+/// data source row-by-row, a `ColumnReader` exposes one column at a time: a caller reads every value of a column before
+/// moving to the next one. Building a segment one fully-materialized column at a time is cheaper than transposing row
+/// records when the source is already columnar (e.g. Arrow, Parquet).
+///
+/// Documents are addressed by a 0-based id, from 0 (inclusive) to [#getTotalDocs()] (exclusive). Two access patterns
+/// are supported:
+/// - **Random access** — [#getValue(int)] and the typed accessors may be called for any document id, in any order.
+/// - **Repeated sequential traversal** — the full range may be traversed by ascending document id more than once. The
+/// reader starts positioned at document 0; call [#rewind()] to reset it before each subsequent traversal. The
+/// segment build uses this: a statistics pass, then an index-writing pass over the same reader.
+///
+/// To read a column:
+/// 1. Call [#getValueType()] once. A non-`null` result names the type that can be read directly through the matching
+/// type-specific accessor; `null` means the column must be read through the boxed [#getValue(int)].
+/// 2. For each document id, call [#isNull(int)] first — type-specific accessors cannot represent null and must not be
+/// called for a null value.
+/// 3. Read each non-null value with the accessor chosen in step 1, e.g. [#getInt(int)] / [#getIntMV(int)] when
+/// [#getValueType()] is [PinotDataType#INT] / [PinotDataType#INT_ARRAY], or [#getValue(int)] otherwise.
+///
+/// Implementations perform any conversion between the source encoding and the returned Pinot type, are not required to
+/// be thread-safe, and must be released with [#close()] when no longer needed.
+///
+/// ```
+/// PinotDataType valueType = columnReader.getValueType();
+/// for (int docId = 0; docId < columnReader.getTotalDocs(); docId++) {
+/// if (columnReader.isNull(docId)) {
+/// continue;
+/// }
+/// if (valueType == PinotDataType.INT) {
+/// consumeInt(columnReader.getInt(docId));
+/// } else {
+/// consumeObject(columnReader.getValue(docId));
+/// }
+/// }
+/// ```
public interface ColumnReader extends Closeable, Serializable {
- /**
- * Return <code>true</code> if more values remain to be read in this column.
- * <p>This method should not throw exception. Caller is not responsible for handling exceptions from this method.
- */
- boolean hasNext();
+ /// Returns the name of the column read by this reader.
+ String getColumnName();
- /**
- * Get the next value in the column.
- * <p>This method should be called only if {@link #hasNext()} returns <code>true</code>. Caller is responsible for
- * handling exceptions from this method and skip the value if user wants to continue reading the remaining values.
- *
- * @return Next column value, or null if the value is null
- * @throws IOException If an I/O error occurs while reading
- */
+ /// Returns the [PinotDataType] that [#getValue(int)] produces for this column, when that type has a matching typed
+ /// accessor; otherwise `null`.
+ ///
+ /// A non-`null` result `T` therefore names an accessor that returns exactly what [#getValue(int)] returns, only
+ /// unboxed: for every document id where [#isNull(int)] is `false`, the accessor matching `T` — e.g. [#getInt(int)]
+ /// for [PinotDataType#INT], [#getIntMV(int)] for [PinotDataType#INT_ARRAY] — returns the same value as
+ /// [#getValue(int)]. `null` means [#getValue(int)]'s type has no typed accessor; read the column through
+ /// [#getValue(int)].
+ ///
+ /// The result is a constant property of the column (independent of the document id) and, when non-`null`, is one of
+ /// the accessor-backed types — [PinotDataType#INT], [PinotDataType#LONG], [PinotDataType#FLOAT],
+ /// [PinotDataType#DOUBLE], [PinotDataType#BIG_DECIMAL], [PinotDataType#STRING], [PinotDataType#BYTES] — or their
+ /// `_ARRAY` variants for a multi-value column.
+ ///
+ /// Because [#getValue(int)] returns the column's logical type, a logical type whose object is not itself an
+ /// accessor-backed type returns `null`: `BOOLEAN` (a `Boolean`), `TIMESTAMP` (a `Timestamp`), and the complex types.
+ /// A `JSON` column, stored and read as its text `String`, reports [PinotDataType#STRING]. Implementations keyed on
+ /// the stored [DataType] can use [#toValueType(DataType, boolean)].
Review Comment:
The `getValueType()` contract doc currently states that `TIMESTAMP` always returns `null` because `getValue()` returns a `Timestamp`, but Arrow implementations can surface a TIMESTAMP column as raw epoch `LONG`/`LONG_ARRAY` when `extractRawTimeValues` is enabled (see ArrowToPinotTypeConverter + ArrowColumnReaderTest). This is a documentation/contract mismatch that can confuse implementers and callers.
##########
pinot-plugins/pinot-input-format/pinot-arrow/src/main/java/org/apache/pinot/plugin/inputformat/arrow/ArrowToPinotTypeConverter.java:
##########
@@ -59,6 +60,53 @@ public final class ArrowToPinotTypeConverter {
private ArrowToPinotTypeConverter() {
}
+ /// Returns the [PinotDataType] that the given Arrow `elementType` (a scalar element type, already unwrapped from any
+ /// list) can be read as through a type-specific `ColumnReader` accessor, or `null` when it has no dedicated accessor
+ /// and must be read through `ColumnReader.getValue()`. Decided from the logical Arrow type, not the physical vector:
+ /// logical `Bool`, temporal types (unless `extractRawTimeValues` surfaces a `Timestamp`'s raw epoch `long`), and
+ /// complex types have no dedicated accessor.
+ ///
+ /// @param elementType scalar Arrow type of the column (or of a list's elements)
+ /// @param singleValue `true` for a single-value column (scalar type), `false` for the `_ARRAY` variant
+ /// @param extractRawTimeValues when `true`, a `Timestamp` column is readable as `LONG` (its raw epoch value);
+ /// otherwise it has no dedicated accessor
+ @Nullable
+ public static PinotDataType toValueType(ArrowType elementType, boolean singleValue, boolean extractRawTimeValues) {
+ switch (elementType.getTypeID()) {
+ case Int:
+ switch (((ArrowType.Int) elementType).getBitWidth()) {
+ case 32:
+ return singleValue ? PinotDataType.INT : PinotDataType.INT_ARRAY;
+ case 64:
+ return singleValue ? PinotDataType.LONG : PinotDataType.LONG_ARRAY;
Review Comment:
`toValueType()` currently returns `INT`/`LONG` for any Arrow `Int(32|64)` regardless of signedness. For unsigned int vectors (e.g. `UInt4Vector`), Arrow often surfaces values as a wider boxed type (e.g. `Long`) via `getObject()`, and the typed accessors here don’t support unsigned vectors. Reporting `INT`/`LONG` can therefore violate the ColumnReader contract (typed accessor must match `getValue()`), and can lead callers to invoke `getInt()`/`getLong()` and hit a type-mismatch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Simplify ColumnReader SPI: drop iterator API, replace type checks with getValueType() [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #18918:
URL: https://github.com/apache/pinot/pull/18918#discussion_r3532260582
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReader.java:
##########
@@ -22,301 +22,180 @@
import java.io.IOException;
import java.io.Serializable;
import java.math.BigDecimal;
+import java.sql.Timestamp;
import javax.annotation.Nullable;
-
-
-/**
- * The <code>ColumnReader</code> interface is used to read column data from various data sources
- * for columnar segment building. Unlike RecordReader which reads row-by-row, ColumnReader provides
- * column-wise access to data, enabling efficient columnar segment creation.
- *
- * <p>This interface provides 3 patterns optimised for different use cases
- * (Some implementations may not support all patterns):
- * <ul>
- * <li>Sequential iteration over all values in a column using hasNext(), next() and rewind()</li>
- * <li>Sequential iteration with type-specific methods (isInt(), isLong(), nextInt(), nextLong(), etc.)
- * and null handling (isNextNull(), skipNext()) and supports hasNext() and rewind()</li>
- * <li>Random access by document ID using getInt(docId), getLong(docId), etc. and isNull(docId) for null checks</li>
- * </ul>
- *
- * <p>Implementations should handle data type conversions and efficient column-wise data access patterns.
- *
- * <h2>Usage Patterns</h2>
- *
- * <p>There are three primary patterns for reading data from a ColumnReader:
- *
- * <h3>Pattern 1: Sequential Iteration with Generic next() and Null Checks</h3>
- * <p>This pattern uses the generic {@link #next()} method which returns Object and may return null.
- * Suitable when you need to handle arbitrary data types or when null handling is done on the return value.
- *
- * <pre>{@code
- * // Read all values in the column
- * while (columnReader.hasNext()) {
- * Object value;
- * try {
- * value = columnReader.next();
- * } catch (Exception e) {
- * // Handle exception / log
- * continue;
- * }
- * if (value != null) {
- * // Process non-null value
- * processValue(value);
- * } else {
- * // Handle null value
- * handleNullValue();
- * }
- * }
- *
- * // Rewind to read the column again
- * columnReader.rewind();
- *
- * // Second pass through the data
- * while (columnReader.hasNext()) {
- * Object value = columnReader.next();
- * if (value != null) {
- * processValueAgain(value);
- * }
- * }
- * }</pre>
- *
- * <h3>Pattern 2: Sequential Iteration with Type-Specific Methods and Explicit Null Checks</h3>
- * <p>This pattern uses {@link #isNextNull()} to check for nulls before calling type-specific methods
- * like {@link #nextInt()}, {@link #nextLong()}, etc. Use {@link #skipNext()} to advance past null values.
- * This is the preferred pattern when you know the column data type and want to avoid boxing overhead.
- * Before using this pattern, check the column data type using methods like {@link #isInt()}, {@link #isLong()}, etc.
- * If the data type does not match, fall back to Pattern 1 with {@link #next()}.
- * <pre>{@code
- * // Read all int values in the column, handling nulls
- * if (columnReader.isInt()) {
- * while (columnReader.hasNext()) {
- * if (columnReader.isNextNull()) {
- * // Skip the null value
- * columnReader.skipNext();
- * handleNullValue();
- * } else {
- * // Read the primitive int value (no boxing)
- * int value = columnReader.nextInt();
- * processIntValue(value);
- * }
- * }
- * } else {
- * // Fallback to Pattern 1 if not INT type
- * }
- *
- * }</pre>
- *
- * <h3>Pattern 3: Random Access by Document ID</h3>
- * <p>This pattern uses {@link #getTotalDocs()} to get the total number of documents, then uses
- * document ID-based accessors like {@link #getInt(int)}, {@link #getLong(int)}, etc. to read
- * specific values. Use {@link #isNull(int)} to check if a value is null before reading.
- * This pattern is useful when you need random access or want to process documents in a specific order.
- *
- * <pre>{@code
- *
- * // Random access example - read specific document IDs
- * int[] docIdsToRead = {5, 10, 15, 20};
- * for (int docId : docIdsToRead) {
- * if (!columnReader.isNull(docId)) {
- * int value = columnReader.getInt(docId);
- * processSpecificDoc(docId, value);
- * }
- * }
- *
- * // Read in reverse order
- * // Get the total number of documents
- * int totalDocs = columnReader.getTotalDocs();
- * for (int docId = totalDocs - 1; docId >= 0; docId--) {
- * if (!columnReader.isNull(docId)) {
- * int value = columnReader.getInt(docId);
- * processReverseOrder(docId, value);
- * }
- * }
- * }</pre>
- *
- * <h3>Choosing the Right Pattern</h3>
- * <ul>
- * <li><b>Pattern 1</b>: Use when dealing with generic Object types or when you don't know
- * the column type at compile time. Less efficient due to boxing.</li>
- * <li><b>Pattern 2</b>: Use when you know the column type and want efficient sequential iteration
- * with primitive types. Preferred for most columnar segment building scenarios.</li>
- * <li><b>Pattern 3</b>: Use when you need random access, want to process documents in a specific
- * order, or need to access the same document multiple times.</li>
- * </ul>
- *
- */
+import org.apache.pinot.spi.data.FieldSpec.DataType;
+import org.apache.pinot.spi.utils.BooleanUtils;
+import org.apache.pinot.spi.utils.PinotDataType;
+
+
+/// Reads the values of a single column by document ID, for columnar segment building. Where a `RecordReader` exposes a
+/// data source row-by-row, a `ColumnReader` exposes one column at a time: a caller reads every value of a column before
+/// moving to the next one. Building a segment one fully-materialized column at a time is cheaper than transposing row
+/// records when the source is already columnar (e.g. Arrow, Parquet).
+///
+/// Documents are addressed by a 0-based id, from 0 (inclusive) to [#getTotalDocs()] (exclusive). Two access patterns
+/// are supported:
+/// - **Random access** — [#getValue(int)] and the typed accessors may be called for any document id, in any order.
+/// - **Repeated sequential traversal** — the full range may be traversed by ascending document id more than once. The
+/// reader starts positioned at document 0; call [#rewind()] to reset it before each subsequent traversal. The
+/// segment build uses this: a statistics pass, then an index-writing pass over the same reader.
+///
+/// To read a column:
+/// 1. Call [#getValueType()] once. A non-`null` result names the type that can be read directly through the matching
+/// type-specific accessor; `null` means the column must be read through the boxed [#getValue(int)].
+/// 2. For each document id, call [#isNull(int)] first — type-specific accessors cannot represent null and must not be
+/// called for a null value.
+/// 3. Read each non-null value with the accessor chosen in step 1, e.g. [#getInt(int)] / [#getIntMV(int)] when
+/// [#getValueType()] is [PinotDataType#INT] / [PinotDataType#INT_ARRAY], or [#getValue(int)] otherwise.
+///
+/// Implementations perform any conversion between the source encoding and the returned Pinot type, are not required to
+/// be thread-safe, and must be released with [#close()] when no longer needed.
+///
+/// ```
+/// PinotDataType valueType = columnReader.getValueType();
+/// for (int docId = 0; docId < columnReader.getTotalDocs(); docId++) {
+/// if (columnReader.isNull(docId)) {
+/// continue;
+/// }
+/// if (valueType == PinotDataType.INT) {
+/// consumeInt(columnReader.getInt(docId));
+/// } else {
+/// consumeObject(columnReader.getValue(docId));
+/// }
+/// }
+/// ```
public interface ColumnReader extends Closeable, Serializable {
- /**
- * Return <code>true</code> if more values remain to be read in this column.
- * <p>This method should not throw exception. Caller is not responsible for handling exceptions from this method.
- */
- boolean hasNext();
+ /// Returns the name of the column read by this reader.
+ String getColumnName();
- /**
- * Get the next value in the column.
- * <p>This method should be called only if {@link #hasNext()} returns <code>true</code>. Caller is responsible for
- * handling exceptions from this method and skip the value if user wants to continue reading the remaining values.
- *
- * @return Next column value, or null if the value is null
- * @throws IOException If an I/O error occurs while reading
- */
+ /// Returns the [PinotDataType] that [#getValue(int)] produces for this column, when that type has a matching typed
+ /// accessor; otherwise `null`.
+ ///
+ /// A non-`null` result `T` therefore names an accessor that returns exactly what [#getValue(int)] returns, only
+ /// unboxed: for every document id where [#isNull(int)] is `false`, the accessor matching `T` — e.g. [#getInt(int)]
+ /// for [PinotDataType#INT], [#getIntMV(int)] for [PinotDataType#INT_ARRAY] — returns the same value as
+ /// [#getValue(int)]. `null` means [#getValue(int)]'s type has no typed accessor; read the column through
+ /// [#getValue(int)].
+ ///
+ /// The result is a constant property of the column (independent of the document id) and, when non-`null`, is one of
+ /// the accessor-backed types — [PinotDataType#INT], [PinotDataType#LONG], [PinotDataType#FLOAT],
+ /// [PinotDataType#DOUBLE], [PinotDataType#BIG_DECIMAL], [PinotDataType#STRING], [PinotDataType#BYTES] — or their
+ /// `_ARRAY` variants for a multi-value column.
+ ///
+ /// Because [#getValue(int)] returns the column's logical type, a logical type whose object is not itself an
+ /// accessor-backed type returns `null`: `BOOLEAN` (a `Boolean`), `TIMESTAMP` (a `Timestamp`), and the complex types.
+ /// A `JSON` column, stored and read as its text `String`, reports [PinotDataType#STRING]. Implementations keyed on
+ /// the stored [DataType] can use [#toValueType(DataType, boolean)].
Review Comment:
Addressed in the latest push. `getValueType()` now reports `null` for `TIMESTAMP` uniformly — the getters no longer special-case raw extraction, so a raw epoch value is always read through `getValue()` (a raw `Long` under `extractRawTimeValues`, a `Timestamp` otherwise). The contract doc now matches the code.
##########
pinot-plugins/pinot-input-format/pinot-arrow/src/main/java/org/apache/pinot/plugin/inputformat/arrow/ArrowToPinotTypeConverter.java:
##########
@@ -59,6 +60,53 @@ public final class ArrowToPinotTypeConverter {
private ArrowToPinotTypeConverter() {
}
+ /// Returns the [PinotDataType] that the given Arrow `elementType` (a scalar element type, already unwrapped from any
+ /// list) can be read as through a type-specific `ColumnReader` accessor, or `null` when it has no dedicated accessor
+ /// and must be read through `ColumnReader.getValue()`. Decided from the logical Arrow type, not the physical vector:
+ /// logical `Bool`, temporal types (unless `extractRawTimeValues` surfaces a `Timestamp`'s raw epoch `long`), and
+ /// complex types have no dedicated accessor.
+ ///
+ /// @param elementType scalar Arrow type of the column (or of a list's elements)
+ /// @param singleValue `true` for a single-value column (scalar type), `false` for the `_ARRAY` variant
+ /// @param extractRawTimeValues when `true`, a `Timestamp` column is readable as `LONG` (its raw epoch value);
+ /// otherwise it has no dedicated accessor
+ @Nullable
+ public static PinotDataType toValueType(ArrowType elementType, boolean singleValue, boolean extractRawTimeValues) {
+ switch (elementType.getTypeID()) {
+ case Int:
+ switch (((ArrowType.Int) elementType).getBitWidth()) {
+ case 32:
+ return singleValue ? PinotDataType.INT : PinotDataType.INT_ARRAY;
+ case 64:
+ return singleValue ? PinotDataType.LONG : PinotDataType.LONG_ARRAY;
Review Comment:
Addressed in the latest push. `toValueType()` now switches on `Types.MinorType` rather than `ArrowType`, matching only the signed `INT` (32-bit) and `BIGINT` (64-bit) minor types. Unsigned vectors (`UINT1`/`UINT2`/`UINT4`/`UINT8`) are distinct minor types that fall through to `null` and are read via `getValue()`, so there is no signed/unsigned accessor mismatch.
##########
pinot-plugins/pinot-input-format/pinot-arrow/src/main/java/org/apache/pinot/plugin/inputformat/arrow/ArrowToPinotTypeConverter.java:
##########
@@ -59,6 +60,53 @@ public final class ArrowToPinotTypeConverter {
private ArrowToPinotTypeConverter() {
}
+ /// Returns the [PinotDataType] that the given Arrow `elementType` (a scalar element type, already unwrapped from any
+ /// list) can be read as through a type-specific `ColumnReader` accessor, or `null` when it has no dedicated accessor
+ /// and must be read through `ColumnReader.getValue()`. Decided from the logical Arrow type, not the physical vector:
+ /// logical `Bool`, temporal types (unless `extractRawTimeValues` surfaces a `Timestamp`'s raw epoch `long`), and
+ /// complex types have no dedicated accessor.
+ ///
+ /// @param elementType scalar Arrow type of the column (or of a list's elements)
+ /// @param singleValue `true` for a single-value column (scalar type), `false` for the `_ARRAY` variant
+ /// @param extractRawTimeValues when `true`, a `Timestamp` column is readable as `LONG` (its raw epoch value);
+ /// otherwise it has no dedicated accessor
+ @Nullable
+ public static PinotDataType toValueType(ArrowType elementType, boolean singleValue, boolean extractRawTimeValues) {
+ switch (elementType.getTypeID()) {
+ case Int:
+ switch (((ArrowType.Int) elementType).getBitWidth()) {
+ case 32:
+ return singleValue ? PinotDataType.INT : PinotDataType.INT_ARRAY;
+ case 64:
+ return singleValue ? PinotDataType.LONG : PinotDataType.LONG_ARRAY;
+ default:
+ return null;
+ }
+ case FloatingPoint:
+ switch (((ArrowType.FloatingPoint) elementType).getPrecision()) {
+ case SINGLE:
+ return singleValue ? PinotDataType.FLOAT : PinotDataType.FLOAT_ARRAY;
+ case DOUBLE:
+ return singleValue ? PinotDataType.DOUBLE : PinotDataType.DOUBLE_ARRAY;
+ default:
+ return null;
+ }
+ case Decimal:
+ return singleValue ? PinotDataType.BIG_DECIMAL : PinotDataType.BIG_DECIMAL_ARRAY;
+ case Utf8:
+ case LargeUtf8:
+ return singleValue ? PinotDataType.STRING : PinotDataType.STRING_ARRAY;
+ case Binary:
+ case LargeBinary:
+ case FixedSizeBinary:
+ return singleValue ? PinotDataType.BYTES : PinotDataType.BYTES_ARRAY;
Review Comment:
Addressed in the latest push. The `MinorType` switch matches only `VARCHAR` and `VARBINARY`. `LARGEVARCHAR` / `LARGEVARBINARY` / `FIXEDSIZEBINARY` are distinct minor types that fall through to `null` and are read through `getValue()`, so the typed accessors only serve the `VarCharVector` / `VarBinaryVector` they actually handle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Simplify ColumnReader SPI: drop iterator API, replace type checks with getValueType() [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #18918:
URL: https://github.com/apache/pinot/pull/18918
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org