You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by xi...@apache.org on 2026/08/10 08:09:04 UTC
(pinot) branch master updated: Expose common segment metadata as built-in virtual columns (#19179)
This is an automated email from the ASF dual-hosted git repository.
xiangfu0 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new a523b7beae2 Expose common segment metadata as built-in virtual columns (#19179)
a523b7beae2 is described below
commit a523b7beae27a25c4e18f43ccac38921b3e894ce
Author: Xiang Fu <xi...@apache.org>
AuthorDate: Mon Aug 10 01:08:58 2026 -0700
Expose common segment metadata as built-in virtual columns (#19179)
* Expose common segment metadata as built-in virtual columns
Adds five built-in virtual columns that surface segment metadata to
queries, alongside the existing $docId / $hostName / $segmentName /
$partitionId:
| Column | Type | Value |
|-----------------|--------|----------------------------------------------|
| $creationTime | LONG | segment creation time, epoch millis |
| $startTimeMs | LONG | segment time range start, epoch millis |
| $endTimeMs | LONG | segment time range end, epoch millis |
| $totalDocs | INT | documents stored in the segment |
| $segmentCrc | STRING | segment CRC |
Example:
SELECT $segmentName, $segmentCrc, $totalDocs, $creationTime
FROM myTable GROUP BY 1, 2, 3, 4
Each column is a constant single-value column within a segment. Values
are read from SegmentMetadata every time the column is built rather than
baked into the field spec, so mutable segments - which rebuild their
virtual data sources on every access - always observe current metadata.
Metadata that genuinely does not exist yet reads as SQL NULL rather than
a sentinel: a CONSUMING segment has no time range and no CRC until it is
committed. This required teaching the virtual column path to carry a null
value vector - VirtualColumnIndexContainer now serves
StandardIndexes.nullValueVector() and VirtualColumnProvider gained a
buildNullValueVector hook (defaulting to no nulls, so existing providers
are unaffected). Without it the placeholder stored in the forward index
would be indistinguishable from a real value once null handling is on,
e.g. MIN($startTimeMs) returning Long.MIN_VALUE on a consuming segment.
$totalDocs counts the documents physically stored in the segment, so for
an upsert table it also includes documents that have been replaced and
are no longer returned by queries.
Naming note: only the time range columns carry the "Ms" suffix.
SegmentMetadata#getStartTime()/#getEndTime() return values in the time
column's own unit, so unsuffixed names would be ambiguous; a creation
time is always epoch millis throughout Pinot.
Supporting changes:
- New BuiltInVirtualColumns in pinot-spi is the single source of each
column's name, data type and single-value/multi-value shape. The broker
side (TableCache#addBuiltInVirtualColumns) and the server side
(VirtualColumnProviderFactory#addBuiltInVirtualColumnsToSegmentSchema)
now both build their field specs from it, so the two can no longer
disagree on a type. Previously these specs were declared twice by hand.
- DefaultNullValueVirtualColumnProvider grew an overridable
getValue(context) so it can back any per-segment constant column, plus
a type check that names the offending column and provider instead of
throwing a bare ClassCastException from inside segment loading.
Behavior is unchanged for existing callers.
- Fixes a pre-existing bug in SchemaInfo, which computed
getDimensionFieldSpecs().size() - 3 with a comment naming three virtual
columns. That has been off by one since $partitionId was added, and
these five would have made GET /schemas/info over-report user dimension
counts by six in the controller UI. It now excludes built-in virtual
columns by name.
* Use TIMESTAMP for the segment time columns, and address review feedback
Follow-up to the segment metadata virtual columns, covering the review
comments and the CI failures on the first revision.
Column types and names
----------------------
The three time columns are now TIMESTAMP rather than LONG, so the unit is
carried by the type instead of by the column name and query results render
them as readable timestamps instead of raw millis. That removes the reason
the "Ms" suffix existed, so the columns are back to their natural names:
$creationTime TIMESTAMP
$startTime TIMESTAMP
$endTime TIMESTAMP
$totalDocs INT
$crc STRING
$segmentCrc is renamed to $crc so that the five names are consistent: none
of them repeats the "segment" scope, which every one of them shares.
Correctness
-----------
The segment metadata used to be read three separate times per build - once
for the dictionary, once for the column metadata, and once for the null
value vector - with nothing tying the three reads together. On a mutable
segment, whose metadata is live, they could disagree and the column would
then serve its placeholder as if it were a real value. The value is now
resolved exactly once per build and shared by all three.
VirtualColumnIndexContainer#close now also closes the null value vector.
Structure
---------
- The constant-value machinery moves out of DefaultNullValueVirtualColumnProvider
into a new BaseConstantValueVirtualColumnProvider in the virtualcolumn
package. DefaultNullValueVirtualColumnProvider becomes a thin subclass; its
fully-qualified name is stored in field specs and resolved reflectively, so
it keeps its name and package.
- VirtualColumnProviderFactory resolves providers through a map whose key set
is checked against the column definitions in a static initializer. A column
added without a provider now fails at class load instead of aborting every
segment load on every server.
- BuiltInVirtualColumns is renamed to BuiltInVirtualColumnDefinitions: the old
name differed from the pre-existing BuiltInVirtualColumn by a single trailing
letter, and VirtualColumnProviderFactory imports both.
Test fixes
----------
Calcite plans reference fields positionally, and all $-prefixed columns sort
first in a table's row type, so five new virtual columns shift every ordinal
in an EXPLAIN assertion by five. Updated in NullHandlingIntegrationTest,
MultiStageEngineExplainIntegrationTest and OfflineClusterIntegrationTest.
MultiNodesOfflineClusterIntegrationTest inherits the latter's fixes.
The aggregate metadata API reports one entry per column of a loaded segment,
which includes the virtual columns, so its expected count grows from 83 to 88.
Test coverage
-------------
- BuiltInVirtualColumnDefinitionsTest pins the definitions against the
declared names, and pins that field specs are never shared between schemas.
- A negative test covers the value type check.
- The CONSUMING-segment queries in BaseClusterIntegrationTestSet now assert
their results instead of only checking that nothing throws. On a hybrid
table the time boundary can hide rows that are physically present in a
segment, so the assertion is that a segment's $totalDocs is at least the
number of rows the query returns from it.
- OfflineClusterIntegrationTest additionally checks the millisecond
normalization per segment, filtering by $crc, and that null handling sees
no nulls on a table where all the metadata is available.
* Address review: suppress min/max for NULL columns, expose $crc as LONG
Fixes found by review on top of the segment metadata virtual columns.
Do not publish min/max for a column that reads as NULL
------------------------------------------------------
When the segment metadata is unavailable the column stores a placeholder
and reports every document as null, but it was still publishing that
placeholder as the column's min/max. Segment pruners read min/max without
consulting the null value vector, so:
SET enableNullHandling=true;
SELECT $segmentName, $creationTime FROM myTable ORDER BY $creationTime ASC LIMIT 10
let a CONSUMING segment - whose $creationTime placeholder is the epoch -
sort first in SelectionQuerySegmentPruner, consume the whole LIMIT, and
prune away every committed segment whose min was greater. The query then
returned rows that are all SQL NULL, which under NULLS-LAST ordering should
have sorted last. $crc had the mirror problem for ORDER BY ... DESC.
The min/max are now left unset whenever the value is a placeholder. Both
SelectionQuerySegmentPruner and ColumnValueSegmentPruner already keep a
segment that reports no min/max, which is the correct conservative
behavior for an all-null column.
Resolve the metadata once per data source, not once per index
-------------------------------------------------------------
buildDataSource calls buildMetadata and buildColumnIndexContainer
separately, so the column metadata came from a different read of
SegmentMetadata than the dictionary and the null value vector. Only the
container was internally consistent. BaseSegmentMetadataVirtualColumnProvider
now overrides buildDataSource and resolves the value once for all three.
Apply the value type check on the path that is actually taken
------------------------------------------------------------
The check was only reachable through the no-argument buildDictionary /
buildMetadata. Every one of the five new providers goes through the
value-taking overloads, which skipped it, so the diagnostic it exists to
give never applied to them. The check now runs on every path.
$crc is a LONG
--------------
A CRC is a long everywhere else in Pinot - SegmentZKMetadata#getCrc returns
one, and SegmentMetadata#getCrc merely renders that long as a String. Typing
the column STRING to match the latter left users unable to write
WHERE $crc = 12345, and contradicted the reasoning used for the time
columns, which are TIMESTAMP precisely so the type carries the semantics.
Other review fixes
------------------
- AllNullValueVector was a byte-for-byte duplicate of a nested class in
OpenStructNullDataSource. Both now share
AllNullValueVectorReader in the readers package.
- SchemaUtils.validate rejects a user column named after a built-in virtual
column. Such a column is filtered out of the segment's physical columns
when the metadata is read, and the virtual provider then takes over the
name, so queries would silently return segment metadata instead of the
user's data.
- The built-in providers are stateless, so they are now shared instances
rather than reflectively constructed once per virtual column per segment.
Segment load did that 9 times per segment after this feature, up from 4.
- Removed the static initializer asserting provider coverage. The factory is
only ever loaded during segment load, so it did not fail any earlier than
the per-column check, and it turned every subsequent load into a bare
NoClassDefFoundError. The invariant is now asserted in a test.
- Corrected the AllNullValueVectorReader publication comment:
toImmutableRoaringBitmap() returns `this`, so safety rests on the volatile
write alone and the returned bitmap must be treated as read-only.
- Imported SegmentMetadata rather than naming it inline in Javadoc.
Naming note: the reviewers proposed prefixing all five columns with
"segment" ($segmentCreationTime, ...) to match $segmentName. The terse forms
are a deliberate choice, recorded here so it is not re-litigated as an
oversight.
---
.../pinot/common/config/provider/TableCache.java | 17 +-
.../PinotQueryResourceStaticValidationTest.java | 5 +-
.../pinot/controller/helix/TableCacheTest.java | 12 +-
.../tests/BaseClusterIntegrationTestSet.java | 40 ++
.../MultiStageEngineExplainIntegrationTest.java | 6 +-
.../tests/NullHandlingIntegrationTest.java | 8 +-
.../tests/OfflineClusterIntegrationTest.java | 82 ++++-
.../DefaultNullValueVirtualColumnProvider.java | 118 +-----
.../index/openstruct/OpenStructNullDataSource.java | 29 +-
.../index/readers/AllNullValueVectorReader.java | 60 +++
.../BaseConstantValueVirtualColumnProvider.java | 220 +++++++++++
.../BaseSegmentMetadataVirtualColumnProvider.java | 97 +++++
.../SegmentCrcVirtualColumnProvider.java | 49 +++
.../SegmentCreationTimeVirtualColumnProvider.java | 38 ++
.../SegmentEndTimeVirtualColumnProvider.java | 38 ++
.../SegmentStartTimeVirtualColumnProvider.java | 38 ++
.../SegmentTotalDocsVirtualColumnProvider.java | 35 ++
.../virtualcolumn/VirtualColumnIndexContainer.java | 18 +
.../virtualcolumn/VirtualColumnProvider.java | 14 +-
.../VirtualColumnProviderFactory.java | 86 ++++-
.../pinot/segment/local/utils/SchemaUtils.java | 7 +
.../mutable/MutableSegmentImplRawMVTest.java | 13 +-
.../mutable/MutableSegmentImplTest.java | 37 +-
.../local/segment/index/loader/LoaderTest.java | 38 +-
.../SegmentMetadataVirtualColumnProviderTest.java | 405 +++++++++++++++++++++
.../spi/data/BuiltInVirtualColumnDefinitions.java | 107 ++++++
.../java/org/apache/pinot/spi/data/SchemaInfo.java | 13 +-
.../apache/pinot/spi/utils/CommonConstants.java | 31 +-
.../data/BuiltInVirtualColumnDefinitionsTest.java | 99 +++++
.../org/apache/pinot/spi/data/SchemaInfoTest.java | 24 ++
30 files changed, 1590 insertions(+), 194 deletions(-)
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/config/provider/TableCache.java b/pinot-common/src/main/java/org/apache/pinot/common/config/provider/TableCache.java
index 02c5e7cf313..2e62e80909d 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/config/provider/TableCache.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/config/provider/TableCache.java
@@ -31,11 +31,9 @@ import org.apache.pinot.spi.config.provider.SchemaChangeListener;
import org.apache.pinot.spi.config.provider.TableConfigChangeListener;
import org.apache.pinot.spi.config.table.QueryConfig;
import org.apache.pinot.spi.config.table.TableConfig;
-import org.apache.pinot.spi.data.DimensionFieldSpec;
-import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.BuiltInVirtualColumnDefinitions;
import org.apache.pinot.spi.data.LogicalTableConfig;
import org.apache.pinot.spi.data.Schema;
-import org.apache.pinot.spi.utils.CommonConstants.Segment.BuiltInVirtualColumn;
import org.apache.pinot.spi.utils.TimestampIndexUtils;
import org.apache.pinot.sql.parsers.CalciteSqlParser;
import org.slf4j.Logger;
@@ -119,18 +117,7 @@ public interface TableCache extends PinotConfigProvider {
/// Adds the built-in virtual columns to the schema.
/// NOTE: The virtual column provider class is not added.
default void addBuiltInVirtualColumns(Schema schema) {
- if (!schema.hasColumn(BuiltInVirtualColumn.DOCID)) {
- schema.addField(new DimensionFieldSpec(BuiltInVirtualColumn.DOCID, FieldSpec.DataType.INT, true));
- }
- if (!schema.hasColumn(BuiltInVirtualColumn.HOSTNAME)) {
- schema.addField(new DimensionFieldSpec(BuiltInVirtualColumn.HOSTNAME, FieldSpec.DataType.STRING, true));
- }
- if (!schema.hasColumn(BuiltInVirtualColumn.SEGMENTNAME)) {
- schema.addField(new DimensionFieldSpec(BuiltInVirtualColumn.SEGMENTNAME, FieldSpec.DataType.STRING, true));
- }
- if (!schema.hasColumn(BuiltInVirtualColumn.PARTITIONID)) {
- schema.addField(new DimensionFieldSpec(BuiltInVirtualColumn.PARTITIONID, FieldSpec.DataType.STRING, false));
- }
+ BuiltInVirtualColumnDefinitions.addToSchema(schema);
}
static Map<Expression, Expression> createExpressionOverrideMap(String physicalOrLogicalTableName,
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotQueryResourceStaticValidationTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotQueryResourceStaticValidationTest.java
index 45dde807c67..0bd7222cb1c 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotQueryResourceStaticValidationTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/PinotQueryResourceStaticValidationTest.java
@@ -26,6 +26,7 @@ import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.config.table.TableType;
import org.apache.pinot.spi.data.FieldSpec;
import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.CommonConstants;
import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
import org.mockito.MockitoAnnotations;
import org.testng.Assert;
@@ -62,7 +63,9 @@ public class PinotQueryResourceStaticValidationTest {
Assert.assertNotNull(provider.getTableConfig("testTable_OFFLINE"));
Assert.assertNotNull(provider.getSchema("testTable"));
Assert.assertNotNull(provider.getColumnNameMap("testTable"));
- Assert.assertEquals(provider.getColumnNameMap("testTable").size(), 6); // 2 columns + 4 built-in virtual columns
+ // 2 columns + all built-in virtual columns
+ Assert.assertEquals(provider.getColumnNameMap("testTable").size(),
+ 2 + CommonConstants.Segment.BuiltInVirtualColumn.BUILT_IN_VIRTUAL_COLUMNS.size());
Assert.assertTrue(provider.getTableNameMap().containsKey("testTable_OFFLINE"));
Assert.assertTrue(provider.getTableNameMap().containsKey("testTable"));
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/TableCacheTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/TableCacheTest.java
index 9eeff07c69f..ca1e4861512 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/TableCacheTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/TableCacheTest.java
@@ -303,6 +303,11 @@ public class TableCacheTest {
expectedColumnMap.put(isCaseInsensitive ? "$hostname" : "$hostName", "$hostName");
expectedColumnMap.put(isCaseInsensitive ? "$segmentname" : "$segmentName", "$segmentName");
expectedColumnMap.put(isCaseInsensitive ? "$partitionid" : "$partitionId", "$partitionId");
+ expectedColumnMap.put(isCaseInsensitive ? "$creationtime" : "$creationTime", "$creationTime");
+ expectedColumnMap.put(isCaseInsensitive ? "$starttime" : "$startTime", "$startTime");
+ expectedColumnMap.put(isCaseInsensitive ? "$endtime" : "$endTime", "$endTime");
+ expectedColumnMap.put(isCaseInsensitive ? "$totaldocs" : "$totalDocs", "$totalDocs");
+ expectedColumnMap.put("$crc", "$crc");
return expectedColumnMap;
}
@@ -311,7 +316,12 @@ public class TableCacheTest {
.addSingleValueDimension(BuiltInVirtualColumn.DOCID, DataType.INT)
.addSingleValueDimension(BuiltInVirtualColumn.HOSTNAME, DataType.STRING)
.addSingleValueDimension(BuiltInVirtualColumn.SEGMENTNAME, DataType.STRING)
- .addMultiValueDimension(BuiltInVirtualColumn.PARTITIONID, DataType.STRING).build();
+ .addMultiValueDimension(BuiltInVirtualColumn.PARTITIONID, DataType.STRING)
+ .addSingleValueDimension(BuiltInVirtualColumn.CREATIONTIME, DataType.TIMESTAMP)
+ .addSingleValueDimension(BuiltInVirtualColumn.STARTTIME, DataType.TIMESTAMP)
+ .addSingleValueDimension(BuiltInVirtualColumn.ENDTIME, DataType.TIMESTAMP)
+ .addSingleValueDimension(BuiltInVirtualColumn.TOTALDOCS, DataType.INT)
+ .addSingleValueDimension(BuiltInVirtualColumn.CRC, DataType.LONG).build();
}
@DataProvider(name = "testTableCacheDataProvider")
diff --git a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTestSet.java b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTestSet.java
index 8a52029a2ab..d13a9723214 100644
--- a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTestSet.java
+++ b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTestSet.java
@@ -22,6 +22,7 @@ import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
+import java.sql.Timestamp;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
@@ -465,6 +466,45 @@ public abstract class BaseClusterIntegrationTestSet extends BaseClusterIntegrati
"select $docId, $segmentName, $hostName, $partitionId from mytable where $docId = 5 limit 50");
getPinotConnection().execute(
"select $docId, $segmentName, $hostName, $partitionId from mytable where $docId > 19998 limit 50");
+
+ // Segment metadata virtual columns. This method is the only place they are exercised against a table that can
+ // have CONSUMING segments, so assert the results rather than just checking that nothing throws.
+ long numTotalDocs = getCountStarResult();
+
+ // $totalDocs is the number of documents stored in the segment. On a hybrid table the broker's time boundary can
+ // hide some of them, so a segment's $totalDocs is at least the number of rows the query sees from it, never less.
+ ResultSet perSegment = getPinotConnection().execute(
+ "select $segmentName, max($totalDocs), count(*) from mytable group by $segmentName limit 10000")
+ .getResultSet(0);
+ assertTrue(perSegment.getRowCount() > 0);
+ long visibleRows = 0;
+ for (int i = 0; i < perSegment.getRowCount(); i++) {
+ String segmentName = perSegment.getString(i, 0);
+ // MAX()/COUNT() render as floating point values in the single-stage engine, so read them as doubles in both
+ long totalDocsInSegment = (long) Double.parseDouble(perSegment.getString(i, 1));
+ long rowsFromSegment = (long) Double.parseDouble(perSegment.getString(i, 2));
+ assertTrue(totalDocsInSegment > 0, "Unexpected $totalDocs: " + totalDocsInSegment + " for: " + segmentName);
+ assertTrue(totalDocsInSegment >= rowsFromSegment,
+ "$totalDocs: " + totalDocsInSegment + " is below the " + rowsFromSegment + " rows returned by: "
+ + segmentName);
+ visibleRows += rowsFromSegment;
+ }
+ assertEquals(visibleRows, numTotalDocs, "Grouping by $segmentName should cover every row exactly once");
+
+ // Every segment - CONSUMING included - is created with a creation time, so none of them falls back to the epoch
+ // placeholder used when the metadata is unavailable
+ ResultSet creationTimes = getPinotConnection()
+ .execute("select $segmentName, $creationTime from mytable group by $segmentName, $creationTime limit 10000")
+ .getResultSet(0);
+ assertTrue(creationTimes.getRowCount() > 0);
+ for (int i = 0; i < creationTimes.getRowCount(); i++) {
+ String creationTime = creationTimes.getString(i, 1);
+ assertTrue(Timestamp.valueOf(creationTime).getTime() > 0,
+ "Unexpected $creationTime: " + creationTime + " for segment: " + creationTimes.getString(i, 0));
+ }
+
+ // Selecting them must not fail on any segment type
+ getPinotConnection().execute("select $creationTime, $startTime, $endTime, $totalDocs, $crc from mytable limit 50");
}
/// Test queries from the query file.
diff --git a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/MultiStageEngineExplainIntegrationTest.java b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/MultiStageEngineExplainIntegrationTest.java
index 69d9861a5b4..1873fe8d1db 100644
--- a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/MultiStageEngineExplainIntegrationTest.java
+++ b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/MultiStageEngineExplainIntegrationTest.java
@@ -178,7 +178,7 @@ public class MultiStageEngineExplainIntegrationTest extends BaseClusterIntegrati
"Execution Plan\n"
+ "PinotLogicalAggregate(group=[{}], agg#0=[SUM($0)], aggType=[FINAL])\n"
+ " PinotLogicalExchange(distribution=[hash])\n"
- + " PinotLogicalAggregate(group=[{}], agg#0=[SUM($6)], aggType=[LEAF])\n"
+ + " PinotLogicalAggregate(group=[{}], agg#0=[SUM($11)], aggType=[LEAF])\n"
+ " PinotLogicalTableScan(table=[[default, mytable]])\n");
// Enable PinotAggregateFunctionRewriteRule through query option, ensure it overrides the broker config
@@ -186,7 +186,7 @@ public class MultiStageEngineExplainIntegrationTest extends BaseClusterIntegrati
"Execution Plan\n"
+ "PinotLogicalAggregate(group=[{}], agg#0=[SUMLONG($0)], aggType=[FINAL])\n"
+ " PinotLogicalExchange(distribution=[hash])\n"
- + " PinotLogicalAggregate(group=[{}], agg#0=[SUMLONG($6)], aggType=[LEAF])\n"
+ + " PinotLogicalAggregate(group=[{}], agg#0=[SUMLONG($11)], aggType=[LEAF])\n"
+ " PinotLogicalTableScan(table=[[default, mytable]])\n",
Map.of("usePlannerRules", CommonConstants.Broker.PlannerRuleNames.AGGREGATE_FUNCTION_REWRITE));
@@ -197,7 +197,7 @@ public class MultiStageEngineExplainIntegrationTest extends BaseClusterIntegrati
+ "LogicalProject(EXPR$0=[CASE(=($1, 0), null:BIGINT, $0)])\n"
+ " PinotLogicalAggregate(group=[{}], agg#0=[SUMLONG($0)], agg#1=[COUNT($1)], aggType=[FINAL])\n"
+ " PinotLogicalExchange(distribution=[hash])\n"
- + " PinotLogicalAggregate(group=[{}], agg#0=[SUMLONG($6)], agg#1=[COUNT()], aggType=[LEAF])\n"
+ + " PinotLogicalAggregate(group=[{}], agg#0=[SUMLONG($11)], agg#1=[COUNT()], aggType=[LEAF])\n"
+ " PinotLogicalTableScan(table=[[default, mytable]])\n",
Map.of("usePlannerRules", CommonConstants.Broker.PlannerRuleNames.AGGREGATE_FUNCTION_REWRITE + ","
+ CommonConstants.Broker.PlannerRuleNames.AGGREGATE_REDUCE_FUNCTIONS));
diff --git a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/NullHandlingIntegrationTest.java b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/NullHandlingIntegrationTest.java
index 8388fc46722..c158d6913cf 100644
--- a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/NullHandlingIntegrationTest.java
+++ b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/NullHandlingIntegrationTest.java
@@ -434,7 +434,7 @@ public class NullHandlingIntegrationTest extends BaseClusterIntegrationTestSet
explainLogical(query,
"Execution Plan\n"
+ "LogicalProject(EXPR$0=[1])\n"
- + " LogicalFilter(condition=[AND(IS NOT NULL($8), <>($8, 0))])\n"
+ + " LogicalFilter(condition=[AND(IS NOT NULL($13), <>($13, 0))])\n"
+ " PinotLogicalTableScan(table=[[default, mytable]])\n",
Map.of(CommonConstants.Broker.Request.QueryOptionKey.ENABLE_NULL_HANDLING, "false"));
}
@@ -452,7 +452,7 @@ public class NullHandlingIntegrationTest extends BaseClusterIntegrationTestSet
explainLogical(query,
"Execution Plan\n"
+ "LogicalProject(EXPR$0=[1])\n"
- + " LogicalFilter(condition=[<>($8, 0)])\n"
+ + " LogicalFilter(condition=[<>($13, 0)])\n"
+ " PinotLogicalTableScan(table=[[default, mytable]])\n",
Map.of(CommonConstants.Broker.Request.QueryOptionKey.ENABLE_NULL_HANDLING, "true"));
}
@@ -470,7 +470,7 @@ public class NullHandlingIntegrationTest extends BaseClusterIntegrationTestSet
explainLogical(query,
"Execution Plan\n"
+ "LogicalProject(EXPR$0=[1])\n"
- + " LogicalFilter(condition=[AND(IS NULL($8), <>($8, 0))])\n"
+ + " LogicalFilter(condition=[AND(IS NULL($13), <>($13, 0))])\n"
+ " PinotLogicalTableScan(table=[[default, mytable]])\n",
Map.of(CommonConstants.Broker.Request.QueryOptionKey.ENABLE_NULL_HANDLING, "false"));
}
@@ -537,7 +537,7 @@ public class NullHandlingIntegrationTest extends BaseClusterIntegrationTestSet
+ "PinotLogicalAggregate(group=[{0}], agg#0=[COUNT($1)], agg#1=[COUNT($2)], aggType=[FINAL])\n"
+ " PinotLogicalExchange(distribution=[hash[0]])\n"
+ " PinotLogicalAggregate(group=[{0}], agg#0=[COUNT()], agg#1=[COUNT() FILTER $1], aggType=[LEAF])\n"
- + " LogicalProject(city=[$5], $f1=[IS TRUE(=($7, _UTF-8'unknown'))])\n"
+ + " LogicalProject(city=[$10], $f1=[IS TRUE(=($12, _UTF-8'unknown'))])\n"
+ " PinotLogicalTableScan(table=[[default, mytable]])\n");
// IS_TRUE should be trimmed off, then the filter becomes always false in the server execution plan
explainAskingServers(query,
diff --git a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java
index 8984ca7027f..6872360a874 100644
--- a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java
+++ b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java
@@ -2449,6 +2449,78 @@ public class OfflineClusterIntegrationTest extends BaseClusterIntegrationTestSet
assertEquals(response.get("numSegmentsMatched").asInt(), numSegmentsToQuery);
}
+ @Test(dataProvider = "useBothQueryEngines")
+ public void testSegmentMetadataVirtualColumns(boolean useMultiStageQueryEngine)
+ throws Exception {
+ setUseMultiStageQueryEngine(useMultiStageQueryEngine);
+
+ String query = "SELECT $creationTime, $startTime, $endTime, $totalDocs, $crc FROM mytable LIMIT 10";
+ JsonNode response = postQuery(query);
+ JsonNode resultTable = response.get("resultTable");
+ JsonNode dataSchema = resultTable.get("dataSchema");
+ assertEquals(dataSchema.get("columnNames").toString(),
+ "[\"$creationTime\",\"$startTime\",\"$endTime\",\"$totalDocs\",\"$crc\"]");
+ // The three time columns are TIMESTAMP, so they render as formatted timestamps rather than raw millis
+ assertEquals(dataSchema.get("columnDataTypes").toString(),
+ "[\"TIMESTAMP\",\"TIMESTAMP\",\"TIMESTAMP\",\"INT\",\"LONG\"]");
+ JsonNode rows = resultTable.get("rows");
+ assertEquals(rows.size(), 10);
+ for (int i = 0; i < 10; i++) {
+ JsonNode row = rows.get(i);
+ long creationTime = Timestamp.valueOf(row.get(0).asText()).getTime();
+ assertTrue(creationTime > 0, "Unexpected segment creation time: " + row.get(0).asText());
+ long startTime = Timestamp.valueOf(row.get(1).asText()).getTime();
+ long endTime = Timestamp.valueOf(row.get(2).asText()).getTime();
+ assertTrue(startTime > 0, "Unexpected segment start time: " + row.get(1).asText());
+ assertTrue(startTime <= endTime, "Segment start time: " + startTime + " is after end time: " + endTime);
+ assertTrue(row.get(3).asInt() > 0, "Unexpected total docs: " + row.get(3).asInt());
+ // A real CRC, not the placeholder the column falls back to when the metadata is unavailable
+ long crc = row.get(4).asLong();
+ assertNotEquals(crc, (long) FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_LONG, "Segment CRC should be a real CRC");
+ }
+
+ // $totalDocs is constant within a segment, so the per-segment values must add up to the total number of rows
+ query = "SELECT $segmentName, MAX($totalDocs) FROM mytable GROUP BY $segmentName LIMIT 10000";
+ rows = postQuery(query).get("resultTable").get("rows");
+ long totalDocs = 0;
+ for (JsonNode row : rows) {
+ totalDocs += row.get(1).asLong();
+ }
+ assertEquals(totalDocs, getCountStarResult());
+
+ // $startTime/$endTime are normalized from the time column's own unit. mytable's time column is DaysSinceEpoch in
+ // DAYS, so each segment's values must be its own day boundaries expressed as timestamps - this is what
+ // distinguishes a correct implementation from one leaking the raw DAYS value.
+ query = "SELECT $segmentName, $startTime, $endTime, MIN(DaysSinceEpoch), MAX(DaysSinceEpoch) FROM mytable "
+ + "GROUP BY $segmentName, $startTime, $endTime LIMIT 10000";
+ rows = postQuery(query).get("resultTable").get("rows");
+ assertFalse(rows.isEmpty());
+ long daysToMillis = TimeUnit.DAYS.toMillis(1);
+ for (JsonNode row : rows) {
+ assertEquals(Timestamp.valueOf(row.get(1).asText()).getTime(), row.get(3).asLong() * daysToMillis,
+ "Unexpected $startTime for segment: " + row.get(0).asText());
+ assertEquals(Timestamp.valueOf(row.get(2).asText()).getTime(), row.get(4).asLong() * daysToMillis,
+ "Unexpected $endTime for segment: " + row.get(0).asText());
+ }
+
+ // Filtering on a single segment's CRC must return exactly the documents grouped under that CRC
+ query = "SELECT $crc, COUNT(*) FROM mytable GROUP BY $crc LIMIT 10000";
+ JsonNode crcRow = postQuery(query).get("resultTable").get("rows").get(0);
+ long segmentCrc = crcRow.get(0).asLong();
+ long docsForCrc = crcRow.get(1).asLong();
+ assertEquals(postQuery("SELECT COUNT(*) FROM mytable WHERE $crc = " + segmentCrc).get("resultTable")
+ .get("rows").get(0).get(0).asLong(), docsForCrc);
+
+ // With null handling on, the engine must consult the null value vector. Every segment of this table has a time
+ // range, a CRC and a creation time, so none of these columns may report a null.
+ for (String column : List.of("$creationTime", "$startTime", "$endTime", "$totalDocs", "$crc")) {
+ assertEquals(postQuery("SET enableNullHandling=true; SELECT COUNT(*) FROM mytable WHERE " + column
+ + " IS NOT NULL").get("resultTable").get("rows").get(0).get(0).asLong(), getCountStarResult(), column);
+ assertEquals(postQuery("SET enableNullHandling=true; SELECT COUNT(*) FROM mytable WHERE " + column
+ + " IS NULL").get("resultTable").get("rows").get(0).get(0).asLong(), 0, column);
+ }
+ }
+
@Test(dataProvider = "useBothQueryEngines")
public void testGroupByUDF(boolean useMultiStageQueryEngine)
throws Exception {
@@ -3581,7 +3653,7 @@ public class OfflineClusterIntegrationTest extends BaseClusterIntegrationTestSet
+ " LogicalProject(count=[$1], name=[$0])\n"
+ " PinotLogicalAggregate(group=[{0}], agg#0=[COUNT($1)], aggType=[FINAL])\n"
+ " PinotLogicalExchange(distribution=[hash[0]])\n"
- + " PinotLogicalAggregate(group=[{18}], agg#0=[COUNT()], aggType=[LEAF])\n"
+ + " PinotLogicalAggregate(group=[{23}], agg#0=[COUNT()], aggType=[LEAF])\n"
+ " PinotLogicalTableScan(table=[[default, mytable]])\n");
assertEquals(response1Json.get("rows").get(0).get(2).asText(), "Rule Execution Times\n"
+ "Rule: SortRemove -> Time:*\n"
@@ -3870,22 +3942,22 @@ public class OfflineClusterIntegrationTest extends BaseClusterIntegrationTestSet
JsonNode starColumnResponse = JsonUtils.objectToJsonNode(
getOrCreateAdminClient().getTableClient().getAggregateMetadata(
TableNameBuilder.OFFLINE.tableNameWithType(getTableName()), "*"));
- validateMetadataResponse(starColumnResponse, 83, 10);
+ validateMetadataResponse(starColumnResponse, 88, 10);
JsonNode starEncodedColumnResponse = JsonUtils.objectToJsonNode(
getOrCreateAdminClient().getTableClient().getAggregateMetadata(
TableNameBuilder.OFFLINE.tableNameWithType(getTableName()), "*"));
- validateMetadataResponse(starEncodedColumnResponse, 83, 10);
+ validateMetadataResponse(starEncodedColumnResponse, 88, 10);
JsonNode starWithExtraColumnResponse = JsonUtils.objectToJsonNode(
getOrCreateAdminClient().getTableClient().getAggregateMetadata(
TableNameBuilder.OFFLINE.tableNameWithType(getTableName()), "CRSElapsedTime,*,OriginStateName"));
- validateMetadataResponse(starWithExtraColumnResponse, 83, 10);
+ validateMetadataResponse(starWithExtraColumnResponse, 88, 10);
JsonNode starWithExtraEncodedColumnResponse = JsonUtils.objectToJsonNode(
getOrCreateAdminClient().getTableClient().getAggregateMetadata(
TableNameBuilder.OFFLINE.tableNameWithType(getTableName()), "CRSElapsedTime,*,OriginStateName"));
- validateMetadataResponse(starWithExtraEncodedColumnResponse, 83, 10);
+ validateMetadataResponse(starWithExtraEncodedColumnResponse, 88, 10);
}
private void validateMetadataResponse(JsonNode response, int numTotalColumn, int numMVColumn) {
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/column/DefaultNullValueVirtualColumnProvider.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/column/DefaultNullValueVirtualColumnProvider.java
index f8db37712bc..05ee8c774d7 100644
--- a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/column/DefaultNullValueVirtualColumnProvider.java
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/column/DefaultNullValueVirtualColumnProvider.java
@@ -18,116 +18,22 @@
*/
package org.apache.pinot.segment.local.segment.index.column;
-import java.math.BigDecimal;
-import org.apache.pinot.segment.local.segment.index.readers.ConstantValueBigDecimalDictionary;
-import org.apache.pinot.segment.local.segment.index.readers.ConstantValueBytesDictionary;
-import org.apache.pinot.segment.local.segment.index.readers.ConstantValueDoubleDictionary;
-import org.apache.pinot.segment.local.segment.index.readers.ConstantValueFloatDictionary;
-import org.apache.pinot.segment.local.segment.index.readers.ConstantValueIntDictionary;
-import org.apache.pinot.segment.local.segment.index.readers.ConstantValueLongDictionary;
-import org.apache.pinot.segment.local.segment.index.readers.ConstantValueStringDictionary;
-import org.apache.pinot.segment.local.segment.index.readers.constant.ConstantMVForwardIndexReader;
-import org.apache.pinot.segment.local.segment.index.readers.constant.ConstantMVInvertedIndexReader;
-import org.apache.pinot.segment.local.segment.index.readers.constant.ConstantSortedIndexReader;
+import org.apache.pinot.segment.local.segment.virtualcolumn.BaseConstantValueVirtualColumnProvider;
import org.apache.pinot.segment.local.segment.virtualcolumn.VirtualColumnContext;
-import org.apache.pinot.segment.local.segment.virtualcolumn.VirtualColumnProvider;
-import org.apache.pinot.segment.spi.index.metadata.ColumnMetadataImpl;
-import org.apache.pinot.segment.spi.index.reader.Dictionary;
-import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
-import org.apache.pinot.segment.spi.index.reader.InvertedIndexReader;
-import org.apache.pinot.spi.data.FieldSpec;
-import org.apache.pinot.spi.utils.ByteArray;
-/// Provide the default null value.
-public class DefaultNullValueVirtualColumnProvider implements VirtualColumnProvider {
+/// Provides the column's default null value for every document.
+///
+/// This is also how `$hostName` and `$segmentName` are served: their value is constant for the segment and is carried
+/// as the field's default null value.
+///
+/// NOTE: The fully-qualified name of this class is stored in field specs (see
+/// `DimensionFieldSpec#DimensionFieldSpec(String, DataType, boolean, Class)`) and resolved reflectively, so it must
+/// not be moved or renamed.
+public class DefaultNullValueVirtualColumnProvider extends BaseConstantValueVirtualColumnProvider {
@Override
- public ForwardIndexReader<?> buildForwardIndex(VirtualColumnContext context) {
- if (context.getFieldSpec().isSingleValueField()) {
- return new ConstantSortedIndexReader(context.getTotalDocCount());
- } else {
- return new ConstantMVForwardIndexReader();
- }
- }
-
- @Override
- public Dictionary buildDictionary(VirtualColumnContext context) {
- FieldSpec fieldSpec = context.getFieldSpec();
- switch (fieldSpec.getDataType().getStoredType()) {
- case INT:
- return new ConstantValueIntDictionary((int) fieldSpec.getDefaultNullValue());
- case LONG:
- return new ConstantValueLongDictionary((long) fieldSpec.getDefaultNullValue());
- case FLOAT:
- return new ConstantValueFloatDictionary((float) fieldSpec.getDefaultNullValue());
- case DOUBLE:
- return new ConstantValueDoubleDictionary((double) fieldSpec.getDefaultNullValue());
- case BIG_DECIMAL:
- return new ConstantValueBigDecimalDictionary((BigDecimal) fieldSpec.getDefaultNullValue());
- case STRING:
- return new ConstantValueStringDictionary((String) fieldSpec.getDefaultNullValue());
- case BYTES:
- case UUID:
- return new ConstantValueBytesDictionary((byte[]) fieldSpec.getDefaultNullValue());
- default:
- throw new IllegalStateException();
- }
- }
-
- @Override
- public InvertedIndexReader<?> buildInvertedIndex(VirtualColumnContext context) {
- if (context.getFieldSpec().isSingleValueField()) {
- return new ConstantSortedIndexReader(context.getTotalDocCount());
- } else {
- return new ConstantMVInvertedIndexReader(context.getTotalDocCount());
- }
- }
-
- @Override
- public ColumnMetadataImpl buildMetadata(VirtualColumnContext context) {
- FieldSpec fieldSpec = context.getFieldSpec();
- ColumnMetadataImpl.Builder builder = new ColumnMetadataImpl.Builder().setFieldSpec(fieldSpec)
- .setTotalDocs(context.getTotalDocCount())
- .setCardinality(1)
- .setHasDictionary(true);
- if (fieldSpec.isSingleValueField()) {
- builder.setSorted(true);
- } else {
- // When there is no value for a multi-value column, the maxNumberOfMultiValues and cardinality should be
- // set as 1 because the MV column bitmap uses 1 to delimit the rows for a MV column. Each MV column will have a
- // default null value based on column's data type
- builder.setMaxNumberOfMultiValues(1);
- }
-
- Object defaultNullValue = fieldSpec.getDefaultNullValue();
- switch (fieldSpec.getDataType().getStoredType()) {
- case INT:
- builder.setMinValue((int) defaultNullValue).setMaxValue((int) defaultNullValue);
- break;
- case LONG:
- builder.setMinValue((long) defaultNullValue).setMaxValue((long) defaultNullValue);
- break;
- case FLOAT:
- builder.setMinValue((float) defaultNullValue).setMaxValue((float) defaultNullValue);
- break;
- case DOUBLE:
- builder.setMinValue((double) defaultNullValue).setMaxValue((double) defaultNullValue);
- break;
- case BIG_DECIMAL:
- builder.setMinValue((BigDecimal) defaultNullValue).setMaxValue((BigDecimal) defaultNullValue);
- break;
- case STRING:
- builder.setMinValue((String) defaultNullValue).setMaxValue((String) defaultNullValue);
- break;
- case BYTES:
- builder.setMinValue(new ByteArray((byte[]) defaultNullValue))
- .setMaxValue(new ByteArray((byte[]) defaultNullValue));
- break;
- default:
- throw new IllegalStateException();
- }
-
- return builder.build();
+ protected Object getValue(VirtualColumnContext context) {
+ return context.getFieldSpec().getDefaultNullValue();
}
}
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/openstruct/OpenStructNullDataSource.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/openstruct/OpenStructNullDataSource.java
index be4ad585650..1b36dceff4d 100644
--- a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/openstruct/OpenStructNullDataSource.java
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/openstruct/OpenStructNullDataSource.java
@@ -23,19 +23,17 @@ import java.util.Map;
import java.util.Set;
import javax.annotation.Nullable;
import org.apache.pinot.segment.local.segment.index.datasource.BaseDataSource;
+import org.apache.pinot.segment.local.segment.index.readers.AllNullValueVectorReader;
import org.apache.pinot.segment.spi.datasource.DataSourceMetadata;
import org.apache.pinot.segment.spi.datasource.OpenStructDataSource;
import org.apache.pinot.segment.spi.index.StandardIndexes;
import org.apache.pinot.segment.spi.index.column.ColumnIndexContainer;
import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
import org.apache.pinot.segment.spi.index.reader.ForwardIndexReaderContext;
-import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
import org.apache.pinot.segment.spi.partition.PartitionFunction;
import org.apache.pinot.spi.data.DimensionFieldSpec;
import org.apache.pinot.spi.data.FieldSpec;
import org.apache.pinot.spi.data.FieldSpec.DataType;
-import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
-import org.roaringbitmap.buffer.MutableRoaringBitmap;
/// Typed all-null {@link org.apache.pinot.segment.spi.datasource.DataSource} for an OPEN_STRUCT key
@@ -54,7 +52,7 @@ public class OpenStructNullDataSource extends BaseDataSource {
public OpenStructNullDataSource(FieldSpec fieldSpec, int numDocs) {
super(new AllNullMetadata(fieldSpec, numDocs), new ColumnIndexContainer.FromMap(Map.of(
StandardIndexes.forward(), new TypedNullForwardIndex(fieldSpec.getDataType().getStoredType()),
- StandardIndexes.nullValueVector(), new AllNullValueVector(numDocs))));
+ StandardIndexes.nullValueVector(), new AllNullValueVectorReader(numDocs))));
}
/// Creates an all-null DataSource for a key absent from this OPEN_STRUCT segment.
@@ -131,29 +129,6 @@ public class OpenStructNullDataSource extends BaseDataSource {
}
}
- /// {@link NullValueVectorReader} where every document is null.
- static class AllNullValueVector implements NullValueVectorReader {
- private final ImmutableRoaringBitmap _nullBitmap;
-
- AllNullValueVector(int numDocs) {
- MutableRoaringBitmap bm = new MutableRoaringBitmap();
- if (numDocs > 0) {
- bm.add(0L, numDocs);
- }
- _nullBitmap = bm.toImmutableRoaringBitmap();
- }
-
- @Override
- public boolean isNull(int docId) {
- return true;
- }
-
- @Override
- public ImmutableRoaringBitmap getNullBitmap() {
- return _nullBitmap;
- }
- }
-
private static class AllNullMetadata implements DataSourceMetadata {
private final FieldSpec _fieldSpec;
private final int _numDocs;
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/AllNullValueVectorReader.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/AllNullValueVectorReader.java
new file mode 100644
index 00000000000..5c8d6a50e06
--- /dev/null
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/AllNullValueVectorReader.java
@@ -0,0 +1,60 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.readers;
+
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.roaringbitmap.buffer.MutableRoaringBitmap;
+
+
+/// [NullValueVectorReader] where every document of the column is null.
+///
+/// Used by columns that carry no value at all: a materialized-but-absent OPEN_STRUCT child, and the segment metadata
+/// virtual columns whose metadata is not available yet (e.g. the time range of a CONSUMING segment).
+///
+/// The bitmap is materialized lazily, because null value vectors are only consulted when null handling is enabled
+/// while the reader itself is created on every data source build for a mutable segment.
+public class AllNullValueVectorReader implements NullValueVectorReader {
+ private final int _numDocs;
+ private volatile ImmutableRoaringBitmap _nullBitmap;
+
+ public AllNullValueVectorReader(int numDocs) {
+ _numDocs = numDocs;
+ }
+
+ @Override
+ public boolean isNull(int docId) {
+ return true;
+ }
+
+ @Override
+ public ImmutableRoaringBitmap getNullBitmap() {
+ ImmutableRoaringBitmap nullBitmap = _nullBitmap;
+ if (nullBitmap == null) {
+ // Benign race: concurrent callers may each build an equivalent bitmap, and publication is safe because the
+ // volatile write happens-after the bitmap is fully built. NOTE: toImmutableRoaringBitmap() returns `this`, so
+ // the returned instance is shared and must be treated as read-only by callers.
+ MutableRoaringBitmap bitmap = _numDocs > 0 ? MutableRoaringBitmap.bitmapOfRange(0, _numDocs)
+ : new MutableRoaringBitmap();
+ nullBitmap = bitmap.toImmutableRoaringBitmap();
+ _nullBitmap = nullBitmap;
+ }
+ return nullBitmap;
+ }
+}
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/BaseConstantValueVirtualColumnProvider.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/BaseConstantValueVirtualColumnProvider.java
new file mode 100644
index 00000000000..d4d7c9fcc10
--- /dev/null
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/BaseConstantValueVirtualColumnProvider.java
@@ -0,0 +1,220 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.virtualcolumn;
+
+import com.google.common.base.Preconditions;
+import java.math.BigDecimal;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.local.segment.index.readers.ConstantValueBigDecimalDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.ConstantValueBytesDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.ConstantValueDoubleDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.ConstantValueFloatDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.ConstantValueIntDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.ConstantValueLongDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.ConstantValueStringDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.constant.ConstantMVForwardIndexReader;
+import org.apache.pinot.segment.local.segment.index.readers.constant.ConstantMVInvertedIndexReader;
+import org.apache.pinot.segment.local.segment.index.readers.constant.ConstantSortedIndexReader;
+import org.apache.pinot.segment.spi.index.column.ColumnIndexContainer;
+import org.apache.pinot.segment.spi.index.metadata.ColumnMetadataImpl;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
+import org.apache.pinot.segment.spi.index.reader.InvertedIndexReader;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.ByteArray;
+
+
+/// Base for virtual columns holding a single constant value for every document of a segment.
+///
+/// Subclasses supply the value by overriding [#getValue(VirtualColumnContext)]; everything else - a constant sorted
+/// forward index, a single-entry dictionary and matching [ColumnMetadataImpl] - is derived from it here.
+public abstract class BaseConstantValueVirtualColumnProvider implements VirtualColumnProvider {
+
+ /// Returns the constant value to be stored for every document of the column. The returned object must match the
+ /// stored type of the field spec (e.g. a `Long` for a LONG or TIMESTAMP column).
+ protected abstract Object getValue(VirtualColumnContext context);
+
+ /// Reads the value via [#getValue(VirtualColumnContext)] and checks it against the field's stored type, so that a
+ /// provider returning the wrong box type fails with the offending column and provider named rather than with a bare
+ /// `ClassCastException` from deep inside segment loading.
+ private Object getCheckedValue(VirtualColumnContext context) {
+ return checkValue(context, getValue(context));
+ }
+
+ /// Checks an already-resolved value against the field's stored type. Every path that turns a value into an index
+ /// goes through here, including the one taken by subclasses that resolve the value themselves.
+ private Object checkValue(VirtualColumnContext context, Object value) {
+ FieldSpec fieldSpec = context.getFieldSpec();
+ Class<?> expectedClass = getValueClass(fieldSpec);
+ Preconditions.checkState(expectedClass.isInstance(value),
+ "Virtual column provider: %s returned value of type: %s for column: %s, expecting: %s", getClass().getName(),
+ value != null ? value.getClass().getName() : "null", fieldSpec.getName(), expectedClass.getName());
+ return value;
+ }
+
+ private static Class<?> getValueClass(FieldSpec fieldSpec) {
+ switch (fieldSpec.getDataType().getStoredType()) {
+ case INT:
+ return Integer.class;
+ case LONG:
+ return Long.class;
+ case FLOAT:
+ return Float.class;
+ case DOUBLE:
+ return Double.class;
+ case BIG_DECIMAL:
+ return BigDecimal.class;
+ case STRING:
+ return String.class;
+ case BYTES:
+ case UUID:
+ return byte[].class;
+ default:
+ throw new IllegalStateException(unsupportedStoredType(fieldSpec));
+ }
+ }
+
+ private static String unsupportedStoredType(FieldSpec fieldSpec) {
+ return "Unsupported stored type: " + fieldSpec.getDataType().getStoredType() + " for virtual column: "
+ + fieldSpec.getName();
+ }
+
+ @Override
+ public ForwardIndexReader<?> buildForwardIndex(VirtualColumnContext context) {
+ if (context.getFieldSpec().isSingleValueField()) {
+ return new ConstantSortedIndexReader(context.getTotalDocCount());
+ } else {
+ return new ConstantMVForwardIndexReader();
+ }
+ }
+
+ @Override
+ public Dictionary buildDictionary(VirtualColumnContext context) {
+ return buildDictionary(context, getCheckedValue(context));
+ }
+
+ /// Builds the dictionary from an already-resolved value, so that callers which need the value more than once can
+ /// resolve it a single time.
+ private Dictionary buildDictionary(VirtualColumnContext context, Object value) {
+ FieldSpec fieldSpec = context.getFieldSpec();
+ checkValue(context, value);
+ switch (fieldSpec.getDataType().getStoredType()) {
+ case INT:
+ return new ConstantValueIntDictionary((int) value);
+ case LONG:
+ return new ConstantValueLongDictionary((long) value);
+ case FLOAT:
+ return new ConstantValueFloatDictionary((float) value);
+ case DOUBLE:
+ return new ConstantValueDoubleDictionary((double) value);
+ case BIG_DECIMAL:
+ return new ConstantValueBigDecimalDictionary((BigDecimal) value);
+ case STRING:
+ return new ConstantValueStringDictionary((String) value);
+ case BYTES:
+ case UUID:
+ return new ConstantValueBytesDictionary((byte[]) value);
+ default:
+ throw new IllegalStateException(unsupportedStoredType(fieldSpec));
+ }
+ }
+
+ @Override
+ public InvertedIndexReader<?> buildInvertedIndex(VirtualColumnContext context) {
+ if (context.getFieldSpec().isSingleValueField()) {
+ return new ConstantSortedIndexReader(context.getTotalDocCount());
+ } else {
+ return new ConstantMVInvertedIndexReader(context.getTotalDocCount());
+ }
+ }
+
+ @Override
+ public ColumnMetadataImpl buildMetadata(VirtualColumnContext context) {
+ return buildMetadata(context, getCheckedValue(context));
+ }
+
+ /// Builds the column metadata from an already-resolved value.
+ protected final ColumnMetadataImpl buildMetadata(VirtualColumnContext context, Object value) {
+ return buildMetadata(context, value, true);
+ }
+
+ /// Builds the column metadata from an already-resolved value.
+ ///
+ /// @param hasValue `false` when `value` is only a placeholder standing in for a value that is not available, in
+ /// which case the min/max are left unset. Segment pruners read min/max without consulting the null
+ /// value vector, so publishing the placeholder there would let an all-null column prune or reorder
+ /// segments as if it held a real extreme value.
+ protected final ColumnMetadataImpl buildMetadata(VirtualColumnContext context, Object value, boolean hasValue) {
+ FieldSpec fieldSpec = context.getFieldSpec();
+ checkValue(context, value);
+ ColumnMetadataImpl.Builder builder = new ColumnMetadataImpl.Builder().setFieldSpec(fieldSpec)
+ .setTotalDocs(context.getTotalDocCount())
+ .setCardinality(1)
+ .setHasDictionary(true);
+ if (fieldSpec.isSingleValueField()) {
+ builder.setSorted(true);
+ } else {
+ // When there is no value for a multi-value column, the maxNumberOfMultiValues and cardinality should be
+ // set as 1 because the MV column bitmap uses 1 to delimit the rows for a MV column. Each MV column will have a
+ // default null value based on column's data type
+ builder.setMaxNumberOfMultiValues(1);
+ }
+
+ if (!hasValue) {
+ return builder.build();
+ }
+
+ switch (fieldSpec.getDataType().getStoredType()) {
+ case INT:
+ builder.setMinValue((int) value).setMaxValue((int) value);
+ break;
+ case LONG:
+ builder.setMinValue((long) value).setMaxValue((long) value);
+ break;
+ case FLOAT:
+ builder.setMinValue((float) value).setMaxValue((float) value);
+ break;
+ case DOUBLE:
+ builder.setMinValue((double) value).setMaxValue((double) value);
+ break;
+ case BIG_DECIMAL:
+ builder.setMinValue((BigDecimal) value).setMaxValue((BigDecimal) value);
+ break;
+ case STRING:
+ builder.setMinValue((String) value).setMaxValue((String) value);
+ break;
+ case BYTES:
+ builder.setMinValue(new ByteArray((byte[]) value)).setMaxValue(new ByteArray((byte[]) value));
+ break;
+ default:
+ throw new IllegalStateException(unsupportedStoredType(fieldSpec));
+ }
+
+ return builder.build();
+ }
+
+ /// Builds the column index container from an already-resolved value, so that a subclass whose value comes from
+ /// mutable state can resolve it a single time and keep every component consistent.
+ protected final ColumnIndexContainer buildColumnIndexContainer(VirtualColumnContext context, Object value,
+ @Nullable NullValueVectorReader nullValueVector) {
+ return new VirtualColumnIndexContainer(buildForwardIndex(context), buildInvertedIndex(context),
+ buildDictionary(context, value), nullValueVector);
+ }
+}
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/BaseSegmentMetadataVirtualColumnProvider.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/BaseSegmentMetadataVirtualColumnProvider.java
new file mode 100644
index 00000000000..9e28ba5c836
--- /dev/null
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/BaseSegmentMetadataVirtualColumnProvider.java
@@ -0,0 +1,97 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.virtualcolumn;
+
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.local.segment.index.datasource.ImmutableDataSource;
+import org.apache.pinot.segment.local.segment.index.readers.AllNullValueVectorReader;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.segment.spi.index.column.ColumnIndexContainer;
+import org.apache.pinot.segment.spi.index.metadata.ColumnMetadataImpl;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+
+
+/// Base class for the built-in virtual columns that expose a piece of the segment metadata (creation time, time range,
+/// CRC, etc.) as a constant single-value column.
+///
+/// The value is read from [SegmentMetadata] when the column is built rather than baked into the field spec, so that a
+/// mutable segment - which rebuilds its virtual data sources on every access - picks up metadata that was not yet
+/// available when the segment was created. Within a single build the value is resolved exactly once (see
+/// [BaseConstantValueVirtualColumnProvider#buildColumnIndexContainer]), so the dictionary and the null value vector
+/// can never disagree about whether the column has a value.
+///
+/// When the metadata is not available at all, or the specific piece of metadata is not set (e.g. the time range of a
+/// CONSUMING segment), the column stores the default null value of its data type *and* reports every document as null,
+/// so that the placeholder is not mistaken for a real value once null handling is enabled.
+public abstract class BaseSegmentMetadataVirtualColumnProvider extends BaseConstantValueVirtualColumnProvider {
+
+ @Override
+ protected Object getValue(VirtualColumnContext context) {
+ return valueOrPlaceholder(context, extractValueOrNull(context));
+ }
+
+ @Nullable
+ @Override
+ public NullValueVectorReader buildNullValueVector(VirtualColumnContext context) {
+ return extractValueOrNull(context) == null ? new AllNullValueVectorReader(context.getTotalDocCount()) : null;
+ }
+
+ @Override
+ public ColumnIndexContainer buildColumnIndexContainer(VirtualColumnContext context) {
+ return buildColumnIndexContainer(context, extractValueOrNull(context));
+ }
+
+ @Override
+ public ColumnMetadataImpl buildMetadata(VirtualColumnContext context) {
+ Object extracted = extractValueOrNull(context);
+ return buildMetadata(context, valueOrPlaceholder(context, extracted), extracted != null);
+ }
+
+ @Override
+ public DataSource buildDataSource(VirtualColumnContext context) {
+ // Resolve the metadata exactly once for the whole data source, so the dictionary, the column metadata's min/max
+ // and the null value vector all agree about whether this column has a value. buildDataSource is the path a
+ // mutable segment takes, and it rebuilds its virtual data sources on every access.
+ Object extracted = extractValueOrNull(context);
+ return new ImmutableDataSource(buildMetadata(context, valueOrPlaceholder(context, extracted), extracted != null),
+ buildColumnIndexContainer(context, extracted));
+ }
+
+ private ColumnIndexContainer buildColumnIndexContainer(VirtualColumnContext context, @Nullable Object extracted) {
+ return buildColumnIndexContainer(context, valueOrPlaceholder(context, extracted),
+ extracted == null ? new AllNullValueVectorReader(context.getTotalDocCount()) : null);
+ }
+
+ /// Falls back to the column's default null value, which is only ever a placeholder: whenever it is used, the column
+ /// also reports every document as null.
+ private static Object valueOrPlaceholder(VirtualColumnContext context, @Nullable Object extracted) {
+ return extracted != null ? extracted : context.getFieldSpec().getDefaultNullValue();
+ }
+
+ @Nullable
+ private Object extractValueOrNull(VirtualColumnContext context) {
+ SegmentMetadata segmentMetadata = context.getSegmentMetadata();
+ return segmentMetadata != null ? extractValue(segmentMetadata) : null;
+ }
+
+ /// Extracts the value for this column from the given segment metadata, or `null` when it is not available.
+ @Nullable
+ protected abstract Object extractValue(SegmentMetadata segmentMetadata);
+}
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentCrcVirtualColumnProvider.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentCrcVirtualColumnProvider.java
new file mode 100644
index 00000000000..d64d0683a61
--- /dev/null
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentCrcVirtualColumnProvider.java
@@ -0,0 +1,49 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.virtualcolumn;
+
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+
+
+/// Virtual column provider for `$crc`, the CRC of the segment.
+///
+/// The CRC is exposed as a LONG, matching how it is stored everywhere else in Pinot ([SegmentMetadata#getCrc()]
+/// renders the same `long` as a String). It reads as NULL for CONSUMING segments, which have no CRC until they are
+/// committed. Grouping by `$segmentName` and `$crc` is a convenient way to detect replicas of a segment that have
+/// diverged.
+public class SegmentCrcVirtualColumnProvider extends BaseSegmentMetadataVirtualColumnProvider {
+ @Nullable
+ @Override
+ protected Object extractValue(SegmentMetadata segmentMetadata) {
+ String crc = segmentMetadata.getCrc();
+ if (crc == null) {
+ return null;
+ }
+ long crcValue;
+ try {
+ crcValue = Long.parseLong(crc);
+ } catch (NumberFormatException e) {
+ // The CRC is always rendered from a long, but never fail a segment load over an unreadable one
+ return null;
+ }
+ // SegmentMetadataImpl renders an unset CRC as Long.MIN_VALUE
+ return crcValue != Long.MIN_VALUE ? crcValue : null;
+ }
+}
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentCreationTimeVirtualColumnProvider.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentCreationTimeVirtualColumnProvider.java
new file mode 100644
index 00000000000..3d564876ab3
--- /dev/null
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentCreationTimeVirtualColumnProvider.java
@@ -0,0 +1,38 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.virtualcolumn;
+
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+
+
+/// Virtual column provider for `$creationTime`, the time the segment was created.
+///
+/// For a CONSUMING segment this is the time the consuming segment was created, not the time it was committed.
+public class SegmentCreationTimeVirtualColumnProvider extends BaseSegmentMetadataVirtualColumnProvider {
+
+ @Nullable
+ @Override
+ protected Object extractValue(SegmentMetadata segmentMetadata) {
+ // An unset creation time is represented as Long.MIN_VALUE in the segment metadata and as -1 in the segment ZK
+ // metadata a CONSUMING segment is created from, so treat any non-positive value as unavailable.
+ long creationTime = segmentMetadata.getIndexCreationTime();
+ return creationTime > 0 ? creationTime : null;
+ }
+}
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentEndTimeVirtualColumnProvider.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentEndTimeVirtualColumnProvider.java
new file mode 100644
index 00000000000..ee9d79fe671
--- /dev/null
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentEndTimeVirtualColumnProvider.java
@@ -0,0 +1,38 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.virtualcolumn;
+
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.joda.time.Interval;
+
+
+/// Virtual column provider for `$endTime`, the end of the segment time range.
+///
+/// The value is normalized from the time unit of the segment's time column, and exposed as a TIMESTAMP. It reads as
+/// NULL for segments without a time range, such as CONSUMING segments and segments of tables without a time column.
+public class SegmentEndTimeVirtualColumnProvider extends BaseSegmentMetadataVirtualColumnProvider {
+
+ @Nullable
+ @Override
+ protected Object extractValue(SegmentMetadata segmentMetadata) {
+ Interval timeInterval = segmentMetadata.getTimeInterval();
+ return timeInterval != null ? timeInterval.getEndMillis() : null;
+ }
+}
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentStartTimeVirtualColumnProvider.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentStartTimeVirtualColumnProvider.java
new file mode 100644
index 00000000000..50600c974a7
--- /dev/null
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentStartTimeVirtualColumnProvider.java
@@ -0,0 +1,38 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.virtualcolumn;
+
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.joda.time.Interval;
+
+
+/// Virtual column provider for `$startTime`, the start of the segment time range.
+///
+/// The value is normalized from the time unit of the segment's time column, and exposed as a TIMESTAMP. It reads as
+/// NULL for segments without a time range, such as CONSUMING segments and segments of tables without a time column.
+public class SegmentStartTimeVirtualColumnProvider extends BaseSegmentMetadataVirtualColumnProvider {
+
+ @Nullable
+ @Override
+ protected Object extractValue(SegmentMetadata segmentMetadata) {
+ Interval timeInterval = segmentMetadata.getTimeInterval();
+ return timeInterval != null ? timeInterval.getStartMillis() : null;
+ }
+}
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentTotalDocsVirtualColumnProvider.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentTotalDocsVirtualColumnProvider.java
new file mode 100644
index 00000000000..2b44503a2ba
--- /dev/null
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentTotalDocsVirtualColumnProvider.java
@@ -0,0 +1,35 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.virtualcolumn;
+
+
+/// Virtual column provider for `$totalDocs`, the number of documents in the segment.
+///
+/// The count is taken from the virtual column context rather than from `SegmentMetadata`,
+/// so that a CONSUMING segment reports the number of documents indexed so far instead of `0`.
+///
+/// This is the number of documents physically stored in the segment, so for an upsert table it also includes the
+/// documents that have been replaced and are no longer returned by queries.
+public class SegmentTotalDocsVirtualColumnProvider extends BaseConstantValueVirtualColumnProvider {
+
+ @Override
+ protected Object getValue(VirtualColumnContext context) {
+ return context.getTotalDocCount();
+ }
+}
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnIndexContainer.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnIndexContainer.java
index d64f7291269..fc6d40beff0 100644
--- a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnIndexContainer.java
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnIndexContainer.java
@@ -27,6 +27,7 @@ import org.apache.pinot.segment.spi.index.column.ColumnIndexContainer;
import org.apache.pinot.segment.spi.index.reader.Dictionary;
import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
import org.apache.pinot.segment.spi.index.reader.InvertedIndexReader;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
/// Column index container for virtual columns.
@@ -34,12 +35,23 @@ public class VirtualColumnIndexContainer implements ColumnIndexContainer {
private final ForwardIndexReader<?> _forwardIndex;
private final InvertedIndexReader<?> _invertedIndex;
private final Dictionary _dictionary;
+ private final NullValueVectorReader _nullValueVector;
public VirtualColumnIndexContainer(ForwardIndexReader<?> forwardIndex, InvertedIndexReader<?> invertedIndex,
Dictionary dictionary) {
+ this(forwardIndex, invertedIndex, dictionary, null);
+ }
+
+ /// @param nullValueVector marks which documents of the virtual column are null, or `null` when the column has no
+ /// null value. A virtual column whose value is genuinely unavailable (e.g. the time range of
+ /// a CONSUMING segment) must supply one, otherwise the engine treats the placeholder value
+ /// stored in the forward index as a real value.
+ public VirtualColumnIndexContainer(ForwardIndexReader<?> forwardIndex, InvertedIndexReader<?> invertedIndex,
+ Dictionary dictionary, @Nullable NullValueVectorReader nullValueVector) {
_forwardIndex = forwardIndex;
_invertedIndex = invertedIndex;
_dictionary = dictionary;
+ _nullValueVector = nullValueVector;
}
@Nullable
@@ -54,6 +66,9 @@ public class VirtualColumnIndexContainer implements ColumnIndexContainer {
if (indexType.equals(StandardIndexes.dictionary())) {
return (I) _dictionary;
}
+ if (indexType.equals(StandardIndexes.nullValueVector())) {
+ return (I) _nullValueVector;
+ }
return null;
}
@@ -67,5 +82,8 @@ public class VirtualColumnIndexContainer implements ColumnIndexContainer {
if (_dictionary != null) {
_dictionary.close();
}
+ if (_nullValueVector != null) {
+ _nullValueVector.close();
+ }
}
}
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnProvider.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnProvider.java
index 194773d0332..0079cdaa2b0 100644
--- a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnProvider.java
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnProvider.java
@@ -18,6 +18,7 @@
*/
package org.apache.pinot.segment.local.segment.virtualcolumn;
+import javax.annotation.Nullable;
import org.apache.pinot.segment.local.segment.index.datasource.ImmutableDataSource;
import org.apache.pinot.segment.spi.ColumnMetadata;
import org.apache.pinot.segment.spi.datasource.DataSource;
@@ -25,6 +26,7 @@ import org.apache.pinot.segment.spi.index.column.ColumnIndexContainer;
import org.apache.pinot.segment.spi.index.reader.Dictionary;
import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
import org.apache.pinot.segment.spi.index.reader.InvertedIndexReader;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
/// Virtual column provider interface, which is used to instantiate the various components (dictionary, reader, etc)
@@ -39,9 +41,19 @@ public interface VirtualColumnProvider {
ColumnMetadata buildMetadata(VirtualColumnContext context);
+ /// Returns the null value vector of the virtual column, or `null` when the column has no null value.
+ ///
+ /// Virtual columns almost always carry a real value for every document, hence the default. A provider whose value
+ /// can be genuinely unavailable must override this, otherwise the placeholder stored in the forward index is
+ /// indistinguishable from a real value once null handling is enabled.
+ @Nullable
+ default NullValueVectorReader buildNullValueVector(VirtualColumnContext context) {
+ return null;
+ }
+
default ColumnIndexContainer buildColumnIndexContainer(VirtualColumnContext context) {
return new VirtualColumnIndexContainer(buildForwardIndex(context), buildInvertedIndex(context),
- buildDictionary(context));
+ buildDictionary(context), buildNullValueVector(context));
}
default DataSource buildDataSource(VirtualColumnContext context) {
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnProviderFactory.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnProviderFactory.java
index b60d9a3cf00..4d9f04908c3 100644
--- a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnProviderFactory.java
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnProviderFactory.java
@@ -18,9 +18,12 @@
*/
package org.apache.pinot.segment.local.segment.virtualcolumn;
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.stream.Collectors;
import org.apache.pinot.segment.local.segment.index.column.DefaultNullValueVirtualColumnProvider;
+import org.apache.pinot.spi.data.BuiltInVirtualColumnDefinitions;
import org.apache.pinot.spi.data.DimensionFieldSpec;
-import org.apache.pinot.spi.data.FieldSpec;
import org.apache.pinot.spi.data.Schema;
import org.apache.pinot.spi.plugin.PluginManager;
import org.apache.pinot.spi.utils.CommonConstants.Segment.BuiltInVirtualColumn;
@@ -29,11 +32,46 @@ import org.apache.pinot.spi.utils.NetUtils;
/// Factory for virtual column providers.
public class VirtualColumnProviderFactory {
+ /// Provider for each built-in virtual column. Its key set is asserted to cover
+ /// [BuiltInVirtualColumnDefinitions#DEFINITIONS] by `SegmentMetadataVirtualColumnProviderTest`, so a column added
+ /// there without a provider here is caught at build time; at runtime `getProviderClass` names the offending column.
+ private static final Map<String, Class<? extends VirtualColumnProvider>> PROVIDER_CLASSES =
+ Map.of(BuiltInVirtualColumn.DOCID, DocIdVirtualColumnProvider.class,
+ BuiltInVirtualColumn.HOSTNAME, DefaultNullValueVirtualColumnProvider.class,
+ BuiltInVirtualColumn.SEGMENTNAME, DefaultNullValueVirtualColumnProvider.class,
+ BuiltInVirtualColumn.PARTITIONID, PartitionIdVirtualColumnProvider.class,
+ BuiltInVirtualColumn.CREATIONTIME, SegmentCreationTimeVirtualColumnProvider.class,
+ BuiltInVirtualColumn.STARTTIME, SegmentStartTimeVirtualColumnProvider.class,
+ BuiltInVirtualColumn.ENDTIME, SegmentEndTimeVirtualColumnProvider.class,
+ BuiltInVirtualColumn.TOTALDOCS, SegmentTotalDocsVirtualColumnProvider.class,
+ BuiltInVirtualColumn.CRC, SegmentCrcVirtualColumnProvider.class);
+
+ /// Shared instances of the built-in providers, keyed by the class name stored in the field spec.
+ private static final Map<String, VirtualColumnProvider> BUILT_IN_PROVIDERS = PROVIDER_CLASSES.values()
+ .stream()
+ .distinct()
+ .collect(Collectors.toUnmodifiableMap(Class::getName, VirtualColumnProviderFactory::newInstance));
+
private VirtualColumnProviderFactory() {
}
+ private static VirtualColumnProvider newInstance(Class<? extends VirtualColumnProvider> providerClass) {
+ try {
+ return providerClass.getDeclaredConstructor().newInstance();
+ } catch (Exception e) {
+ throw new IllegalStateException("Caught exception while creating instance of: " + providerClass.getName(), e);
+ }
+ }
+
public static VirtualColumnProvider buildProvider(VirtualColumnContext virtualColumnContext) {
String virtualColumnProvider = virtualColumnContext.getFieldSpec().getVirtualColumnProvider();
+ // The built-in providers are stateless - everything they need comes from the VirtualColumnContext - so a single
+ // shared instance serves every column of every segment. This matters at segment load, which resolves a provider
+ // per virtual column per segment, and on the mutable path, which resolves one per query.
+ VirtualColumnProvider builtInProvider = BUILT_IN_PROVIDERS.get(virtualColumnProvider);
+ if (builtInProvider != null) {
+ return builtInProvider;
+ }
try {
return PluginManager.get().createInstance(virtualColumnProvider);
} catch (Exception e) {
@@ -41,25 +79,37 @@ public class VirtualColumnProviderFactory {
}
}
+ /// Adds the built-in virtual columns to the schema of a segment, together with the provider that produces their
+ /// values.
+ ///
+ /// The shape of each column (name, data type, single-value vs multi-value) comes from
+ /// [BuiltInVirtualColumnDefinitions#DEFINITIONS], which the broker side uses as well, so the two can never
+ /// disagree on a type.
+ /// This method only layers on the provider class, and the constant value for the columns whose value is already
+ /// known here.
public static void addBuiltInVirtualColumnsToSegmentSchema(Schema schema, String segmentName) {
- if (!schema.hasColumn(BuiltInVirtualColumn.DOCID)) {
- schema.addField(new DimensionFieldSpec(BuiltInVirtualColumn.DOCID, FieldSpec.DataType.INT, true,
- DocIdVirtualColumnProvider.class));
- }
-
- if (!schema.hasColumn(BuiltInVirtualColumn.HOSTNAME)) {
- schema.addField(new DimensionFieldSpec(BuiltInVirtualColumn.HOSTNAME, FieldSpec.DataType.STRING, true,
- DefaultNullValueVirtualColumnProvider.class, NetUtils.getHostnameOrAddress()));
- }
-
- if (!schema.hasColumn(BuiltInVirtualColumn.SEGMENTNAME)) {
- schema.addField(new DimensionFieldSpec(BuiltInVirtualColumn.SEGMENTNAME, FieldSpec.DataType.STRING, true,
- DefaultNullValueVirtualColumnProvider.class, segmentName));
+ for (BuiltInVirtualColumnDefinitions.Definition definition : BuiltInVirtualColumnDefinitions.DEFINITIONS) {
+ String column = definition.getName();
+ if (schema.hasColumn(column)) {
+ continue;
+ }
+ DimensionFieldSpec fieldSpec = definition.createFieldSpec();
+ fieldSpec.setVirtualColumnProvider(getProviderClass(column).getName());
+ // $hostName and $segmentName are constants known at schema construction time, and are carried as the field's
+ // default null value, which DefaultNullValueVirtualColumnProvider reads back.
+ if (BuiltInVirtualColumn.HOSTNAME.equals(column)) {
+ fieldSpec.setDefaultNullValue(NetUtils.getHostnameOrAddress());
+ } else if (BuiltInVirtualColumn.SEGMENTNAME.equals(column)) {
+ fieldSpec.setDefaultNullValue(segmentName);
+ }
+ schema.addField(fieldSpec);
}
+ }
- if (!schema.hasColumn(BuiltInVirtualColumn.PARTITIONID)) {
- schema.addField(new DimensionFieldSpec(BuiltInVirtualColumn.PARTITIONID, FieldSpec.DataType.STRING, false,
- PartitionIdVirtualColumnProvider.class));
- }
+ private static Class<? extends VirtualColumnProvider> getProviderClass(String column) {
+ Class<? extends VirtualColumnProvider> providerClass = PROVIDER_CLASSES.get(column);
+ Preconditions.checkState(providerClass != null, "No virtual column provider registered for built-in column: %s",
+ column);
+ return providerClass;
}
}
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/SchemaUtils.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/SchemaUtils.java
index 9c257b06eee..1481a5b26d8 100644
--- a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/SchemaUtils.java
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/SchemaUtils.java
@@ -31,6 +31,7 @@ import javax.annotation.Nullable;
import org.apache.commons.lang3.StringUtils;
import org.apache.pinot.common.evaluator.FunctionEvaluatorFactory;
import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.BuiltInVirtualColumnDefinitions;
import org.apache.pinot.spi.data.DateTimeFieldSpec;
import org.apache.pinot.spi.data.DateTimeFormatSpec;
import org.apache.pinot.spi.data.DateTimeGranularitySpec;
@@ -162,6 +163,12 @@ public class SchemaUtils {
String column = fieldSpec.getName();
Preconditions.checkState(!StringUtils.containsWhitespace(column),
"The column name \"%s\" should not contain blank space.", column);
+ // A user column of the same name would be shadowed at query time: the built-in virtual column is filtered out
+ // of the segment's physical columns when the segment metadata is read, and the virtual provider then takes over
+ // the name, so queries would silently return segment metadata instead of the user's data.
+ Preconditions.checkState(
+ fieldSpec.isVirtualColumn() || !BuiltInVirtualColumnDefinitions.NAMES.contains(column),
+ "The column name \"%s\" is reserved for a built-in virtual column.", column);
if (!fieldSpec.isVirtualColumn()) {
primaryKeyColumnCandidates.add(column);
}
diff --git a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImplRawMVTest.java b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImplRawMVTest.java
index 448bfd242b9..82153f600e8 100644
--- a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImplRawMVTest.java
+++ b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImplRawMVTest.java
@@ -67,6 +67,14 @@ import static org.testng.Assert.assertNull;
public class MutableSegmentImplRawMVTest implements PinotBuffersAfterClassCheckRule {
private static final String AVRO_FILE = "data/test_data-mv.avro";
private static final File TEMP_DIR = new File(FileUtils.getTempDirectory(), "MutableSegmentImplRawMVTest");
+ /// Virtual columns describing the segment itself, which are expected to differ between a mutable segment and an
+ /// immutable segment built from the same records.
+ private static final Set<String> SEGMENT_LEVEL_VIRTUAL_COLUMNS =
+ Set.of(CommonConstants.Segment.BuiltInVirtualColumn.SEGMENTNAME,
+ CommonConstants.Segment.BuiltInVirtualColumn.CREATIONTIME,
+ CommonConstants.Segment.BuiltInVirtualColumn.STARTTIME,
+ CommonConstants.Segment.BuiltInVirtualColumn.ENDTIME,
+ CommonConstants.Segment.BuiltInVirtualColumn.CRC);
private Schema _schema;
private MutableSegmentImpl _mutableSegmentImpl;
@@ -166,8 +174,9 @@ public class MutableSegmentImplRawMVTest implements PinotBuffersAfterClassCheckR
Dictionary expectedDictionary = expectedDataSource.getDictionary();
assertEquals(actualDictionary.length(), expectedDictionary.length());
- // Allow the segment name to be different
- if (column.equals(CommonConstants.Segment.BuiltInVirtualColumn.SEGMENTNAME)) {
+ // Allow the segment level metadata to be different between the mutable segment and the immutable segment
+ // built from the same records
+ if (SEGMENT_LEVEL_VIRTUAL_COLUMNS.contains(column)) {
continue;
}
diff --git a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImplTest.java b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImplTest.java
index c837922f05e..537704ed557 100644
--- a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImplTest.java
+++ b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImplTest.java
@@ -38,6 +38,7 @@ import org.apache.pinot.segment.spi.index.creator.VectorIndexConfig;
import org.apache.pinot.segment.spi.index.reader.Dictionary;
import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
import org.apache.pinot.segment.spi.index.reader.ForwardIndexReaderContext;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
import org.apache.pinot.spi.config.instance.InstanceType;
import org.apache.pinot.spi.data.FieldSpec;
import org.apache.pinot.spi.data.Schema;
@@ -62,6 +63,14 @@ import static org.testng.Assert.assertEquals;
public class MutableSegmentImplTest {
private static final String AVRO_FILE = "data/test_data-mv.avro";
private static final File TEMP_DIR = new File(FileUtils.getTempDirectory(), "MutableSegmentImplTest");
+ /// Virtual columns describing the segment itself, which are expected to differ between a mutable segment and an
+ /// immutable segment built from the same records.
+ private static final Set<String> SEGMENT_LEVEL_VIRTUAL_COLUMNS =
+ Set.of(CommonConstants.Segment.BuiltInVirtualColumn.SEGMENTNAME,
+ CommonConstants.Segment.BuiltInVirtualColumn.CREATIONTIME,
+ CommonConstants.Segment.BuiltInVirtualColumn.STARTTIME,
+ CommonConstants.Segment.BuiltInVirtualColumn.ENDTIME,
+ CommonConstants.Segment.BuiltInVirtualColumn.CRC);
private Schema _schema;
private MutableSegmentImpl _mutableSegmentImpl;
@@ -132,6 +141,29 @@ public class MutableSegmentImplTest {
}
}
+ /// The segment metadata virtual columns are skipped in the mutable-vs-immutable comparisons above because they
+ /// legitimately differ, so their values on a real mutable segment are pinned here instead. This mutable segment is
+ /// built from ZK metadata without a creation time and never gets a time range or a CRC, which is exactly the shape
+ /// of a CONSUMING segment.
+ @Test
+ public void testSegmentMetadataVirtualColumnsOnMutableSegment() {
+ for (String column : Set.of(CommonConstants.Segment.BuiltInVirtualColumn.CREATIONTIME,
+ CommonConstants.Segment.BuiltInVirtualColumn.STARTTIME,
+ CommonConstants.Segment.BuiltInVirtualColumn.ENDTIME,
+ CommonConstants.Segment.BuiltInVirtualColumn.CRC)) {
+ DataSource dataSource = _mutableSegmentImpl.getDataSource(column);
+ NullValueVectorReader nullValueVector = dataSource.getNullValueVector();
+ Assert.assertNotNull(nullValueVector, "Expecting a null value vector for virtual column: " + column);
+ assertEquals(nullValueVector.getNullBitmap().getCardinality(), _mutableSegmentImpl.getNumDocsIndexed());
+ }
+
+ // $totalDocs tracks the documents indexed so far, and is never null
+ DataSource totalDocsDataSource =
+ _mutableSegmentImpl.getDataSource(CommonConstants.Segment.BuiltInVirtualColumn.TOTALDOCS);
+ assertEquals(totalDocsDataSource.getDictionary().getIntValue(0), _mutableSegmentImpl.getNumDocsIndexed());
+ Assert.assertNull(totalDocsDataSource.getNullValueVector());
+ }
+
@Test
public void testDataSourceForSVColumns() {
for (FieldSpec fieldSpec : _schema.getAllFieldSpecs()) {
@@ -148,8 +180,9 @@ public class MutableSegmentImplTest {
Dictionary expectedDictionary = expectedDataSource.getDictionary();
assertEquals(actualDictionary.length(), expectedDictionary.length());
- // Allow the segment name to be different
- if (column.equals(CommonConstants.Segment.BuiltInVirtualColumn.SEGMENTNAME)) {
+ // Allow the segment level metadata to be different between the mutable segment and the immutable segment
+ // built from the same records
+ if (SEGMENT_LEVEL_VIRTUAL_COLUMNS.contains(column)) {
continue;
}
diff --git a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/LoaderTest.java b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/LoaderTest.java
index cd50d0de804..ac22a9a3e56 100644
--- a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/LoaderTest.java
+++ b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/LoaderTest.java
@@ -20,7 +20,6 @@ package org.apache.pinot.segment.local.segment.index.loader;
import java.io.File;
import java.net.URL;
-import java.util.Arrays;
import java.util.List;
import java.util.Map;
import javax.annotation.Nullable;
@@ -34,11 +33,14 @@ import org.apache.pinot.segment.local.utils.SegmentOperationsThrottler;
import org.apache.pinot.segment.local.utils.SegmentOperationsThrottlerSet;
import org.apache.pinot.segment.spi.ImmutableSegment;
import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.SegmentMetadata;
import org.apache.pinot.segment.spi.V1Constants;
import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
import org.apache.pinot.segment.spi.creator.SegmentVersion;
+import org.apache.pinot.segment.spi.datasource.DataSource;
import org.apache.pinot.segment.spi.index.StandardIndexes;
import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
import org.apache.pinot.segment.spi.store.SegmentDirectory;
import org.apache.pinot.segment.spi.store.SegmentDirectoryPaths;
import org.apache.pinot.spi.config.table.FieldConfig;
@@ -217,11 +219,35 @@ public class LoaderTest {
}
private void testBuiltInVirtualColumns(IndexSegment indexSegment) {
- assertTrue(indexSegment.getColumnNames().containsAll(
- Arrays.asList(BuiltInVirtualColumn.DOCID, BuiltInVirtualColumn.HOSTNAME, BuiltInVirtualColumn.SEGMENTNAME)));
- assertNotNull(indexSegment.getDataSource(BuiltInVirtualColumn.DOCID));
- assertNotNull(indexSegment.getDataSource(BuiltInVirtualColumn.HOSTNAME));
- assertNotNull(indexSegment.getDataSource(BuiltInVirtualColumn.SEGMENTNAME));
+ // Iterate the full set so that a newly added built-in virtual column is covered automatically
+ assertTrue(indexSegment.getColumnNames().containsAll(BuiltInVirtualColumn.BUILT_IN_VIRTUAL_COLUMNS));
+ for (String column : BuiltInVirtualColumn.BUILT_IN_VIRTUAL_COLUMNS) {
+ assertNotNull(indexSegment.getDataSource(column), "Missing data source for virtual column: " + column);
+ }
+
+ // Segment metadata that this segment carries is exposed as a real value, and is not marked null
+ SegmentMetadata segmentMetadata = indexSegment.getSegmentMetadata();
+ assertEquals(indexSegment.getDataSource(BuiltInVirtualColumn.TOTALDOCS).getDictionary().get(0),
+ segmentMetadata.getTotalDocs());
+ assertEquals(indexSegment.getDataSource(BuiltInVirtualColumn.CRC).getDictionary().get(0),
+ Long.parseLong(segmentMetadata.getCrc()));
+ assertEquals(indexSegment.getDataSource(BuiltInVirtualColumn.CREATIONTIME).getDictionary().get(0),
+ segmentMetadata.getIndexCreationTime());
+ for (String column : List.of(BuiltInVirtualColumn.TOTALDOCS, BuiltInVirtualColumn.CRC,
+ BuiltInVirtualColumn.CREATIONTIME)) {
+ assertNull(indexSegment.getDataSource(column).getNullValueVector(),
+ "Unexpected null value vector for virtual column: " + column);
+ }
+
+ // This table has no time column, so the segment has no time range and the two time columns read as NULL
+ assertNull(segmentMetadata.getTimeInterval());
+ for (String column : List.of(BuiltInVirtualColumn.STARTTIME, BuiltInVirtualColumn.ENDTIME)) {
+ DataSource dataSource = indexSegment.getDataSource(column);
+ assertEquals(dataSource.getDictionary().get(0), FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_TIMESTAMP);
+ NullValueVectorReader nullValueVector = dataSource.getNullValueVector();
+ assertNotNull(nullValueVector, "Missing null value vector for virtual column: " + column);
+ assertEquals(nullValueVector.getNullBitmap().getCardinality(), segmentMetadata.getTotalDocs());
+ }
}
/// Tests loading default string column with empty ("") default null value.
diff --git a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentMetadataVirtualColumnProviderTest.java b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentMetadataVirtualColumnProviderTest.java
new file mode 100644
index 00000000000..a336a2e831f
--- /dev/null
+++ b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/virtualcolumn/SegmentMetadataVirtualColumnProviderTest.java
@@ -0,0 +1,405 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.virtualcolumn;
+
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSourceMetadata;
+import org.apache.pinot.segment.spi.index.StandardIndexes;
+import org.apache.pinot.segment.spi.index.column.ColumnIndexContainer;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+import org.apache.pinot.spi.data.BuiltInVirtualColumnDefinitions;
+import org.apache.pinot.spi.data.DimensionFieldSpec;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.utils.CommonConstants.Segment.BuiltInVirtualColumn;
+import org.joda.time.DateTimeZone;
+import org.joda.time.Interval;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertTrue;
+import static org.testng.Assert.fail;
+
+
+/// Tests for the built-in virtual columns exposing segment metadata (`$creationTime`, `$startTime`, `$endTime`,
+/// `$totalDocs` and `$crc`).
+public class SegmentMetadataVirtualColumnProviderTest {
+ private static final int NUM_DOCS = 17;
+ private static final long CREATION_TIME_MS = 1_700_000_000_000L;
+ private static final long START_TIME_MS = 1_690_000_000_000L;
+ private static final long END_TIME_MS = 1_695_000_000_000L;
+ private static final long CRC = 1234567890L;
+
+ @DataProvider(name = "unsetCreationTimes")
+ public Object[][] unsetCreationTimes() {
+ return new Object[][]{{Long.MIN_VALUE}, {-1L}, {0L}};
+ }
+
+ /// Returns the schema a segment gets after the built-in virtual columns are added to it.
+ private static Schema buildSegmentSchema() {
+ Schema schema = new Schema();
+ VirtualColumnProviderFactory.addBuiltInVirtualColumnsToSegmentSchema(schema, "testSegment");
+ return schema;
+ }
+
+ private static SegmentMetadata mockSegmentMetadata() {
+ SegmentMetadata segmentMetadata = mock(SegmentMetadata.class);
+ when(segmentMetadata.getIndexCreationTime()).thenReturn(CREATION_TIME_MS);
+ when(segmentMetadata.getTimeInterval()).thenReturn(new Interval(START_TIME_MS, END_TIME_MS, DateTimeZone.UTC));
+ when(segmentMetadata.getCrc()).thenReturn(String.valueOf(CRC));
+ when(segmentMetadata.getTotalDocs()).thenReturn(NUM_DOCS);
+ return segmentMetadata;
+ }
+
+ /// Returns the metadata a real CONSUMING segment is created with: no time range and no CRC yet. Uses the real
+ /// `SegmentMetadataImpl` rather than a mock so that the unset-value contract stays pinned to what
+ /// `pinot-segment-spi` actually produces.
+ private static SegmentMetadata consumingSegmentMetadata(long creationTime) {
+ return new SegmentMetadataImpl("testTable", "testTable__0__0__20240101T0000Z", new Schema(), creationTime);
+ }
+
+ private static Dictionary buildDictionary(Schema schema, String column, SegmentMetadata segmentMetadata) {
+ FieldSpec fieldSpec = schema.getFieldSpecFor(column);
+ assertNotNull(fieldSpec, column + " should be added to the segment schema");
+ assertTrue(fieldSpec.isVirtualColumn(), column + " should be a virtual column");
+ VirtualColumnContext context = new VirtualColumnContext(fieldSpec, NUM_DOCS, segmentMetadata);
+ return VirtualColumnProviderFactory.buildProvider(context).buildDictionary(context);
+ }
+
+ @Nullable
+ private static NullValueVectorReader buildNullValueVector(Schema schema, String column,
+ SegmentMetadata segmentMetadata) {
+ FieldSpec fieldSpec = schema.getFieldSpecFor(column);
+ VirtualColumnContext context = new VirtualColumnContext(fieldSpec, NUM_DOCS, segmentMetadata);
+ return VirtualColumnProviderFactory.buildProvider(context).buildNullValueVector(context);
+ }
+
+ /// Asserts the column reads as SQL NULL for every document, and that the data source exposes the null vector (which
+ /// is what the query engine consults once null handling is enabled).
+ private static void assertColumnIsNull(Schema schema, String column, SegmentMetadata segmentMetadata) {
+ NullValueVectorReader nullValueVector = buildNullValueVector(schema, column, segmentMetadata);
+ assertNotNull(nullValueVector, column + " should report a null value vector when its metadata is unavailable");
+ assertEquals(nullValueVector.getNullBitmap().getCardinality(), NUM_DOCS);
+ for (int docId = 0; docId < NUM_DOCS; docId++) {
+ assertTrue(nullValueVector.isNull(docId));
+ }
+ FieldSpec fieldSpec = schema.getFieldSpecFor(column);
+ VirtualColumnContext context = new VirtualColumnContext(fieldSpec, NUM_DOCS, segmentMetadata);
+ assertNotNull(VirtualColumnProviderFactory.buildProvider(context).buildDataSource(context).getNullValueVector(),
+ column + " data source should expose the null value vector");
+ }
+
+ private static ColumnMetadata buildColumnMetadata(Schema schema, String column, SegmentMetadata segmentMetadata) {
+ FieldSpec fieldSpec = schema.getFieldSpecFor(column);
+ VirtualColumnContext context = new VirtualColumnContext(fieldSpec, NUM_DOCS, segmentMetadata);
+ return VirtualColumnProviderFactory.buildProvider(context).buildMetadata(context);
+ }
+
+ /// Replaces the class-load assertion that used to live in VirtualColumnProviderFactory: a definition added without
+ /// a provider must be caught here at build time rather than at segment load.
+ @Test
+ public void testEveryDefinitionHasAProvider() {
+ Schema schema = buildSegmentSchema();
+ for (BuiltInVirtualColumnDefinitions.Definition definition : BuiltInVirtualColumnDefinitions.DEFINITIONS) {
+ FieldSpec fieldSpec = schema.getFieldSpecFor(definition.getName());
+ assertNotNull(fieldSpec, "No field spec added for: " + definition.getName());
+ assertTrue(fieldSpec.isVirtualColumn(), "No provider configured for: " + definition.getName());
+ assertEquals(fieldSpec.getDataType(), definition.getDataType());
+ assertEquals(fieldSpec.isSingleValueField(), definition.isSingleValueField());
+ }
+ }
+
+ @Test
+ public void testAllBuiltInVirtualColumnsAreAddedToSegmentSchema() {
+ Schema schema = buildSegmentSchema();
+ for (String column : BuiltInVirtualColumn.BUILT_IN_VIRTUAL_COLUMNS) {
+ FieldSpec fieldSpec = schema.getFieldSpecFor(column);
+ assertNotNull(fieldSpec, column + " should be added to the segment schema");
+ assertTrue(fieldSpec.isVirtualColumn(), column + " should have a virtual column provider configured");
+ // Every configured provider must be resolvable
+ assertNotNull(VirtualColumnProviderFactory.buildProvider(
+ new VirtualColumnContext(fieldSpec, NUM_DOCS, mockSegmentMetadata())));
+ }
+ }
+
+ @Test
+ public void testSegmentMetadataColumnTypes() {
+ Schema schema = buildSegmentSchema();
+ assertEquals(schema.getFieldSpecFor(BuiltInVirtualColumn.CREATIONTIME).getDataType(),
+ FieldSpec.DataType.TIMESTAMP);
+ assertEquals(schema.getFieldSpecFor(BuiltInVirtualColumn.STARTTIME).getDataType(),
+ FieldSpec.DataType.TIMESTAMP);
+ assertEquals(schema.getFieldSpecFor(BuiltInVirtualColumn.ENDTIME).getDataType(),
+ FieldSpec.DataType.TIMESTAMP);
+ assertEquals(schema.getFieldSpecFor(BuiltInVirtualColumn.TOTALDOCS).getDataType(), FieldSpec.DataType.INT);
+ assertEquals(schema.getFieldSpecFor(BuiltInVirtualColumn.CRC).getDataType(), FieldSpec.DataType.LONG);
+ for (String column : List.of(BuiltInVirtualColumn.CREATIONTIME, BuiltInVirtualColumn.STARTTIME,
+ BuiltInVirtualColumn.ENDTIME, BuiltInVirtualColumn.TOTALDOCS, BuiltInVirtualColumn.CRC)) {
+ assertTrue(schema.getFieldSpecFor(column).isSingleValueField(), column + " should be single-value");
+ }
+ }
+
+ @Test
+ public void testValuesFromSegmentMetadata() {
+ Schema schema = buildSegmentSchema();
+ SegmentMetadata segmentMetadata = mockSegmentMetadata();
+
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.CREATIONTIME, segmentMetadata).getLongValue(0),
+ CREATION_TIME_MS);
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.STARTTIME, segmentMetadata).getLongValue(0),
+ START_TIME_MS);
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.ENDTIME, segmentMetadata).getLongValue(0), END_TIME_MS);
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.TOTALDOCS, segmentMetadata).getIntValue(0), NUM_DOCS);
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.CRC, segmentMetadata).getLongValue(0), CRC);
+ }
+
+ @Test
+ public void testColumnMetadataMatchesValue() {
+ Schema schema = buildSegmentSchema();
+ SegmentMetadata segmentMetadata = mockSegmentMetadata();
+
+ ColumnMetadata creationTimeMetadata =
+ buildColumnMetadata(schema, BuiltInVirtualColumn.CREATIONTIME, segmentMetadata);
+ assertEquals(creationTimeMetadata.getTotalDocs(), NUM_DOCS);
+ assertEquals(creationTimeMetadata.getCardinality(), 1);
+ assertTrue(creationTimeMetadata.isSorted());
+ assertTrue(creationTimeMetadata.hasDictionary());
+ assertEquals(creationTimeMetadata.getMinValue(), CREATION_TIME_MS);
+ assertEquals(creationTimeMetadata.getMaxValue(), CREATION_TIME_MS);
+
+ ColumnMetadata crcMetadata = buildColumnMetadata(schema, BuiltInVirtualColumn.CRC, segmentMetadata);
+ assertEquals(crcMetadata.getMinValue(), CRC);
+ assertEquals(crcMetadata.getMaxValue(), CRC);
+
+ // When the metadata is available the column carries a real value, so there is no null vector
+ for (String column : List.of(BuiltInVirtualColumn.CREATIONTIME, BuiltInVirtualColumn.STARTTIME,
+ BuiltInVirtualColumn.ENDTIME, BuiltInVirtualColumn.TOTALDOCS, BuiltInVirtualColumn.CRC)) {
+ assertNull(buildNullValueVector(schema, column, segmentMetadata), column + " should not be null");
+ }
+ }
+
+ /// A segment without a time range and without a CRC must fall back to the default null value for those columns,
+ /// while `$creationTime` and `$totalDocs` remain meaningful. An unset creation time is `Long.MIN_VALUE` in the
+ /// segment metadata of an immutable segment and `-1` in the ZK metadata a CONSUMING segment is created from.
+ @Test(dataProvider = "unsetCreationTimes")
+ public void testUnavailableMetadataFallsBackToDefaultNullValue(long unsetCreationTime) {
+ Schema schema = buildSegmentSchema();
+ SegmentMetadata segmentMetadata = consumingSegmentMetadata(unsetCreationTime);
+
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.CREATIONTIME, segmentMetadata).getLongValue(0),
+ (long) FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_TIMESTAMP);
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.STARTTIME, segmentMetadata).getLongValue(0),
+ (long) FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_TIMESTAMP);
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.ENDTIME, segmentMetadata).getLongValue(0),
+ (long) FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_TIMESTAMP);
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.CRC, segmentMetadata).getLongValue(0),
+ (long) FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_LONG);
+ // The stored value is only a placeholder: the column must additionally report every document as null
+ assertColumnIsNull(schema, BuiltInVirtualColumn.CREATIONTIME, segmentMetadata);
+ assertColumnIsNull(schema, BuiltInVirtualColumn.STARTTIME, segmentMetadata);
+ assertColumnIsNull(schema, BuiltInVirtualColumn.ENDTIME, segmentMetadata);
+ assertColumnIsNull(schema, BuiltInVirtualColumn.CRC, segmentMetadata);
+
+ // $totalDocs comes from the context, so it is always available and never null
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.TOTALDOCS, segmentMetadata).getIntValue(0), NUM_DOCS);
+ assertNull(buildNullValueVector(schema, BuiltInVirtualColumn.TOTALDOCS, segmentMetadata));
+ }
+
+ /// Segment pruners read `ColumnMetadata` min/max without consulting the null value vector, so a column whose value
+ /// is unavailable must not publish the placeholder as its min/max. Otherwise a CONSUMING segment reporting epoch 0
+ /// for `$creationTime` sorts first in `ORDER BY $creationTime ASC LIMIT n` and prunes away the committed segments
+ /// that actually hold the answer.
+ @Test
+ public void testUnavailableMetadataPublishesNoMinMax() {
+ Schema schema = buildSegmentSchema();
+ SegmentMetadata segmentMetadata = consumingSegmentMetadata(-1L);
+ for (String column : List.of(BuiltInVirtualColumn.CREATIONTIME, BuiltInVirtualColumn.STARTTIME,
+ BuiltInVirtualColumn.ENDTIME, BuiltInVirtualColumn.CRC)) {
+ ColumnMetadata columnMetadata = buildColumnMetadata(schema, column, segmentMetadata);
+ assertNull(columnMetadata.getMinValue(), "Unavailable " + column + " should not publish a min value");
+ assertNull(columnMetadata.getMaxValue(), "Unavailable " + column + " should not publish a max value");
+
+ // The same has to hold through buildDataSource, which is the path a mutable segment takes
+ FieldSpec fieldSpec = schema.getFieldSpecFor(column);
+ VirtualColumnContext context = new VirtualColumnContext(fieldSpec, NUM_DOCS, segmentMetadata);
+ DataSourceMetadata dataSourceMetadata =
+ VirtualColumnProviderFactory.buildProvider(context).buildDataSource(context).getDataSourceMetadata();
+ assertNull(dataSourceMetadata.getMinValue(), column);
+ assertNull(dataSourceMetadata.getMaxValue(), column);
+ }
+
+ // A column whose value IS available still publishes min/max, so pruning keeps working
+ ColumnMetadata available = buildColumnMetadata(schema, BuiltInVirtualColumn.CREATIONTIME, mockSegmentMetadata());
+ assertEquals(available.getMinValue(), CREATION_TIME_MS);
+ assertEquals(available.getMaxValue(), CREATION_TIME_MS);
+ }
+
+ /// A segment that has not indexed anything yet still has to produce a well-formed, empty null bitmap rather than
+ /// failing or reporting a stale document count.
+ @Test
+ public void testNullValueVectorOnAnEmptySegment() {
+ Schema schema = buildSegmentSchema();
+ FieldSpec fieldSpec = schema.getFieldSpecFor(BuiltInVirtualColumn.STARTTIME);
+ VirtualColumnContext context = new VirtualColumnContext(fieldSpec, 0, consumingSegmentMetadata(-1L));
+ NullValueVectorReader nullValueVector =
+ VirtualColumnProviderFactory.buildProvider(context).buildNullValueVector(context);
+ assertNotNull(nullValueVector);
+ assertTrue(nullValueVector.getNullBitmap().isEmpty());
+ }
+
+ /// The null value vector is built on every data source access for a mutable segment, so the bitmap is materialized
+ /// lazily. Repeated reads must return an equal - and stable - bitmap.
+ @Test
+ public void testNullValueVectorBitmapIsStableAcrossReads() {
+ Schema schema = buildSegmentSchema();
+ FieldSpec fieldSpec = schema.getFieldSpecFor(BuiltInVirtualColumn.ENDTIME);
+ VirtualColumnContext context = new VirtualColumnContext(fieldSpec, NUM_DOCS, consumingSegmentMetadata(-1L));
+ NullValueVectorReader nullValueVector =
+ VirtualColumnProviderFactory.buildProvider(context).buildNullValueVector(context);
+ assertNotNull(nullValueVector);
+ assertEquals(nullValueVector.getNullBitmap(), nullValueVector.getNullBitmap());
+ assertEquals(nullValueVector.getNullBitmap().getCardinality(), NUM_DOCS);
+ }
+
+ /// The index container must hand the null value vector to the engine, and must not hand one back for an index type
+ /// a virtual column does not have.
+ @Test
+ public void testIndexContainerExposesTheNullValueVector()
+ throws IOException {
+ Schema schema = buildSegmentSchema();
+ FieldSpec fieldSpec = schema.getFieldSpecFor(BuiltInVirtualColumn.CRC);
+ VirtualColumnContext context = new VirtualColumnContext(fieldSpec, NUM_DOCS, consumingSegmentMetadata(-1L));
+ try (ColumnIndexContainer container =
+ VirtualColumnProviderFactory.buildProvider(context).buildColumnIndexContainer(context)) {
+ assertNotNull(container.getIndex(StandardIndexes.nullValueVector()));
+ assertNotNull(container.getIndex(StandardIndexes.forward()));
+ assertNotNull(container.getIndex(StandardIndexes.dictionary()));
+ assertNull(container.getIndex(StandardIndexes.range()));
+ }
+
+ // With the metadata available there is no null value vector at all
+ VirtualColumnContext availableContext = new VirtualColumnContext(fieldSpec, NUM_DOCS, mockSegmentMetadata());
+ try (ColumnIndexContainer container =
+ VirtualColumnProviderFactory.buildProvider(availableContext).buildColumnIndexContainer(availableContext)) {
+ assertNull(container.getIndex(StandardIndexes.nullValueVector()));
+ }
+ }
+
+ /// Every stored type the constant-value base supports must round-trip through the type check, so that a new virtual
+ /// column of any type is rejected loudly rather than silently mis-cast.
+ @Test(dataProvider = "storedTypeValues")
+ public void testValueTypeCheckAcceptsEveryStoredType(FieldSpec.DataType dataType, Object value) {
+ FieldSpec fieldSpec = new DimensionFieldSpec("col", dataType, true);
+ VirtualColumnContext context = new VirtualColumnContext(fieldSpec, NUM_DOCS, null);
+ BaseConstantValueVirtualColumnProvider provider = new BaseConstantValueVirtualColumnProvider() {
+ @Override
+ protected Object getValue(VirtualColumnContext ctx) {
+ return value;
+ }
+ };
+ assertNotNull(provider.buildDictionary(context));
+ assertNotNull(provider.buildMetadata(context));
+
+ // ... and the same type check rejects a value of the wrong type
+ BaseConstantValueVirtualColumnProvider wrong = new BaseConstantValueVirtualColumnProvider() {
+ @Override
+ protected Object getValue(VirtualColumnContext ctx) {
+ return new Object();
+ }
+ };
+ try {
+ wrong.buildDictionary(context);
+ fail("Expecting an IllegalStateException for data type: " + dataType);
+ } catch (IllegalStateException e) {
+ assertTrue(e.getMessage().contains("col"), e.getMessage());
+ }
+ }
+
+ @DataProvider(name = "storedTypeValues")
+ public Object[][] storedTypeValues() {
+ return new Object[][]{
+ {FieldSpec.DataType.INT, 1},
+ {FieldSpec.DataType.LONG, 1L},
+ {FieldSpec.DataType.FLOAT, 1.0f},
+ {FieldSpec.DataType.DOUBLE, 1.0d},
+ {FieldSpec.DataType.BIG_DECIMAL, BigDecimal.ONE},
+ {FieldSpec.DataType.STRING, "value"},
+ {FieldSpec.DataType.BYTES, new byte[]{1, 2}},
+ {FieldSpec.DataType.TIMESTAMP, 1L}
+ };
+ }
+
+ /// The type check exists so that a provider returning the wrong box type names the column and the provider instead
+ /// of throwing a bare ClassCastException from deep inside segment loading.
+ @Test
+ public void testWrongValueTypeIsRejectedWithADiagnosticMessage() {
+ FieldSpec fieldSpec = buildSegmentSchema().getFieldSpecFor(BuiltInVirtualColumn.CREATIONTIME);
+ VirtualColumnContext context = new VirtualColumnContext(fieldSpec, NUM_DOCS, mockSegmentMetadata());
+ // A LONG-stored column handed an Integer
+ BaseConstantValueVirtualColumnProvider provider = new BaseConstantValueVirtualColumnProvider() {
+ @Override
+ protected Object getValue(VirtualColumnContext ctx) {
+ return 1;
+ }
+ };
+ try {
+ provider.buildDictionary(context);
+ fail("Expecting an IllegalStateException for a wrongly typed virtual column value");
+ } catch (IllegalStateException e) {
+ String message = e.getMessage();
+ assertTrue(message.contains(BuiltInVirtualColumn.CREATIONTIME), message);
+ assertTrue(message.contains("java.lang.Integer"), message);
+ assertTrue(message.contains("java.lang.Long"), message);
+ }
+ }
+
+ /// The segment metadata is not always available (e.g. when the virtual column is built for a column that is missing
+ /// from the segment). The providers must not fail in that case.
+ @Test
+ public void testMissingSegmentMetadataFallsBackToDefaultNullValue() {
+ Schema schema = buildSegmentSchema();
+
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.CREATIONTIME, null).getLongValue(0),
+ (long) FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_TIMESTAMP);
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.STARTTIME, null).getLongValue(0),
+ (long) FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_TIMESTAMP);
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.ENDTIME, null).getLongValue(0),
+ (long) FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_TIMESTAMP);
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.CRC, null).getLongValue(0),
+ (long) FieldSpec.DEFAULT_DIMENSION_NULL_VALUE_OF_LONG);
+ assertEquals(buildDictionary(schema, BuiltInVirtualColumn.TOTALDOCS, null).getIntValue(0), NUM_DOCS);
+
+ assertColumnIsNull(schema, BuiltInVirtualColumn.CREATIONTIME, null);
+ assertColumnIsNull(schema, BuiltInVirtualColumn.STARTTIME, null);
+ assertColumnIsNull(schema, BuiltInVirtualColumn.ENDTIME, null);
+ assertColumnIsNull(schema, BuiltInVirtualColumn.CRC, null);
+ }
+}
diff --git a/pinot-spi/src/main/java/org/apache/pinot/spi/data/BuiltInVirtualColumnDefinitions.java b/pinot-spi/src/main/java/org/apache/pinot/spi/data/BuiltInVirtualColumnDefinitions.java
new file mode 100644
index 00000000000..ab3903468bc
--- /dev/null
+++ b/pinot-spi/src/main/java/org/apache/pinot/spi/data/BuiltInVirtualColumnDefinitions.java
@@ -0,0 +1,107 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.spi.data;
+
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
+import org.apache.pinot.spi.utils.CommonConstants.Segment.BuiltInVirtualColumn;
+
+
+/// Single source of truth for the shape (name, data type, single-value vs multi-value) of the built-in virtual columns.
+///
+/// The columns are declared here once because they are materialized in two independent places that must agree:
+///
+/// - the broker/controller side adds them to the table schema used for query planning, without a provider class
+/// (see `TableCache#addBuiltInVirtualColumns`), and
+/// - the server side adds them to the segment schema together with the provider that produces the values
+/// (see `VirtualColumnProviderFactory#addBuiltInVirtualColumnsToSegmentSchema`).
+///
+/// If the two sides disagreed on a data type or on single-value vs multi-value, the broker would declare one type
+/// while the server produced another. Both sides therefore build their field specs from [#DEFINITIONS].
+///
+/// The column *names* additionally live in [BuiltInVirtualColumn], which is where code that only needs to recognize a
+/// built-in virtual column by name looks them up. [#NAMES] exposes the names derived from [#DEFINITIONS] so the two
+/// can be asserted equal in a test.
+public class BuiltInVirtualColumnDefinitions {
+ private BuiltInVirtualColumnDefinitions() {
+ }
+
+ /// Shape of a single built-in virtual column. The provider class is intentionally not part of this definition: it
+ /// lives in `pinot-segment-local` and is only known to the server side.
+ public static class Definition {
+ private final String _name;
+ private final DataType _dataType;
+ private final boolean _singleValueField;
+
+ private Definition(String name, DataType dataType, boolean singleValueField) {
+ _name = name;
+ _dataType = dataType;
+ _singleValueField = singleValueField;
+ }
+
+ public String getName() {
+ return _name;
+ }
+
+ public DataType getDataType() {
+ return _dataType;
+ }
+
+ public boolean isSingleValueField() {
+ return _singleValueField;
+ }
+
+ /// Creates a new field spec for this column, without a virtual column provider. Callers that can resolve a
+ /// provider should set it on the returned spec.
+ ///
+ /// NOTE: Returns a fresh instance on every call. Field specs are mutable and are stored by reference in the
+ /// schema they are added to, so they must never be shared across schemas.
+ public DimensionFieldSpec createFieldSpec() {
+ return new DimensionFieldSpec(_name, _dataType, _singleValueField);
+ }
+ }
+
+ public static final List<Definition> DEFINITIONS = List.of(
+ new Definition(BuiltInVirtualColumn.DOCID, DataType.INT, true),
+ new Definition(BuiltInVirtualColumn.HOSTNAME, DataType.STRING, true),
+ new Definition(BuiltInVirtualColumn.SEGMENTNAME, DataType.STRING, true),
+ new Definition(BuiltInVirtualColumn.PARTITIONID, DataType.STRING, false),
+ new Definition(BuiltInVirtualColumn.CREATIONTIME, DataType.TIMESTAMP, true),
+ new Definition(BuiltInVirtualColumn.STARTTIME, DataType.TIMESTAMP, true),
+ new Definition(BuiltInVirtualColumn.ENDTIME, DataType.TIMESTAMP, true),
+ new Definition(BuiltInVirtualColumn.TOTALDOCS, DataType.INT, true),
+ new Definition(BuiltInVirtualColumn.CRC, DataType.LONG, true));
+
+ /// Names of all the built-in virtual columns, derived from [#DEFINITIONS].
+ public static final Set<String> NAMES =
+ DEFINITIONS.stream().map(Definition::getName).collect(Collectors.toUnmodifiableSet());
+
+ /// Adds the built-in virtual columns to the given schema, without a virtual column provider.
+ ///
+ /// Existing columns are left untouched, so a user-defined column of the same name always wins.
+ public static void addToSchema(Schema schema) {
+ for (Definition definition : DEFINITIONS) {
+ if (!schema.hasColumn(definition.getName())) {
+ schema.addField(definition.createFieldSpec());
+ }
+ }
+ }
+}
diff --git a/pinot-spi/src/main/java/org/apache/pinot/spi/data/SchemaInfo.java b/pinot-spi/src/main/java/org/apache/pinot/spi/data/SchemaInfo.java
index a0364117b6a..fa7c2de0335 100644
--- a/pinot-spi/src/main/java/org/apache/pinot/spi/data/SchemaInfo.java
+++ b/pinot-spi/src/main/java/org/apache/pinot/spi/data/SchemaInfo.java
@@ -66,8 +66,17 @@ public class SchemaInfo {
public SchemaInfo(Schema schema) {
_schemaName = schema.getSchemaName();
- //Removed virtual columns($docId, $hostName, $segmentName) from dimension fields count
- _numDimensionFields = schema.getDimensionFieldSpecs().size() - 3;
+ // The schema handed in here has usually been through TableCache#addBuiltInVirtualColumns, which adds the built-in
+ // virtual columns as plain dimension fields (without a provider class, so isVirtualColumn() does not identify
+ // them). Exclude them by name so that the reported count only covers the user's own dimensions, and stays correct
+ // as built-in virtual columns are added.
+ int numDimensionFields = 0;
+ for (DimensionFieldSpec fieldSpec : schema.getDimensionFieldSpecs()) {
+ if (!BuiltInVirtualColumnDefinitions.NAMES.contains(fieldSpec.getName())) {
+ numDimensionFields++;
+ }
+ }
+ _numDimensionFields = numDimensionFields;
_numDateTimeFields = schema.getDateTimeFieldSpecs().size();
_numMetricFields = schema.getMetricFieldSpecs().size();
_numComplexFields = schema.getComplexFieldSpecs().size();
diff --git a/pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java b/pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java
index e00598b582e..c117713b8c8 100644
--- a/pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java
+++ b/pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java
@@ -2246,7 +2246,36 @@ public class CommonConstants {
public static final String HOSTNAME = "$hostName";
public static final String SEGMENTNAME = "$segmentName";
public static final String PARTITIONID = "$partitionId";
- public static final Set<String> BUILT_IN_VIRTUAL_COLUMNS = Set.of(DOCID, HOSTNAME, SEGMENTNAME, PARTITIONID);
+
+ // Segment metadata virtual columns. Each of them is a constant single-value column within a segment, exposing a
+ // piece of the segment metadata to queries. When the underlying metadata is not available (e.g. on a CONSUMING
+ // segment, which has no time range and no CRC yet), the column reads as NULL.
+ //
+ // The three time columns are TIMESTAMP rather than LONG, so the unit is carried by the type instead of by the
+ // column name, and query results render them as readable timestamps.
+
+ /// Segment creation time (TIMESTAMP). This is the index creation time recorded in the segment's creation
+ /// metadata; for a CONSUMING segment it is the time the consuming segment was created.
+ public static final String CREATIONTIME = "$creationTime";
+ /// Start of the segment time range (TIMESTAMP), normalized from the time column's own unit.
+ /// NULL for segments without a time range, such as CONSUMING segments and tables without a time column.
+ public static final String STARTTIME = "$startTime";
+ /// End of the segment time range (TIMESTAMP), normalized from the time column's own unit.
+ /// NULL for segments without a time range, such as CONSUMING segments and tables without a time column.
+ public static final String ENDTIME = "$endTime";
+ /// Number of documents in the segment (INT). On a CONSUMING segment this is the number of documents indexed so
+ /// far. NOTE: This counts all the documents physically stored in the segment, so for an upsert table it also
+ /// includes the documents that have been replaced and are no longer returned by queries.
+ public static final String TOTALDOCS = "$totalDocs";
+ /// Segment CRC (LONG). NULL on CONSUMING segments, which have no CRC until they are committed.
+ /// NOTE: Do not confuse this constant with the enclosing [Segment#CRC], which is the `segment.crc` metadata key.
+ public static final String CRC = "$crc";
+
+ /// NOTE: Kept in sync with `BuiltInVirtualColumnDefinitions#DEFINITIONS`, which additionally carries the data
+ /// type and single-value/multi-value shape of each column. `BuiltInVirtualColumnDefinitionsTest` asserts the two
+ /// agree.
+ public static final Set<String> BUILT_IN_VIRTUAL_COLUMNS =
+ Set.of(DOCID, HOSTNAME, SEGMENTNAME, PARTITIONID, CREATIONTIME, STARTTIME, ENDTIME, TOTALDOCS, CRC);
}
}
diff --git a/pinot-spi/src/test/java/org/apache/pinot/spi/data/BuiltInVirtualColumnDefinitionsTest.java b/pinot-spi/src/test/java/org/apache/pinot/spi/data/BuiltInVirtualColumnDefinitionsTest.java
new file mode 100644
index 00000000000..c8cc253d808
--- /dev/null
+++ b/pinot-spi/src/test/java/org/apache/pinot/spi/data/BuiltInVirtualColumnDefinitionsTest.java
@@ -0,0 +1,99 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.spi.data;
+
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.spi.utils.CommonConstants.Segment.BuiltInVirtualColumn;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertNotSame;
+import static org.testng.Assert.assertTrue;
+
+
+public class BuiltInVirtualColumnDefinitionsTest {
+
+ /// The names are declared twice - once as constants in [BuiltInVirtualColumn] and once implicitly by
+ /// [BuiltInVirtualColumnDefinitions#DEFINITIONS]. A definition missing from the name set would stop being filtered
+ /// out of a segment's physical columns and would be counted as a user dimension by [SchemaInfo].
+ @Test
+ public void testDefinitionsMatchTheDeclaredNames() {
+ assertEquals(BuiltInVirtualColumnDefinitions.NAMES, BuiltInVirtualColumn.BUILT_IN_VIRTUAL_COLUMNS);
+ assertEquals(BuiltInVirtualColumnDefinitions.DEFINITIONS.size(),
+ BuiltInVirtualColumn.BUILT_IN_VIRTUAL_COLUMNS.size(), "Duplicate name in DEFINITIONS");
+ }
+
+ /// Every built-in virtual column name must start with `$`: that prefix is what excludes them from `SELECT *` in
+ /// both query engines.
+ @Test
+ public void testNamesAreDollarPrefixed() {
+ for (String name : BuiltInVirtualColumnDefinitions.NAMES) {
+ assertTrue(name.startsWith("$"), "Built-in virtual column must be $-prefixed: " + name);
+ }
+ }
+
+ /// Field specs are mutable and are stored by reference in the schema they are added to, so each call must return a
+ /// fresh instance - the server side mutates them to attach the provider class and, for `$segmentName`, a per-segment
+ /// value.
+ @Test
+ public void testCreateFieldSpecReturnsFreshInstances() {
+ for (BuiltInVirtualColumnDefinitions.Definition definition : BuiltInVirtualColumnDefinitions.DEFINITIONS) {
+ DimensionFieldSpec first = definition.createFieldSpec();
+ DimensionFieldSpec second = definition.createFieldSpec();
+ assertNotSame(first, second, "Field spec must not be shared for: " + definition.getName());
+ assertEquals(first, second);
+ assertEquals(first.getName(), definition.getName());
+ assertEquals(first.getDataType(), definition.getDataType());
+ assertEquals(first.isSingleValueField(), definition.isSingleValueField());
+ }
+ }
+
+ @Test
+ public void testAddToSchemaIsIdempotentAndNeverOverwrites() {
+ Schema schema = new Schema.SchemaBuilder().setSchemaName("test")
+ .addSingleValueDimension("dim", FieldSpec.DataType.STRING)
+ .build();
+ BuiltInVirtualColumnDefinitions.addToSchema(schema);
+ Set<String> afterFirst = new HashSet<>(schema.getColumnNames());
+ assertTrue(afterFirst.containsAll(BuiltInVirtualColumnDefinitions.NAMES));
+
+ // A second call must not duplicate or replace anything
+ BuiltInVirtualColumnDefinitions.addToSchema(schema);
+ assertEquals(new HashSet<>(schema.getColumnNames()), afterFirst);
+
+ for (BuiltInVirtualColumnDefinitions.Definition definition : BuiltInVirtualColumnDefinitions.DEFINITIONS) {
+ FieldSpec fieldSpec = schema.getFieldSpecFor(definition.getName());
+ assertNotNull(fieldSpec);
+ assertEquals(fieldSpec.getDataType(), definition.getDataType());
+ }
+ }
+
+ /// A user-defined column of the same name must win, so that adding a built-in virtual column never silently
+ /// changes the type of an existing user column in the broker's schema.
+ @Test
+ public void testAddToSchemaDoesNotOverrideUserColumn() {
+ Schema schema = new Schema.SchemaBuilder().setSchemaName("test")
+ .addSingleValueDimension(BuiltInVirtualColumn.TOTALDOCS, FieldSpec.DataType.STRING)
+ .build();
+ BuiltInVirtualColumnDefinitions.addToSchema(schema);
+ assertEquals(schema.getFieldSpecFor(BuiltInVirtualColumn.TOTALDOCS).getDataType(), FieldSpec.DataType.STRING);
+ }
+}
diff --git a/pinot-spi/src/test/java/org/apache/pinot/spi/data/SchemaInfoTest.java b/pinot-spi/src/test/java/org/apache/pinot/spi/data/SchemaInfoTest.java
index d3ee0e31aeb..4266bbc2e5a 100644
--- a/pinot-spi/src/test/java/org/apache/pinot/spi/data/SchemaInfoTest.java
+++ b/pinot-spi/src/test/java/org/apache/pinot/spi/data/SchemaInfoTest.java
@@ -34,6 +34,30 @@ import static org.testng.Assert.assertEquals;
public class SchemaInfoTest {
+ /// The schema `SchemaInfo` is built from has usually been through the broker's `addBuiltInVirtualColumns`, which adds
+ /// every built-in virtual column as a plain dimension field. None of them may be reported as a user dimension,
+ /// whatever their number.
+ @Test
+ public void testBuiltInVirtualColumnsAreNotCounted() {
+ Schema schema = new Schema.SchemaBuilder().setSchemaName("TestSchema")
+ .addDimensionField("dim1", FieldSpec.DataType.STRING)
+ .addDimensionField("dim2", FieldSpec.DataType.INT)
+ .addDateTimeField("dt1", FieldSpec.DataType.LONG, "1:HOURS:EPOCH", "1:HOURS")
+ .addMetricField("metric", INT)
+ .build();
+ assertEquals(new SchemaInfo(schema).getNumDimensionFields(), 2);
+
+ // Adding the built-in virtual columns must not change the reported dimension count
+ BuiltInVirtualColumnDefinitions.addToSchema(schema);
+ assertEquals(schema.getDimensionFieldSpecs().size(),
+ 2 + CommonConstants.Segment.BuiltInVirtualColumn.BUILT_IN_VIRTUAL_COLUMNS.size());
+ SchemaInfo schemaInfo = new SchemaInfo(schema);
+ assertEquals(schemaInfo.getNumDimensionFields(), 2);
+ assertEquals(schemaInfo.getNumDateTimeFields(), 1);
+ assertEquals(schemaInfo.getNumMetricFields(), 1);
+ assertEquals(schemaInfo.getNumComplexFields(), 0);
+ }
+
@Test
public void testSchemaInfoSerDeserWithVirtualColumns()
throws IOException {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org