You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "rohityadav1993 (via GitHub)" <gi...@apache.org> on 2025/09/01 08:28:41 UTC
[PR] Columnar segment build clean [pinot]
rohityadav1993 opened a new pull request, #16727:
URL: https://github.com/apache/pinot/pull/16727
`feature``performance``release-notes`
# Feature
Feature to build segment in columnar fashion for columnar input data sources like Pinot segment, Parquet files, etc.
## Description
The requirement arises from issue: https://github.com/apache/pinot/issues/16461:
In short, when minion refreshes a servers segment for schema or index config change. It rebuilds segment row wise. There are two bottlenecks observed:
1. Row wise segment stats collection for each column in SegmentIndexCreationDriverImpl https://github.com/apache/pinot/blob/868e2c9e7ced50c865bfe4fd0aa0e4200f6666ae/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java#L573
2. Row wise segment generation in SegmentIndexCreationDriverImpl https://github.com/apache/pinot/blob/868e2c9e7ced50c865bfe4fd0aa0e4200f6666ae/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java#L266
Record wise generation also reprocesses transformations. This is not needed unless there is a change in transformation logic both in gather stats step and segment regeneration step. The bottlenecks can be seen in the CPU profile in the attached issue https://github.com/apache/pinot/issues/16461
# Design and Implementation
New interfaces:
1. ColumnReader: used to read column data from various data sources for columnar segment building
1. Sequential iteration over all values in a column using hasNext() and next()
2. Null value detection for the current value
2. ColumnReaderFactory: create column readers for various data sources
* such as Pinot segments, Parquet files, etc
1. Creating column readers for existing columns
2. Creating default value readers for new columns
3. Data type conversions between source and target schemas
(**) Use `release-notes` label for scenarios like:
- New configuration options
- Deprecation of configurations
- Signature changes to public methods/interfaces
- New plugins added or old plugins removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2434778172
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.stats;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Stats container that efficiently collects statistics from columnar data using ColumnReader instances.
+ *
+ * <p>This implementation collects statistics by iterating column-wise instead of row-wise,
+ * which is more efficient for columnar data sources. It supports:
+ * <ul>
+ * <li>Column-wise statistics collection</li>
+ * <li>Existing columns from source data</li>
+ * <li>New columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ *
+ * <p>The statistics are collected using the same underlying collectors as the row-based approach
+ * (SegmentPreIndexStatsCollectorImpl) but with more efficient column-wise iteration.
+ */
+public class ColumnarSegmentPreIndexStatsContainer implements SegmentPreIndexStatsCollector {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentPreIndexStatsContainer.class);
+
+ private final Map<String, ColumnReader> _columnReaders;
+ private final StatsCollectorConfig _statsCollectorConfig;
+ private final Schema _targetSchema;
+ private final Map<String, AbstractColumnStatisticsCollector> _columnStatsCollectorMap;
+ private int _totalDocCount;
+
+ /**
+ * Create a ColumnarSegmentPreIndexStatsContainer.
+ *
+ * @param columnReaders Map of column name to ColumnReader instances
+ * @param statsCollectorConfig Configuration for statistics collection
+ */
+ public ColumnarSegmentPreIndexStatsContainer(Map<String, ColumnReader> columnReaders,
+ StatsCollectorConfig statsCollectorConfig) {
+ _columnReaders = columnReaders;
+ _statsCollectorConfig = statsCollectorConfig;
+ _targetSchema = statsCollectorConfig.getSchema();
+ _columnStatsCollectorMap = new HashMap<>();
+ _totalDocCount = -1; // indicates unset
+
+ initializeStatsCollectors();
+ collectColumnStats();
+ }
+
+ /**
+ * Initialize stats collectors for all columns in the target schema.
+ */
+ private void initializeStatsCollectors() {
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
+ switch (fieldSpec.getDataType().getStoredType()) {
+ case INT:
+ _columnStatsCollectorMap.put(columnName,
+ new IntColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case LONG:
+ _columnStatsCollectorMap.put(columnName,
+ new LongColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case FLOAT:
+ _columnStatsCollectorMap.put(columnName,
+ new FloatColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case DOUBLE:
+ _columnStatsCollectorMap.put(columnName,
+ new DoubleColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BIG_DECIMAL:
+ _columnStatsCollectorMap.put(columnName,
+ new BigDecimalColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case STRING:
+ _columnStatsCollectorMap.put(columnName,
+ new StringColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BYTES:
+ _columnStatsCollectorMap.put(columnName,
+ new BytesColumnPredIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case MAP:
+ _columnStatsCollectorMap.put(columnName,
+ new MapColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ default:
+ throw new IllegalStateException("Unsupported data type: " + fieldSpec.getDataType());
+ }
+ }
+ }
+
+ /**
+ * Collect stats by iterating column-wise using the provided ColumnReader instances.
+ */
+ private void collectColumnStats() {
+ LOGGER.info("Collecting stats for {} columns using column-wise iteration", _columnReaders.size());
+
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
+ AbstractColumnStatisticsCollector statsCollector = _columnStatsCollectorMap.get(columnName);
+ ColumnReader columnReader = _columnReaders.get(columnName);
+
+ if (columnReader == null) {
+ throw new RuntimeException("Column reader for column " + columnName + " not found");
+ }
+
+ LOGGER.debug("Collecting stats for column: {}", columnName);
+ collectStatsFromColumnReader(columnName, columnReader, statsCollector);
+
+ // Seal the stats collector
+ statsCollector.seal();
+ }
+ }
+
+ /**
+ * Collect stats from a column reader by iterating over all values using the iterator pattern.
+ */
+ private void collectStatsFromColumnReader(String columnName, ColumnReader columnReader,
+ AbstractColumnStatisticsCollector statsCollector) {
+ try {
+ // Reset the column reader to start from the beginning
+ columnReader.rewind();
+
+ int docCount = 0;
+ while (columnReader.hasNext()) {
+ Object value = columnReader.next();
+ statsCollector.collect(value);
Review Comment:
It seems it is to handle transform related exceptions (#9377 ). We are not doing transformations yet in columnar builds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2431188236
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.stats;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Stats container that efficiently collects statistics from columnar data using ColumnReader instances.
+ *
+ * <p>This implementation collects statistics by iterating column-wise instead of row-wise,
+ * which is more efficient for columnar data sources. It supports:
+ * <ul>
+ * <li>Column-wise statistics collection</li>
+ * <li>Existing columns from source data</li>
+ * <li>New columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ *
+ * <p>The statistics are collected using the same underlying collectors as the row-based approach
+ * (SegmentPreIndexStatsCollectorImpl) but with more efficient column-wise iteration.
+ */
+public class ColumnarSegmentPreIndexStatsContainer implements SegmentPreIndexStatsCollector {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentPreIndexStatsContainer.class);
+
+ private final Map<String, ColumnReader> _columnReaders;
+ private final StatsCollectorConfig _statsCollectorConfig;
+ private final Schema _targetSchema;
+ private final Map<String, AbstractColumnStatisticsCollector> _columnStatsCollectorMap;
+ private int _totalDocCount;
+
+ /**
+ * Create a ColumnarSegmentPreIndexStatsContainer.
+ *
+ * @param columnReaders Map of column name to ColumnReader instances
+ * @param statsCollectorConfig Configuration for statistics collection
+ */
+ public ColumnarSegmentPreIndexStatsContainer(Map<String, ColumnReader> columnReaders,
+ StatsCollectorConfig statsCollectorConfig) {
+ _columnReaders = columnReaders;
+ _statsCollectorConfig = statsCollectorConfig;
+ _targetSchema = statsCollectorConfig.getSchema();
+ _columnStatsCollectorMap = new HashMap<>();
+ _totalDocCount = -1; // indicates unset
+
+ initializeStatsCollectors();
Review Comment:
There isn't a requirement for lazy loading(initializeStatsCollectors) by invoking init separately.
Column stats are collected in build method, `buildColumnar()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2366361527
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReaderImpl.java:
##########
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.readers;
+
+import java.io.IOException;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.utils.PinotDataType;
+import org.apache.pinot.segment.local.recordtransformer.DataTypeTransformer;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Implementation of ColumnReader for Pinot segments.
+ *
+ * <p>This class wraps the existing PinotSegmentColumnReader and provides the ColumnReader interface
+ * for columnar segment building. It handles:
+ * <ul>
+ * <li>Reading column values from Pinot segments</li>
+ * <li>Data type conversions when target schema differs from source</li>
+ * <li>Null value detection</li>
+ * <li>Resource cleanup</li>
+ * </ul>
+ */
+public class PinotSegmentColumnReaderImpl implements ColumnReader {
+ private static final Logger LOGGER = LoggerFactory.getLogger(PinotSegmentColumnReaderImpl.class);
+
+ private final PinotSegmentColumnReader _segmentColumnReader;
+ private final String _columnName;
+ private final int _numDocs;
+ private final boolean _needsConversion;
+ private final FieldSpec _targetFieldSpec;
+
+ private int _currentIndex;
+ private Object _currentValue;
+ private boolean _currentIsNull;
+
+ /**
+ * Create a PinotSegmentColumnReaderImpl for an existing column in the segment.
+ *
+ * @param indexSegment Source segment to read from
+ * @param columnName Name of the column
+ * @param targetFieldSpec Target field specification (may differ from source for data type conversion)
+ */
+ public PinotSegmentColumnReaderImpl(IndexSegment indexSegment, String columnName, FieldSpec targetFieldSpec) {
+ _segmentColumnReader = new PinotSegmentColumnReader(indexSegment, columnName);
+ _columnName = columnName;
+ _numDocs = indexSegment.getSegmentMetadata().getTotalDocs();
+ _targetFieldSpec = targetFieldSpec;
+ _currentIndex = 0;
+
+ // Check if data type conversion is needed
+ FieldSpec sourceFieldSpec = indexSegment.getSegmentMetadata().getSchema().getFieldSpecFor(columnName);
+ _needsConversion = sourceFieldSpec != null
Review Comment:
This was a conscious decision to not support `RecordTransformer`. Supporting `RecordTransformer` and getting columnar benefits would be a complex logic. See current implementation of transformPipeline of records: https://github.com/apache/pinot/blob/7f60f1316030059b538d299c91e056b29735c609/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java#L293
For external sources, it may even be more convenient to remodel data externally (e.g. spark job)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2430164422
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/creator/impl/ColumnarSegmentBuildingTest.java:
##########
@@ -0,0 +1,500 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl;
+
+import com.google.common.collect.Lists;
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.segment.readers.PinotSegmentColumnReaderFactory;
+import org.apache.pinot.segment.spi.ImmutableSegment;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.data.readers.RecordReader;
+import org.apache.pinot.spi.data.readers.RecordReaderConfig;
+import org.apache.pinot.spi.utils.ReadMode;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+
+/**
+ * Tests for columnar segment building functionality.
+ *
+ * <p>This test class validates the new columnar segment building capability including:
+ * <ul>
+ * <li>ColumnReader interface implementations</li>
+ * <li>ColumnReaderFactory for different data sources</li>
+ * <li>Columnar segment building vs row-major building equivalence</li>
+ * <li>Support for new columns with default values</li>
+ * </ul>
+ */
+public class ColumnarSegmentBuildingTest {
+ private static final String TEMP_DIR = System.getProperty("java.io.tmpdir");
+ private static final String TABLE_NAME = "testTable";
+ private static final String SEGMENT_NAME = "testSegment";
+
+ // Test columns
+ private static final String STRING_COL_1 = "stringCol1";
+ private static final String STRING_COL_2 = "stringCol2";
+ private static final String INT_COL_1 = "intCol1";
+ private static final String INT_COL_2 = "intCol2";
+ private static final String LONG_COL = "longCol";
+ private static final String FLOAT_COL = "floatCol";
+ private static final String DOUBLE_COL = "doubleCol";
+ private static final String BIG_DECIMAL_COL = "bigDecimalCol";
+ private static final String BYTES_COL = "bytesCol";
+ private static final String TIME_COL = "timeCol";
+ private static final String MV_INT_COL = "mvIntCol";
+ private static final String MV_STRING_COL = "mvStringCol";
+
+ // New column for testing default value handling
+ private static final String NEW_STRING_COL = "newStringCol";
+ private static final String NEW_INT_COL = "newIntCol";
+
+ private File _tempDir;
+ private Schema _originalSchema;
+ private Schema _extendedSchema; // Schema with additional columns
+ private TableConfig _tableConfig;
+ private List<GenericRow> _testData;
+
+ @BeforeClass
+ public void setUp() throws IOException {
+ _tempDir = new File(TEMP_DIR, "ColumnarSegmentBuildingTest");
+ FileUtils.deleteQuietly(_tempDir);
+ _tempDir.mkdirs();
+
+ // Create original schema
+ _originalSchema = new Schema.SchemaBuilder()
+ .addSingleValueDimension(STRING_COL_1, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(STRING_COL_2, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(INT_COL_1, FieldSpec.DataType.INT)
+ .addSingleValueDimension(INT_COL_2, FieldSpec.DataType.INT)
+ .addSingleValueDimension(LONG_COL, FieldSpec.DataType.LONG)
+ .addSingleValueDimension(FLOAT_COL, FieldSpec.DataType.FLOAT)
+ .addSingleValueDimension(DOUBLE_COL, FieldSpec.DataType.DOUBLE)
+ .addSingleValueDimension(BIG_DECIMAL_COL, FieldSpec.DataType.BIG_DECIMAL)
+ .addSingleValueDimension(BYTES_COL, FieldSpec.DataType.BYTES)
+ .addMultiValueDimension(MV_INT_COL, FieldSpec.DataType.INT)
+ .addMultiValueDimension(MV_STRING_COL, FieldSpec.DataType.STRING)
+ .addDateTime(TIME_COL, FieldSpec.DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS")
+ .build();
+
+ // Create extended schema with additional columns
+ _extendedSchema = new Schema.SchemaBuilder()
+ .addSingleValueDimension(STRING_COL_1, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(STRING_COL_2, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(INT_COL_1, FieldSpec.DataType.INT)
+ .addSingleValueDimension(INT_COL_2, FieldSpec.DataType.INT)
+ .addSingleValueDimension(LONG_COL, FieldSpec.DataType.LONG)
+ .addSingleValueDimension(FLOAT_COL, FieldSpec.DataType.FLOAT)
+ .addSingleValueDimension(DOUBLE_COL, FieldSpec.DataType.DOUBLE)
+ .addSingleValueDimension(BIG_DECIMAL_COL, FieldSpec.DataType.BIG_DECIMAL)
+ .addSingleValueDimension(BYTES_COL, FieldSpec.DataType.BYTES)
+ .addMultiValueDimension(MV_INT_COL, FieldSpec.DataType.INT)
+ .addMultiValueDimension(MV_STRING_COL, FieldSpec.DataType.STRING)
+ .addDateTime(TIME_COL, FieldSpec.DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS")
+ .addSingleValueDimension(NEW_STRING_COL, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(NEW_INT_COL, FieldSpec.DataType.INT)
+ .build();
+
+ // Create table config
+ _tableConfig = new TableConfigBuilder(TableType.OFFLINE)
+ .setTableName(TABLE_NAME)
+ .setTimeColumnName(TIME_COL)
+ .setInvertedIndexColumns(Lists.newArrayList(STRING_COL_1, INT_COL_1))
Review Comment:
There is no logic based on index type in columnar segment build. This will not add value to the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2430151427
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -396,27 +372,58 @@ public void indexColumn(String columnName, @Nullable int[] sortedDocIds, IndexSe
if (sortedDocIds != null) {
int onDiskDocId = 0;
for (int docId : sortedDocIds) {
- // If validDodIds are provided, only index column if it's a valid doc
- if (validDocIds == null || validDocIds.contains(docId)) {
- indexColumnValue(colReader, creatorsByIndex, columnName, fieldSpec, dictionaryCreator, docId, onDiskDocId,
- nullVec);
- onDiskDocId++;
- }
+ indexColumnValue(colReader, creatorsByIndex, columnName, fieldSpec, dictionaryCreator, docId, onDiskDocId,
+ nullVec);
+ onDiskDocId++;
}
} else {
- int onDiskDocId = 0;
for (int docId = 0; docId < numDocs; docId++) {
- // If validDodIds are provided, only index column if it's a valid doc
- if (validDocIds == null || validDocIds.contains(docId)) {
- indexColumnValue(colReader, creatorsByIndex, columnName, fieldSpec, dictionaryCreator, docId, onDiskDocId,
- nullVec);
- onDiskDocId++;
- }
+ indexColumnValue(colReader, creatorsByIndex, columnName, fieldSpec, dictionaryCreator, docId, docId, nullVec);
}
}
}
}
+ /**
+ * Index a column using a ColumnReader (column-major approach).
+ * This method processes the column values using the iterator pattern from ColumnReader.
+ *
+ * @param columnName Name of the column to index
+ * @param columnReader ColumnReader for the column data
+ * @throws IOException if indexing fails
+ */
+ @Override
+ public void indexColumn(String columnName, ColumnReader columnReader) throws IOException {
+ Map<IndexType<?, ?, ?>, IndexCreator> creatorsByIndex = _creatorsByColAndIndex.get(columnName);
+ NullValueVectorCreator nullVec = _nullValueVectorCreatorMap.get(columnName);
+ FieldSpec fieldSpec = _schema.getFieldSpecFor(columnName);
+ SegmentDictionaryCreator dictionaryCreator = _dictionaryCreatorMap.get(columnName);
+ Object defaultNullValue = fieldSpec.getDefaultNullValue();
+ Object reuseColumnValueToIndex;
+
+ // Reset column reader to start from beginning
+ columnReader.rewind();
+
+ int docId = 0;
+ while (columnReader.hasNext()) {
+ reuseColumnValueToIndex = columnReader.next();
+
+ // Handle null values
+ if (nullVec != null && reuseColumnValueToIndex == null) {
+ nullVec.setNull(docId);
+ reuseColumnValueToIndex = defaultNullValue;
Review Comment:
Good catch, so in current implementation we are doing segment to segment transformation and with above we will never have a null value to index.
For non Pinot segment datasource, we can have a null value, I previously removed a util method to do null check on the read value basedon PR comment.
Let me handle case for a read null value with default value when we don't have null handling enabled. cc: @Jackie-Jiang , @noob-se7en to review as well.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java:
##########
@@ -171,31 +170,72 @@ public void init(SegmentGeneratorConfig config, RecordReader recordReader)
new TransformPipeline(config.getTableConfig(), config.getSchema()));
}
+ /**
+ * Initialize the driver for columnar segment building using a ColumnReaderFactory.
+ * This method sets up the driver to use column-wise input data access instead of row-wise.
+ *
+ * @param config Segment generator configuration
+ * @param columnReaderFactory Factory for creating column readers
+ * @throws Exception if initialization fails
+ */
+ public void init(SegmentGeneratorConfig config, ColumnReaderFactory columnReaderFactory)
+ throws Exception {
+ // Initialize the column reader factory with target schema
+ columnReaderFactory.init(config.getSchema());
+
+ // Create all column readers for the target schema
+ Map<String, ColumnReader> columnReaders = columnReaderFactory.getAllColumnReaders();
+
+ // Create columnar data source
+ ColumnarSegmentCreationDataSource columnarDataSource =
+ new ColumnarSegmentCreationDataSource(columnReaderFactory, columnReaders);
+
+ // Use the existing init method with columnar data source and no transform pipeline
+ init(config, columnarDataSource, null);
Review Comment:
Yes, we will add transformation support later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "krishan1390 (via GitHub)" <gi...@apache.org>.
krishan1390 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2425240734
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.stats;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Stats container that efficiently collects statistics from columnar data using ColumnReader instances.
+ *
+ * <p>This implementation collects statistics by iterating column-wise instead of row-wise,
+ * which is more efficient for columnar data sources. It supports:
+ * <ul>
+ * <li>Column-wise statistics collection</li>
+ * <li>Existing columns from source data</li>
+ * <li>New columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ *
+ * <p>The statistics are collected using the same underlying collectors as the row-based approach
+ * (SegmentPreIndexStatsCollectorImpl) but with more efficient column-wise iteration.
+ */
+public class ColumnarSegmentPreIndexStatsContainer implements SegmentPreIndexStatsCollector {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentPreIndexStatsContainer.class);
+
+ private final Map<String, ColumnReader> _columnReaders;
+ private final StatsCollectorConfig _statsCollectorConfig;
+ private final Schema _targetSchema;
+ private final Map<String, AbstractColumnStatisticsCollector> _columnStatsCollectorMap;
+ private int _totalDocCount;
+
+ /**
+ * Create a ColumnarSegmentPreIndexStatsContainer.
+ *
+ * @param columnReaders Map of column name to ColumnReader instances
+ * @param statsCollectorConfig Configuration for statistics collection
+ */
+ public ColumnarSegmentPreIndexStatsContainer(Map<String, ColumnReader> columnReaders,
+ StatsCollectorConfig statsCollectorConfig) {
+ _columnReaders = columnReaders;
+ _statsCollectorConfig = statsCollectorConfig;
+ _targetSchema = statsCollectorConfig.getSchema();
+ _columnStatsCollectorMap = new HashMap<>();
+ _totalDocCount = -1; // indicates unset
+
+ initializeStatsCollectors();
+ collectColumnStats();
+ }
+
+ /**
+ * Initialize stats collectors for all columns in the target schema.
+ */
+ private void initializeStatsCollectors() {
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
Review Comment:
we don't ignore virtual column in row based collector and SegmentIndexCreationDriverImpl._segmentStats has stats for virtual columns.
why are we ignoring it here ? not sure if ignore virtual columns in _segmentStats will break some other assumption
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.stats;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Stats container that efficiently collects statistics from columnar data using ColumnReader instances.
+ *
+ * <p>This implementation collects statistics by iterating column-wise instead of row-wise,
+ * which is more efficient for columnar data sources. It supports:
+ * <ul>
+ * <li>Column-wise statistics collection</li>
+ * <li>Existing columns from source data</li>
+ * <li>New columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ *
+ * <p>The statistics are collected using the same underlying collectors as the row-based approach
+ * (SegmentPreIndexStatsCollectorImpl) but with more efficient column-wise iteration.
+ */
+public class ColumnarSegmentPreIndexStatsContainer implements SegmentPreIndexStatsCollector {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentPreIndexStatsContainer.class);
+
+ private final Map<String, ColumnReader> _columnReaders;
+ private final StatsCollectorConfig _statsCollectorConfig;
+ private final Schema _targetSchema;
+ private final Map<String, AbstractColumnStatisticsCollector> _columnStatsCollectorMap;
+ private int _totalDocCount;
+
+ /**
+ * Create a ColumnarSegmentPreIndexStatsContainer.
+ *
+ * @param columnReaders Map of column name to ColumnReader instances
+ * @param statsCollectorConfig Configuration for statistics collection
+ */
+ public ColumnarSegmentPreIndexStatsContainer(Map<String, ColumnReader> columnReaders,
+ StatsCollectorConfig statsCollectorConfig) {
+ _columnReaders = columnReaders;
+ _statsCollectorConfig = statsCollectorConfig;
+ _targetSchema = statsCollectorConfig.getSchema();
+ _columnStatsCollectorMap = new HashMap<>();
+ _totalDocCount = -1; // indicates unset
+
+ initializeStatsCollectors();
Review Comment:
initializeStatsCollectors and collectColumnStats needn't be called in constructor right ?
initializeStatsCollectors should be called in init() method ?
and similarly collectColumnStats in build() method ?
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java:
##########
@@ -171,31 +170,72 @@ public void init(SegmentGeneratorConfig config, RecordReader recordReader)
new TransformPipeline(config.getTableConfig(), config.getSchema()));
}
+ /**
+ * Initialize the driver for columnar segment building using a ColumnReaderFactory.
+ * This method sets up the driver to use column-wise input data access instead of row-wise.
+ *
+ * @param config Segment generator configuration
+ * @param columnReaderFactory Factory for creating column readers
+ * @throws Exception if initialization fails
+ */
+ public void init(SegmentGeneratorConfig config, ColumnReaderFactory columnReaderFactory)
+ throws Exception {
+ // Initialize the column reader factory with target schema
+ columnReaderFactory.init(config.getSchema());
+
+ // Create all column readers for the target schema
+ Map<String, ColumnReader> columnReaders = columnReaderFactory.getAllColumnReaders();
+
+ // Create columnar data source
+ ColumnarSegmentCreationDataSource columnarDataSource =
+ new ColumnarSegmentCreationDataSource(columnReaderFactory, columnReaders);
+
+ // Use the existing init method with columnar data source and no transform pipeline
+ init(config, columnarDataSource, null);
Review Comment:
noob question - is it OK to skip transform pipeline for all use cases where columnar segment build is used ?
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.stats;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Stats container that efficiently collects statistics from columnar data using ColumnReader instances.
+ *
+ * <p>This implementation collects statistics by iterating column-wise instead of row-wise,
+ * which is more efficient for columnar data sources. It supports:
+ * <ul>
+ * <li>Column-wise statistics collection</li>
+ * <li>Existing columns from source data</li>
+ * <li>New columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ *
+ * <p>The statistics are collected using the same underlying collectors as the row-based approach
+ * (SegmentPreIndexStatsCollectorImpl) but with more efficient column-wise iteration.
+ */
+public class ColumnarSegmentPreIndexStatsContainer implements SegmentPreIndexStatsCollector {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentPreIndexStatsContainer.class);
+
+ private final Map<String, ColumnReader> _columnReaders;
+ private final StatsCollectorConfig _statsCollectorConfig;
+ private final Schema _targetSchema;
+ private final Map<String, AbstractColumnStatisticsCollector> _columnStatsCollectorMap;
+ private int _totalDocCount;
+
+ /**
+ * Create a ColumnarSegmentPreIndexStatsContainer.
+ *
+ * @param columnReaders Map of column name to ColumnReader instances
+ * @param statsCollectorConfig Configuration for statistics collection
+ */
+ public ColumnarSegmentPreIndexStatsContainer(Map<String, ColumnReader> columnReaders,
+ StatsCollectorConfig statsCollectorConfig) {
+ _columnReaders = columnReaders;
+ _statsCollectorConfig = statsCollectorConfig;
+ _targetSchema = statsCollectorConfig.getSchema();
+ _columnStatsCollectorMap = new HashMap<>();
+ _totalDocCount = -1; // indicates unset
+
+ initializeStatsCollectors();
+ collectColumnStats();
+ }
+
+ /**
+ * Initialize stats collectors for all columns in the target schema.
+ */
+ private void initializeStatsCollectors() {
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
+ switch (fieldSpec.getDataType().getStoredType()) {
+ case INT:
+ _columnStatsCollectorMap.put(columnName,
+ new IntColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case LONG:
+ _columnStatsCollectorMap.put(columnName,
+ new LongColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case FLOAT:
+ _columnStatsCollectorMap.put(columnName,
+ new FloatColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case DOUBLE:
+ _columnStatsCollectorMap.put(columnName,
+ new DoubleColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BIG_DECIMAL:
+ _columnStatsCollectorMap.put(columnName,
+ new BigDecimalColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case STRING:
+ _columnStatsCollectorMap.put(columnName,
+ new StringColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BYTES:
+ _columnStatsCollectorMap.put(columnName,
+ new BytesColumnPredIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case MAP:
+ _columnStatsCollectorMap.put(columnName,
+ new MapColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ default:
+ throw new IllegalStateException("Unsupported data type: " + fieldSpec.getDataType());
+ }
+ }
+ }
+
+ /**
+ * Collect stats by iterating column-wise using the provided ColumnReader instances.
+ */
+ private void collectColumnStats() {
+ LOGGER.info("Collecting stats for {} columns using column-wise iteration", _columnReaders.size());
+
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
+ AbstractColumnStatisticsCollector statsCollector = _columnStatsCollectorMap.get(columnName);
+ ColumnReader columnReader = _columnReaders.get(columnName);
+
+ if (columnReader == null) {
+ throw new RuntimeException("Column reader for column " + columnName + " not found");
+ }
+
+ LOGGER.debug("Collecting stats for column: {}", columnName);
+ collectStatsFromColumnReader(columnName, columnReader, statsCollector);
+
+ // Seal the stats collector
+ statsCollector.seal();
+ }
+ }
+
+ /**
+ * Collect stats from a column reader by iterating over all values using the iterator pattern.
+ */
+ private void collectStatsFromColumnReader(String columnName, ColumnReader columnReader,
+ AbstractColumnStatisticsCollector statsCollector) {
+ try {
+ // Reset the column reader to start from the beginning
+ columnReader.rewind();
+
+ int docCount = 0;
+ while (columnReader.hasNext()) {
+ Object value = columnReader.next();
+ statsCollector.collect(value);
+ docCount++;
+ }
+
+ // if totalDocCount is unset then set total doc count from the first column
+ if (_totalDocCount == -1) {
+ _totalDocCount = docCount;
+ } else if (_totalDocCount != docCount) {
+ // all columns should have same count
+ LOGGER.warn("Column {} has {} documents, but expected {} documents",
+ columnName, docCount, _totalDocCount);
+ }
+ } catch (IOException e) {
+ LOGGER.error("Failed to collect stats for column: {}", columnName, e);
+ throw new RuntimeException("Failed to collect stats for column: " + columnName, e);
+ }
+ }
+
+ @Override
+ public ColumnStatistics getColumnProfileFor(String column) {
+ return _columnStatsCollectorMap.get(column);
+ }
+
+ @Override
+ public int getTotalDocCount() {
+ return _totalDocCount;
+ }
+
+ @Override
+ public void init() {
+ // Already initialized in constructor
+ }
+
+ @Override
+ public void build() {
+ // Stats are already collected in constructor
+ }
+
+ @Override
+ public void logStats() {
+ LOGGER.info("Columnar segment stats collection completed for {} columns with {} documents",
Review Comment:
we should log the full stats here. same comment as earlier to refactor common code of this class and SegmentPreIndexStatsCollectorImpl to get same benefits without duplicate code.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java:
##########
@@ -677,4 +706,63 @@ public int getSkippedRowsFound() {
public int getSanitizedRowsFound() {
return _sanitizedRowsFound;
}
+
+ /**
+ * Build segment using columnar approach.
+ * This method builds the segment by processing data column-wise instead of row-wise.
+ *
+ * @throws Exception if segment building fails
+ */
+ public void buildColumnar() throws Exception {
+ if (!(_dataSource instanceof ColumnarSegmentCreationDataSource)) {
+ throw new IllegalStateException("buildColumnar() can only be called after initColumnar()");
+ }
+
+ ColumnarSegmentCreationDataSource columnarDataSource = (ColumnarSegmentCreationDataSource) _dataSource;
+ Map<String, ColumnReader> columnReaders = columnarDataSource.getColumnReaders();
+
+ LOGGER.info("Starting columnar segment building for {} columns", columnReaders.size());
+
+ // Reuse existing stats collection and index creation info logic
+ LOGGER.debug("Start building StatsCollector!");
+ collectStatsAndIndexCreationInfo();
+ LOGGER.info("Finished building StatsCollector!");
+ LOGGER.info("Collected stats for {} documents", _totalDocs);
+
+ if (_totalDocs == 0) {
+ LOGGER.warn("No documents found in data source");
+ handlePostCreation();
+ return;
+ }
+
+ try {
+ // Initialize the index creation using the per-column statistics information
+ _indexCreator.init(_config, _segmentIndexCreationInfo, _indexCreationInfoMap, _dataSchema, _tempIndexDir, null);
Review Comment:
whats the consequnce of not setting immutableToMutableIdMap which is set in buildByColumn() too
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java:
##########
@@ -171,31 +170,72 @@ public void init(SegmentGeneratorConfig config, RecordReader recordReader)
new TransformPipeline(config.getTableConfig(), config.getSchema()));
}
+ /**
+ * Initialize the driver for columnar segment building using a ColumnReaderFactory.
+ * This method sets up the driver to use column-wise input data access instead of row-wise.
+ *
+ * @param config Segment generator configuration
+ * @param columnReaderFactory Factory for creating column readers
+ * @throws Exception if initialization fails
+ */
+ public void init(SegmentGeneratorConfig config, ColumnReaderFactory columnReaderFactory)
+ throws Exception {
+ // Initialize the column reader factory with target schema
+ columnReaderFactory.init(config.getSchema());
+
+ // Create all column readers for the target schema
+ Map<String, ColumnReader> columnReaders = columnReaderFactory.getAllColumnReaders();
+
+ // Create columnar data source
+ ColumnarSegmentCreationDataSource columnarDataSource =
+ new ColumnarSegmentCreationDataSource(columnReaderFactory, columnReaders);
+
+ // Use the existing init method with columnar data source and no transform pipeline
+ init(config, columnarDataSource, null);
+
+ LOGGER.info("Initialized SegmentIndexCreationDriverImpl for columnar building with {} columns",
+ columnReaders.size());
+ }
+
public void init(SegmentGeneratorConfig config, SegmentCreationDataSource dataSource,
TransformPipeline transformPipeline)
throws Exception {
_config = config;
- _recordReader = dataSource.getRecordReader();
_dataSchema = config.getSchema();
_continueOnError = config.isContinueOnError();
+ String readerClassName = null;
+
+ // Handle columnar data sources differently
+ if (dataSource instanceof ColumnarSegmentCreationDataSource) {
+ // For columnar data sources, we don't have a record reader
+ _recordReader = null;
Review Comment:
lets annotate _recordReader and _transformPipeline as Nullable
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.stats;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Stats container that efficiently collects statistics from columnar data using ColumnReader instances.
+ *
+ * <p>This implementation collects statistics by iterating column-wise instead of row-wise,
+ * which is more efficient for columnar data sources. It supports:
+ * <ul>
+ * <li>Column-wise statistics collection</li>
+ * <li>Existing columns from source data</li>
+ * <li>New columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ *
+ * <p>The statistics are collected using the same underlying collectors as the row-based approach
+ * (SegmentPreIndexStatsCollectorImpl) but with more efficient column-wise iteration.
+ */
+public class ColumnarSegmentPreIndexStatsContainer implements SegmentPreIndexStatsCollector {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentPreIndexStatsContainer.class);
+
+ private final Map<String, ColumnReader> _columnReaders;
+ private final StatsCollectorConfig _statsCollectorConfig;
+ private final Schema _targetSchema;
+ private final Map<String, AbstractColumnStatisticsCollector> _columnStatsCollectorMap;
+ private int _totalDocCount;
+
+ /**
+ * Create a ColumnarSegmentPreIndexStatsContainer.
+ *
+ * @param columnReaders Map of column name to ColumnReader instances
+ * @param statsCollectorConfig Configuration for statistics collection
+ */
+ public ColumnarSegmentPreIndexStatsContainer(Map<String, ColumnReader> columnReaders,
+ StatsCollectorConfig statsCollectorConfig) {
+ _columnReaders = columnReaders;
+ _statsCollectorConfig = statsCollectorConfig;
+ _targetSchema = statsCollectorConfig.getSchema();
+ _columnStatsCollectorMap = new HashMap<>();
+ _totalDocCount = -1; // indicates unset
+
+ initializeStatsCollectors();
+ collectColumnStats();
+ }
+
+ /**
+ * Initialize stats collectors for all columns in the target schema.
+ */
+ private void initializeStatsCollectors() {
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
Review Comment:
we can probably abstract the common logic in SegmentPreIndexStatsCollectorImpl and this class to populate _columnStatsCollectorMap to avoid redundant code.
Also SegmentPreIndexStatsCollectorImpl has added support for NoDictColumnStatisticsCollector which needs to be added here too - https://github.com/apache/pinot/blob/master/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/SegmentPreIndexStatsCollectorImpl.java#L61
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.stats;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Stats container that efficiently collects statistics from columnar data using ColumnReader instances.
+ *
+ * <p>This implementation collects statistics by iterating column-wise instead of row-wise,
+ * which is more efficient for columnar data sources. It supports:
+ * <ul>
+ * <li>Column-wise statistics collection</li>
+ * <li>Existing columns from source data</li>
+ * <li>New columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ *
+ * <p>The statistics are collected using the same underlying collectors as the row-based approach
+ * (SegmentPreIndexStatsCollectorImpl) but with more efficient column-wise iteration.
+ */
+public class ColumnarSegmentPreIndexStatsContainer implements SegmentPreIndexStatsCollector {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentPreIndexStatsContainer.class);
+
+ private final Map<String, ColumnReader> _columnReaders;
+ private final StatsCollectorConfig _statsCollectorConfig;
+ private final Schema _targetSchema;
+ private final Map<String, AbstractColumnStatisticsCollector> _columnStatsCollectorMap;
+ private int _totalDocCount;
+
+ /**
+ * Create a ColumnarSegmentPreIndexStatsContainer.
+ *
+ * @param columnReaders Map of column name to ColumnReader instances
+ * @param statsCollectorConfig Configuration for statistics collection
+ */
+ public ColumnarSegmentPreIndexStatsContainer(Map<String, ColumnReader> columnReaders,
+ StatsCollectorConfig statsCollectorConfig) {
+ _columnReaders = columnReaders;
+ _statsCollectorConfig = statsCollectorConfig;
+ _targetSchema = statsCollectorConfig.getSchema();
+ _columnStatsCollectorMap = new HashMap<>();
+ _totalDocCount = -1; // indicates unset
+
+ initializeStatsCollectors();
+ collectColumnStats();
+ }
+
+ /**
+ * Initialize stats collectors for all columns in the target schema.
+ */
+ private void initializeStatsCollectors() {
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
+ switch (fieldSpec.getDataType().getStoredType()) {
+ case INT:
+ _columnStatsCollectorMap.put(columnName,
+ new IntColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case LONG:
+ _columnStatsCollectorMap.put(columnName,
+ new LongColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case FLOAT:
+ _columnStatsCollectorMap.put(columnName,
+ new FloatColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case DOUBLE:
+ _columnStatsCollectorMap.put(columnName,
+ new DoubleColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BIG_DECIMAL:
+ _columnStatsCollectorMap.put(columnName,
+ new BigDecimalColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case STRING:
+ _columnStatsCollectorMap.put(columnName,
+ new StringColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BYTES:
+ _columnStatsCollectorMap.put(columnName,
+ new BytesColumnPredIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case MAP:
+ _columnStatsCollectorMap.put(columnName,
+ new MapColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ default:
+ throw new IllegalStateException("Unsupported data type: " + fieldSpec.getDataType());
+ }
+ }
+ }
+
+ /**
+ * Collect stats by iterating column-wise using the provided ColumnReader instances.
+ */
+ private void collectColumnStats() {
+ LOGGER.info("Collecting stats for {} columns using column-wise iteration", _columnReaders.size());
+
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
+ AbstractColumnStatisticsCollector statsCollector = _columnStatsCollectorMap.get(columnName);
+ ColumnReader columnReader = _columnReaders.get(columnName);
+
+ if (columnReader == null) {
+ throw new RuntimeException("Column reader for column " + columnName + " not found");
+ }
+
+ LOGGER.debug("Collecting stats for column: {}", columnName);
+ collectStatsFromColumnReader(columnName, columnReader, statsCollector);
+
+ // Seal the stats collector
+ statsCollector.seal();
+ }
+ }
+
+ /**
+ * Collect stats from a column reader by iterating over all values using the iterator pattern.
+ */
+ private void collectStatsFromColumnReader(String columnName, ColumnReader columnReader,
+ AbstractColumnStatisticsCollector statsCollector) {
+ try {
+ // Reset the column reader to start from the beginning
+ columnReader.rewind();
+
+ int docCount = 0;
+ while (columnReader.hasNext()) {
+ Object value = columnReader.next();
+ statsCollector.collect(value);
Review Comment:
need to handle exceptions and continueOnError in case of an exception here.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -396,27 +372,58 @@ public void indexColumn(String columnName, @Nullable int[] sortedDocIds, IndexSe
if (sortedDocIds != null) {
int onDiskDocId = 0;
for (int docId : sortedDocIds) {
- // If validDodIds are provided, only index column if it's a valid doc
- if (validDocIds == null || validDocIds.contains(docId)) {
- indexColumnValue(colReader, creatorsByIndex, columnName, fieldSpec, dictionaryCreator, docId, onDiskDocId,
- nullVec);
- onDiskDocId++;
- }
+ indexColumnValue(colReader, creatorsByIndex, columnName, fieldSpec, dictionaryCreator, docId, onDiskDocId,
+ nullVec);
+ onDiskDocId++;
}
} else {
- int onDiskDocId = 0;
for (int docId = 0; docId < numDocs; docId++) {
- // If validDodIds are provided, only index column if it's a valid doc
- if (validDocIds == null || validDocIds.contains(docId)) {
- indexColumnValue(colReader, creatorsByIndex, columnName, fieldSpec, dictionaryCreator, docId, onDiskDocId,
- nullVec);
- onDiskDocId++;
- }
+ indexColumnValue(colReader, creatorsByIndex, columnName, fieldSpec, dictionaryCreator, docId, docId, nullVec);
}
}
}
}
+ /**
+ * Index a column using a ColumnReader (column-major approach).
+ * This method processes the column values using the iterator pattern from ColumnReader.
+ *
+ * @param columnName Name of the column to index
+ * @param columnReader ColumnReader for the column data
+ * @throws IOException if indexing fails
+ */
+ @Override
+ public void indexColumn(String columnName, ColumnReader columnReader) throws IOException {
+ Map<IndexType<?, ?, ?>, IndexCreator> creatorsByIndex = _creatorsByColAndIndex.get(columnName);
+ NullValueVectorCreator nullVec = _nullValueVectorCreatorMap.get(columnName);
+ FieldSpec fieldSpec = _schema.getFieldSpecFor(columnName);
+ SegmentDictionaryCreator dictionaryCreator = _dictionaryCreatorMap.get(columnName);
+ Object defaultNullValue = fieldSpec.getDefaultNullValue();
+ Object reuseColumnValueToIndex;
+
+ // Reset column reader to start from beginning
+ columnReader.rewind();
+
+ int docId = 0;
+ while (columnReader.hasNext()) {
+ reuseColumnValueToIndex = columnReader.next();
+
+ // Handle null values
+ if (nullVec != null && reuseColumnValueToIndex == null) {
+ nullVec.setNull(docId);
+ reuseColumnValueToIndex = defaultNullValue;
+ }
+
+ if (fieldSpec.isSingleValueField()) {
+ indexSingleValueRow(dictionaryCreator, reuseColumnValueToIndex, creatorsByIndex);
+ } else {
+ indexMultiValueRow(dictionaryCreator, (Object[]) reuseColumnValueToIndex, creatorsByIndex);
+ }
Review Comment:
catch JsonParseException and throw ColumnJsonParserException(columnName, jpe) similar to indexRow ?
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java:
##########
@@ -677,4 +706,63 @@ public int getSkippedRowsFound() {
public int getSanitizedRowsFound() {
return _sanitizedRowsFound;
}
+
+ /**
+ * Build segment using columnar approach.
+ * This method builds the segment by processing data column-wise instead of row-wise.
+ *
+ * @throws Exception if segment building fails
+ */
+ public void buildColumnar() throws Exception {
Review Comment:
we need to handle continueOnError cases here too.
and in case of column readers, the added complexity is that we shouldn't index any column of a row if any single column results in an exception.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/ColumnarSegmentCreationDataSource.java:
##########
@@ -0,0 +1,96 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator;
+
+import java.util.Map;
+import org.apache.pinot.segment.local.segment.creator.impl.stats.ColumnarSegmentPreIndexStatsContainer;
+import org.apache.pinot.segment.spi.creator.SegmentCreationDataSource;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.ColumnReaderFactory;
+import org.apache.pinot.spi.data.readers.RecordReader;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * SegmentCreationDataSource implementation that uses ColumnReaderFactory for columnar data access.
+ *
+ * <p>This data source enables columnar segment building by providing access to column readers
+ * instead of row-based record readers. It supports:
+ * <ul>
+ * <li>Columnar statistics collection</li>
+ * <li>Column-wise data access through ColumnReaderFactory</li>
+ * <li>Efficient handling of new columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ */
+public class ColumnarSegmentCreationDataSource implements SegmentCreationDataSource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentCreationDataSource.class);
+
+ private final ColumnReaderFactory _columnReaderFactory;
Review Comment:
I think we can remove _columnReaderFactory from this class
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.stats;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Stats container that efficiently collects statistics from columnar data using ColumnReader instances.
+ *
+ * <p>This implementation collects statistics by iterating column-wise instead of row-wise,
+ * which is more efficient for columnar data sources. It supports:
+ * <ul>
+ * <li>Column-wise statistics collection</li>
+ * <li>Existing columns from source data</li>
+ * <li>New columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ *
+ * <p>The statistics are collected using the same underlying collectors as the row-based approach
+ * (SegmentPreIndexStatsCollectorImpl) but with more efficient column-wise iteration.
+ */
+public class ColumnarSegmentPreIndexStatsContainer implements SegmentPreIndexStatsCollector {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentPreIndexStatsContainer.class);
+
+ private final Map<String, ColumnReader> _columnReaders;
+ private final StatsCollectorConfig _statsCollectorConfig;
+ private final Schema _targetSchema;
+ private final Map<String, AbstractColumnStatisticsCollector> _columnStatsCollectorMap;
+ private int _totalDocCount;
+
+ /**
+ * Create a ColumnarSegmentPreIndexStatsContainer.
+ *
+ * @param columnReaders Map of column name to ColumnReader instances
+ * @param statsCollectorConfig Configuration for statistics collection
+ */
+ public ColumnarSegmentPreIndexStatsContainer(Map<String, ColumnReader> columnReaders,
+ StatsCollectorConfig statsCollectorConfig) {
+ _columnReaders = columnReaders;
+ _statsCollectorConfig = statsCollectorConfig;
+ _targetSchema = statsCollectorConfig.getSchema();
+ _columnStatsCollectorMap = new HashMap<>();
+ _totalDocCount = -1; // indicates unset
+
+ initializeStatsCollectors();
+ collectColumnStats();
+ }
+
+ /**
+ * Initialize stats collectors for all columns in the target schema.
+ */
+ private void initializeStatsCollectors() {
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
+ switch (fieldSpec.getDataType().getStoredType()) {
+ case INT:
+ _columnStatsCollectorMap.put(columnName,
+ new IntColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case LONG:
+ _columnStatsCollectorMap.put(columnName,
+ new LongColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case FLOAT:
+ _columnStatsCollectorMap.put(columnName,
+ new FloatColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case DOUBLE:
+ _columnStatsCollectorMap.put(columnName,
+ new DoubleColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BIG_DECIMAL:
+ _columnStatsCollectorMap.put(columnName,
+ new BigDecimalColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case STRING:
+ _columnStatsCollectorMap.put(columnName,
+ new StringColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BYTES:
+ _columnStatsCollectorMap.put(columnName,
+ new BytesColumnPredIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case MAP:
+ _columnStatsCollectorMap.put(columnName,
+ new MapColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ default:
+ throw new IllegalStateException("Unsupported data type: " + fieldSpec.getDataType());
+ }
+ }
+ }
+
+ /**
+ * Collect stats by iterating column-wise using the provided ColumnReader instances.
+ */
+ private void collectColumnStats() {
+ LOGGER.info("Collecting stats for {} columns using column-wise iteration", _columnReaders.size());
+
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
+ AbstractColumnStatisticsCollector statsCollector = _columnStatsCollectorMap.get(columnName);
+ ColumnReader columnReader = _columnReaders.get(columnName);
+
+ if (columnReader == null) {
+ throw new RuntimeException("Column reader for column " + columnName + " not found");
+ }
+
+ LOGGER.debug("Collecting stats for column: {}", columnName);
+ collectStatsFromColumnReader(columnName, columnReader, statsCollector);
+
+ // Seal the stats collector
+ statsCollector.seal();
+ }
+ }
+
+ /**
+ * Collect stats from a column reader by iterating over all values using the iterator pattern.
+ */
+ private void collectStatsFromColumnReader(String columnName, ColumnReader columnReader,
+ AbstractColumnStatisticsCollector statsCollector) {
+ try {
+ // Reset the column reader to start from the beginning
+ columnReader.rewind();
+
+ int docCount = 0;
+ while (columnReader.hasNext()) {
+ Object value = columnReader.next();
+ statsCollector.collect(value);
+ docCount++;
+ }
+
+ // if totalDocCount is unset then set total doc count from the first column
+ if (_totalDocCount == -1) {
+ _totalDocCount = docCount;
+ } else if (_totalDocCount != docCount) {
+ // all columns should have same count
+ LOGGER.warn("Column {} has {} documents, but expected {} documents",
Review Comment:
I guess this is probably not fine and we should throw an exception in such a case. otherwise how will we map the data to the right row
in case it is fine, we should update _totalDocCount to max of _totalDocCount and docCount ?
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -396,27 +372,58 @@ public void indexColumn(String columnName, @Nullable int[] sortedDocIds, IndexSe
if (sortedDocIds != null) {
int onDiskDocId = 0;
for (int docId : sortedDocIds) {
- // If validDodIds are provided, only index column if it's a valid doc
- if (validDocIds == null || validDocIds.contains(docId)) {
- indexColumnValue(colReader, creatorsByIndex, columnName, fieldSpec, dictionaryCreator, docId, onDiskDocId,
- nullVec);
- onDiskDocId++;
- }
+ indexColumnValue(colReader, creatorsByIndex, columnName, fieldSpec, dictionaryCreator, docId, onDiskDocId,
+ nullVec);
+ onDiskDocId++;
}
} else {
- int onDiskDocId = 0;
for (int docId = 0; docId < numDocs; docId++) {
- // If validDodIds are provided, only index column if it's a valid doc
- if (validDocIds == null || validDocIds.contains(docId)) {
- indexColumnValue(colReader, creatorsByIndex, columnName, fieldSpec, dictionaryCreator, docId, onDiskDocId,
- nullVec);
- onDiskDocId++;
- }
+ indexColumnValue(colReader, creatorsByIndex, columnName, fieldSpec, dictionaryCreator, docId, docId, nullVec);
}
}
}
}
+ /**
+ * Index a column using a ColumnReader (column-major approach).
+ * This method processes the column values using the iterator pattern from ColumnReader.
+ *
+ * @param columnName Name of the column to index
+ * @param columnReader ColumnReader for the column data
+ * @throws IOException if indexing fails
+ */
+ @Override
+ public void indexColumn(String columnName, ColumnReader columnReader) throws IOException {
+ Map<IndexType<?, ?, ?>, IndexCreator> creatorsByIndex = _creatorsByColAndIndex.get(columnName);
+ NullValueVectorCreator nullVec = _nullValueVectorCreatorMap.get(columnName);
+ FieldSpec fieldSpec = _schema.getFieldSpecFor(columnName);
+ SegmentDictionaryCreator dictionaryCreator = _dictionaryCreatorMap.get(columnName);
+ Object defaultNullValue = fieldSpec.getDefaultNullValue();
+ Object reuseColumnValueToIndex;
+
+ // Reset column reader to start from beginning
+ columnReader.rewind();
+
+ int docId = 0;
+ while (columnReader.hasNext()) {
+ reuseColumnValueToIndex = columnReader.next();
+
+ // Handle null values
+ if (nullVec != null && reuseColumnValueToIndex == null) {
+ nullVec.setNull(docId);
+ reuseColumnValueToIndex = defaultNullValue;
Review Comment:
what happens if the field doesn't support null (I guess defaultNullValue is null or nullVec is null) ? we should error out here ? and handle it via continueOnError flag where indexColumn() is called ?
For reference - indexRow() throws an exception if row.getValue(columnName) is null
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java:
##########
@@ -209,7 +249,7 @@ public void init(SegmentGeneratorConfig config, SegmentCreationDataSource dataSo
}
_ingestionSchemaValidator =
- SchemaValidatorFactory.getSchemaValidator(_dataSchema, _recordReader.getClass().getName(),
+ SchemaValidatorFactory.getSchemaValidator(_dataSchema, readerClassName,
Review Comment:
we're relying on getSchemaValidator to support null readerClassName which isn't ideal.
can we think of a more appropriate way to handle this ?
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java:
##########
@@ -677,4 +706,63 @@ public int getSkippedRowsFound() {
public int getSanitizedRowsFound() {
return _sanitizedRowsFound;
}
+
+ /**
+ * Build segment using columnar approach.
+ * This method builds the segment by processing data column-wise instead of row-wise.
+ *
+ * @throws Exception if segment building fails
+ */
+ public void buildColumnar() throws Exception {
+ if (!(_dataSource instanceof ColumnarSegmentCreationDataSource)) {
+ throw new IllegalStateException("buildColumnar() can only be called after initColumnar()");
+ }
+
+ ColumnarSegmentCreationDataSource columnarDataSource = (ColumnarSegmentCreationDataSource) _dataSource;
+ Map<String, ColumnReader> columnReaders = columnarDataSource.getColumnReaders();
+
+ LOGGER.info("Starting columnar segment building for {} columns", columnReaders.size());
+
+ // Reuse existing stats collection and index creation info logic
+ LOGGER.debug("Start building StatsCollector!");
+ collectStatsAndIndexCreationInfo();
+ LOGGER.info("Finished building StatsCollector!");
+ LOGGER.info("Collected stats for {} documents", _totalDocs);
+
+ if (_totalDocs == 0) {
+ LOGGER.warn("No documents found in data source");
+ handlePostCreation();
+ return;
+ }
+
+ try {
+ // Initialize the index creation using the per-column statistics information
+ _indexCreator.init(_config, _segmentIndexCreationInfo, _indexCreationInfoMap, _dataSchema, _tempIndexDir, null);
+
+ // Build the indexes column-wise (true column-major approach)
+ LOGGER.info("Start building Index using columnar approach");
+ long indexStartTime = System.nanoTime();
+
+ TreeSet<String> columns = _dataSchema.getPhysicalColumnNames();
+ for (String columnName : columns) {
+ LOGGER.debug("Indexing column: {}", columnName);
+ ColumnReader columnReader = columnReaders.get(columnName);
+ if (columnReader == null) {
+ throw new IllegalStateException("No column reader found for column: " + columnName);
+ }
+
+ // Index each column independently using true column-major approach
+ // This is similar to how buildByColumn works but uses ColumnReader instead of IndexSegment
+ ((SegmentColumnarIndexCreator) _indexCreator).indexColumn(columnName, columnReader);
+ }
+
+ _totalIndexTimeNs = System.nanoTime() - indexStartTime;
+ } catch (Exception e) {
+ _indexCreator.close();
+ throw e;
+ }
Review Comment:
we need to close all column readers
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReaderFactory.java:
##########
@@ -0,0 +1,154 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.readers;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.ColumnReaderFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * ColumnReaderFactory implementation for immutable Pinot segments.
+ *
+ * <p>This factory creates ColumnReader instances for reading data from Pinot segments
+ * in a columnar fashion. It handles:
+ * <ul>
+ * <li>Creating readers for existing columns in the segment</li>
+ * <li>Creating default value readers for new columns</li>
+ * <li>Resource management for all created readers</li>
+ * </ul>
+ */
+public class PinotSegmentColumnReaderFactory implements ColumnReaderFactory {
Review Comment:
I think this class / interface needs to be refactored a bit
1. init() shouldn't take targetSchema as that can be passed in constructor itself.
2. getAllColumnReaders() calls createColumnReader() and both are public which isn't ideal. Rather init() should call createColumnReader() and createColumnReader() should be private. and getAllColumnReaders() should just get existing objects without creating new objects
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on PR #16727:
URL: https://github.com/apache/pinot/pull/16727#issuecomment-3412127864
@noob-se7en, @KKcorps can you take a look again, all the review comments are addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2434781937
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java:
##########
@@ -171,31 +170,72 @@ public void init(SegmentGeneratorConfig config, RecordReader recordReader)
new TransformPipeline(config.getTableConfig(), config.getSchema()));
}
+ /**
+ * Initialize the driver for columnar segment building using a ColumnReaderFactory.
+ * This method sets up the driver to use column-wise input data access instead of row-wise.
+ *
+ * @param config Segment generator configuration
+ * @param columnReaderFactory Factory for creating column readers
+ * @throws Exception if initialization fails
+ */
+ public void init(SegmentGeneratorConfig config, ColumnReaderFactory columnReaderFactory)
+ throws Exception {
+ // Initialize the column reader factory with target schema
+ columnReaderFactory.init(config.getSchema());
+
+ // Create all column readers for the target schema
+ Map<String, ColumnReader> columnReaders = columnReaderFactory.getAllColumnReaders();
+
+ // Create columnar data source
+ ColumnarSegmentCreationDataSource columnarDataSource =
+ new ColumnarSegmentCreationDataSource(columnReaderFactory, columnReaders);
+
+ // Use the existing init method with columnar data source and no transform pipeline
+ init(config, columnarDataSource, null);
Review Comment:
It is added in the PR description, will refine the code documentations if this is not clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on PR #16727:
URL: https://github.com/apache/pinot/pull/16727#issuecomment-3257600102
@Jackie-Jiang, could you review or assign somebody to review this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "noob-se7en (via GitHub)" <gi...@apache.org>.
noob-se7en commented on PR #16727:
URL: https://github.com/apache/pinot/pull/16727#issuecomment-3484145233
@krishan1390 pls approve it.
And @swaminathanmanish pls help merging this PR after that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2430155182
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java:
##########
@@ -677,4 +706,63 @@ public int getSkippedRowsFound() {
public int getSanitizedRowsFound() {
return _sanitizedRowsFound;
}
+
+ /**
+ * Build segment using columnar approach.
+ * This method builds the segment by processing data column-wise instead of row-wise.
+ *
+ * @throws Exception if segment building fails
+ */
+ public void buildColumnar() throws Exception {
+ if (!(_dataSource instanceof ColumnarSegmentCreationDataSource)) {
+ throw new IllegalStateException("buildColumnar() can only be called after initColumnar()");
+ }
+
+ ColumnarSegmentCreationDataSource columnarDataSource = (ColumnarSegmentCreationDataSource) _dataSource;
+ Map<String, ColumnReader> columnReaders = columnarDataSource.getColumnReaders();
+
+ LOGGER.info("Starting columnar segment building for {} columns", columnReaders.size());
+
+ // Reuse existing stats collection and index creation info logic
+ LOGGER.debug("Start building StatsCollector!");
+ collectStatsAndIndexCreationInfo();
+ LOGGER.info("Finished building StatsCollector!");
+ LOGGER.info("Collected stats for {} documents", _totalDocs);
+
+ if (_totalDocs == 0) {
+ LOGGER.warn("No documents found in data source");
+ handlePostCreation();
+ return;
+ }
+
+ try {
+ // Initialize the index creation using the per-column statistics information
+ _indexCreator.init(_config, _segmentIndexCreationInfo, _indexCreationInfoMap, _dataSchema, _tempIndexDir, null);
Review Comment:
This is needed for mutable to immutable segment conversion. We are not making any change for this flow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2430160637
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/creator/impl/ColumnarSegmentBuildingTest.java:
##########
@@ -0,0 +1,500 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl;
+
+import com.google.common.collect.Lists;
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.segment.readers.PinotSegmentColumnReaderFactory;
+import org.apache.pinot.segment.spi.ImmutableSegment;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.data.readers.RecordReader;
+import org.apache.pinot.spi.data.readers.RecordReaderConfig;
+import org.apache.pinot.spi.utils.ReadMode;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+
+/**
+ * Tests for columnar segment building functionality.
+ *
+ * <p>This test class validates the new columnar segment building capability including:
+ * <ul>
+ * <li>ColumnReader interface implementations</li>
+ * <li>ColumnReaderFactory for different data sources</li>
+ * <li>Columnar segment building vs row-major building equivalence</li>
+ * <li>Support for new columns with default values</li>
+ * </ul>
+ */
+public class ColumnarSegmentBuildingTest {
+ private static final String TEMP_DIR = System.getProperty("java.io.tmpdir");
+ private static final String TABLE_NAME = "testTable";
+ private static final String SEGMENT_NAME = "testSegment";
+
+ // Test columns
+ private static final String STRING_COL_1 = "stringCol1";
+ private static final String STRING_COL_2 = "stringCol2";
+ private static final String INT_COL_1 = "intCol1";
+ private static final String INT_COL_2 = "intCol2";
+ private static final String LONG_COL = "longCol";
+ private static final String FLOAT_COL = "floatCol";
+ private static final String DOUBLE_COL = "doubleCol";
+ private static final String BIG_DECIMAL_COL = "bigDecimalCol";
+ private static final String BYTES_COL = "bytesCol";
+ private static final String TIME_COL = "timeCol";
+ private static final String MV_INT_COL = "mvIntCol";
+ private static final String MV_STRING_COL = "mvStringCol";
+
+ // New column for testing default value handling
+ private static final String NEW_STRING_COL = "newStringCol";
+ private static final String NEW_INT_COL = "newIntCol";
+
+ private File _tempDir;
+ private Schema _originalSchema;
+ private Schema _extendedSchema; // Schema with additional columns
+ private TableConfig _tableConfig;
+ private List<GenericRow> _testData;
+
+ @BeforeClass
+ public void setUp() throws IOException {
+ _tempDir = new File(TEMP_DIR, "ColumnarSegmentBuildingTest");
+ FileUtils.deleteQuietly(_tempDir);
+ _tempDir.mkdirs();
+
+ // Create original schema
+ _originalSchema = new Schema.SchemaBuilder()
+ .addSingleValueDimension(STRING_COL_1, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(STRING_COL_2, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(INT_COL_1, FieldSpec.DataType.INT)
+ .addSingleValueDimension(INT_COL_2, FieldSpec.DataType.INT)
+ .addSingleValueDimension(LONG_COL, FieldSpec.DataType.LONG)
+ .addSingleValueDimension(FLOAT_COL, FieldSpec.DataType.FLOAT)
+ .addSingleValueDimension(DOUBLE_COL, FieldSpec.DataType.DOUBLE)
+ .addSingleValueDimension(BIG_DECIMAL_COL, FieldSpec.DataType.BIG_DECIMAL)
+ .addSingleValueDimension(BYTES_COL, FieldSpec.DataType.BYTES)
+ .addMultiValueDimension(MV_INT_COL, FieldSpec.DataType.INT)
+ .addMultiValueDimension(MV_STRING_COL, FieldSpec.DataType.STRING)
+ .addDateTime(TIME_COL, FieldSpec.DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS")
+ .build();
+
+ // Create extended schema with additional columns
+ _extendedSchema = new Schema.SchemaBuilder()
+ .addSingleValueDimension(STRING_COL_1, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(STRING_COL_2, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(INT_COL_1, FieldSpec.DataType.INT)
+ .addSingleValueDimension(INT_COL_2, FieldSpec.DataType.INT)
+ .addSingleValueDimension(LONG_COL, FieldSpec.DataType.LONG)
+ .addSingleValueDimension(FLOAT_COL, FieldSpec.DataType.FLOAT)
+ .addSingleValueDimension(DOUBLE_COL, FieldSpec.DataType.DOUBLE)
+ .addSingleValueDimension(BIG_DECIMAL_COL, FieldSpec.DataType.BIG_DECIMAL)
+ .addSingleValueDimension(BYTES_COL, FieldSpec.DataType.BYTES)
+ .addMultiValueDimension(MV_INT_COL, FieldSpec.DataType.INT)
+ .addMultiValueDimension(MV_STRING_COL, FieldSpec.DataType.STRING)
+ .addDateTime(TIME_COL, FieldSpec.DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS")
+ .addSingleValueDimension(NEW_STRING_COL, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(NEW_INT_COL, FieldSpec.DataType.INT)
+ .build();
+
+ // Create table config
+ _tableConfig = new TableConfigBuilder(TableType.OFFLINE)
+ .setTableName(TABLE_NAME)
+ .setTimeColumnName(TIME_COL)
+ .setInvertedIndexColumns(Lists.newArrayList(STRING_COL_1, INT_COL_1))
+ .setSortedColumn(INT_COL_1)
+ .build();
+
+ // Generate test data
+ _testData = generateTestData(100);
+ }
+
+ @AfterClass
+ public void tearDown() throws IOException {
+ FileUtils.deleteQuietly(_tempDir);
+ }
+
+ @Test
+ public void testBasicColumnarBuilding() throws Exception {
+ // First create a segment using traditional row-major approach
+ File rowMajorSegmentDir = createRowMajorSegment();
+
+ // Then create a segment using columnar approach from the row-major segment
+ File columnarSegmentDir = createColumnarSegment(rowMajorSegmentDir);
+
+ // Validate that both segments have identical data
+ validateSegmentsIdentical(rowMajorSegmentDir, columnarSegmentDir);
+ }
+
+ @Test
+ public void testColumnarBuildingWithNewColumns() throws Exception {
+ // Create original segment with original schema
+ File originalSegmentDir = createRowMajorSegment();
+
+ // Create new segment with extended schema (has additional columns)
+ File newSegmentDir = createColumnarSegmentWithNewColumns(originalSegmentDir);
+
+ // Validate that the new segment has the additional columns with default values
+ validateSegmentWithNewColumns(newSegmentDir);
+ }
+
+ @Test
+ public void testColumnReaderFactory() throws Exception {
+ // Create a segment to test column reader factory with
+ File segmentDir = createRowMajorSegment();
+ ImmutableSegment segment = ImmutableSegmentLoader.load(segmentDir, ReadMode.mmap);
+
+ try {
+ // Test PinotSegmentColumnReaderFactory
+ try (PinotSegmentColumnReaderFactory factory = new PinotSegmentColumnReaderFactory(segment)) {
+ factory.init(_originalSchema);
+
+ // Test that all expected columns are available
+ Set<String> availableColumns = factory.getAvailableColumns();
+ Assert.assertTrue(availableColumns.contains(STRING_COL_1));
+ Assert.assertTrue(availableColumns.contains(INT_COL_1));
+ Assert.assertTrue(availableColumns.contains(MV_INT_COL));
+
+ // Test creating individual column readers
+ ColumnReader stringReader = factory.createColumnReader(STRING_COL_1,
+ _originalSchema.getFieldSpecFor(STRING_COL_1));
+ Assert.assertEquals(stringReader.getColumnName(), STRING_COL_1);
+
+ ColumnReader mvReader = factory.createColumnReader(MV_INT_COL,
+ _originalSchema.getFieldSpecFor(MV_INT_COL));
+ Assert.assertEquals(mvReader.getColumnName(), MV_INT_COL);
+
+ // Test reading values using iterator pattern
+ Assert.assertTrue(stringReader.hasNext());
+ Object firstStringValue = stringReader.next();
+ Assert.assertEquals(firstStringValue, "string1_0");
+
+ // Test that we can continue reading
+ Assert.assertTrue(stringReader.hasNext());
+ Object secondStringValue = stringReader.next();
+ Assert.assertEquals(secondStringValue, "string1_1");
+
+ // Reset and test again
+ stringReader.rewind();
+ Assert.assertTrue(stringReader.hasNext());
+ firstStringValue = stringReader.next();
+ Assert.assertEquals(firstStringValue, "string1_0");
+
+ // Test creating all column readers
+ Map<String, ColumnReader> allReaders = factory.getAllColumnReaders();
+ Assert.assertEquals(allReaders.size(), _originalSchema.getPhysicalColumnNames().size());
+ }
+ } finally {
+ segment.destroy();
+ }
+ }
+
+ @Test
+ public void testColumnReaderWithNewColumns() throws Exception {
+ // Create a segment to test with
+ File segmentDir = createRowMajorSegment();
+ ImmutableSegment segment = ImmutableSegmentLoader.load(segmentDir, ReadMode.mmap);
+
+ try {
+ // Test creating readers for new columns (should return default value readers)
+ try (PinotSegmentColumnReaderFactory factory = new PinotSegmentColumnReaderFactory(segment)) {
+ factory.init(_extendedSchema);
+
+ // Test creating reader for new column
+ ColumnReader newStringReader = factory.createColumnReader(NEW_STRING_COL,
+ _extendedSchema.getFieldSpecFor(NEW_STRING_COL));
+ Assert.assertEquals(newStringReader.getColumnName(), NEW_STRING_COL);
+
+ // Verify it returns default values using iterator pattern
+ Assert.assertTrue(newStringReader.hasNext());
+ Object defaultValue = newStringReader.next();
+ Assert.assertEquals(defaultValue, _extendedSchema.getFieldSpecFor(NEW_STRING_COL).getDefaultNullValue());
+
+ // Test that all values are the same (default)
+ int valueCount = 1; // We already read one value
+ while (newStringReader.hasNext()) {
+ Object value = newStringReader.next();
+ Assert.assertEquals(value, defaultValue);
+ valueCount++;
+ }
+ Assert.assertEquals(valueCount, _testData.size());
+ }
+ } finally {
+ segment.destroy();
+ }
+ }
+
+ @Test
+ public void testAllDataTypes() throws Exception {
+ // This test validates that all supported data types work correctly with columnar building
+ File rowMajorSegmentDir = createRowMajorSegment();
+ File columnarSegmentDir = createColumnarSegment(rowMajorSegmentDir);
+
+ // Validate that both segments have identical data for all data types
+ validateSegmentsIdentical(rowMajorSegmentDir, columnarSegmentDir);
+
+ // Additionally validate that all expected columns and data types are present
+ ImmutableSegment segment = ImmutableSegmentLoader.load(columnarSegmentDir, ReadMode.mmap);
+ try {
+ Set<String> columnNames = segment.getPhysicalColumnNames();
+
+ // Validate all data types are present
+ Assert.assertTrue(columnNames.contains(STRING_COL_1), "STRING column missing");
+ Assert.assertTrue(columnNames.contains(INT_COL_1), "INT column missing");
+ Assert.assertTrue(columnNames.contains(LONG_COL), "LONG column missing");
+ Assert.assertTrue(columnNames.contains(FLOAT_COL), "FLOAT column missing");
+ Assert.assertTrue(columnNames.contains(DOUBLE_COL), "DOUBLE column missing");
+ Assert.assertTrue(columnNames.contains(BIG_DECIMAL_COL), "BIG_DECIMAL column missing");
+ Assert.assertTrue(columnNames.contains(BYTES_COL), "BYTES column missing");
+ Assert.assertTrue(columnNames.contains(MV_INT_COL), "Multi-value INT column missing");
+ Assert.assertTrue(columnNames.contains(MV_STRING_COL), "Multi-value STRING column missing");
+ Assert.assertTrue(columnNames.contains(TIME_COL), "TIME column missing");
+ } finally {
+ segment.destroy();
+ }
+ }
+
+
+
+ private File createRowMajorSegment() throws Exception {
+ File outputDir = new File(_tempDir, "rowMajorSegment");
+ FileUtils.deleteQuietly(outputDir);
+ outputDir.mkdirs();
+
+ SegmentGeneratorConfig config = new SegmentGeneratorConfig(_tableConfig, _originalSchema);
+ config.setOutDir(outputDir.getAbsolutePath());
+ config.setSegmentName(SEGMENT_NAME + "_rowMajor");
+
+ SegmentIndexCreationDriverImpl driver = new SegmentIndexCreationDriverImpl();
+ TestRecordReader recordReader = new TestRecordReader(_testData);
+ driver.init(config, recordReader);
+ driver.build();
+
+ return new File(outputDir, SEGMENT_NAME + "_rowMajor");
+ }
+
+ private File createColumnarSegment(File sourceSegmentDir) throws Exception {
+ File outputDir = new File(_tempDir, "columnarSegment");
+ FileUtils.deleteQuietly(outputDir);
+ outputDir.mkdirs();
+
+ // Load the source segment
+ ImmutableSegment sourceSegment = ImmutableSegmentLoader.load(sourceSegmentDir, ReadMode.mmap);
+
+ try {
+ SegmentGeneratorConfig config = new SegmentGeneratorConfig(_tableConfig, _originalSchema);
+ config.setOutDir(outputDir.getAbsolutePath());
+ config.setSegmentName(SEGMENT_NAME + "_columnar");
+
+ SegmentIndexCreationDriverImpl driver = new SegmentIndexCreationDriverImpl();
+
+ // Use the new columnar building approach with config
+ try (PinotSegmentColumnReaderFactory factory = new PinotSegmentColumnReaderFactory(sourceSegment)) {
+ driver.init(config, factory);
+ driver.build();
+ }
+
+ return new File(outputDir, SEGMENT_NAME + "_columnar");
+ } finally {
+ sourceSegment.destroy();
+ }
+ }
+
+ private File createColumnarSegmentWithNewColumns(File sourceSegmentDir) throws Exception {
+ File outputDir = new File(_tempDir, "columnarSegmentWithNewColumns");
+ FileUtils.deleteQuietly(outputDir);
+ outputDir.mkdirs();
+
+ // Load the source segment
+ ImmutableSegment sourceSegment = ImmutableSegmentLoader.load(sourceSegmentDir, ReadMode.mmap);
+
+ try {
+ // Use extended schema with new columns
+ SegmentGeneratorConfig config = new SegmentGeneratorConfig(_tableConfig, _extendedSchema);
+ config.setOutDir(outputDir.getAbsolutePath());
+ config.setSegmentName(SEGMENT_NAME + "_withNewColumns");
+
+ SegmentIndexCreationDriverImpl driver = new SegmentIndexCreationDriverImpl();
+
+ // Use columnar building with extended schema
+ try (PinotSegmentColumnReaderFactory factory = new PinotSegmentColumnReaderFactory(sourceSegment)) {
+ driver.init(config, factory);
+ driver.build();
+ }
+
+ return new File(outputDir, SEGMENT_NAME + "_withNewColumns");
+ } finally {
+ sourceSegment.destroy();
+ }
+ }
+
+ private void validateSegmentsIdentical(File segment1Dir, File segment2Dir) throws Exception {
+ ImmutableSegment segment1 = ImmutableSegmentLoader.load(segment1Dir, ReadMode.mmap);
+ ImmutableSegment segment2 = ImmutableSegmentLoader.load(segment2Dir, ReadMode.mmap);
+
+ try {
+ // Validate metadata
+ Assert.assertEquals(segment1.getSegmentMetadata().getTotalDocs(), segment2.getSegmentMetadata().getTotalDocs());
+ Assert.assertEquals(segment1.getSegmentMetadata().getAllColumns(), segment2.getSegmentMetadata().getAllColumns());
+
+ // Validate data for each column
+ for (String columnName : segment1.getPhysicalColumnNames()) {
+ validateColumnData(segment1, segment2, columnName);
+ }
+ } finally {
+ segment1.destroy();
+ segment2.destroy();
+ }
+ }
+
+ private void validateSegmentWithNewColumns(File segmentDir) throws Exception {
+ ImmutableSegment segment = ImmutableSegmentLoader.load(segmentDir, ReadMode.mmap);
+
+ try {
+ // Validate that new columns exist
+ Assert.assertTrue(segment.getPhysicalColumnNames().contains(NEW_STRING_COL));
+ Assert.assertTrue(segment.getPhysicalColumnNames().contains(NEW_INT_COL));
+
+ // Validate that new columns have default values
+ GenericRow row = new GenericRow();
+ for (int docId = 0; docId < segment.getSegmentMetadata().getTotalDocs(); docId++) {
+ segment.getRecord(docId, row);
+
+ // Check default values
+ Assert.assertEquals(row.getValue(NEW_STRING_COL),
+ _extendedSchema.getFieldSpecFor(NEW_STRING_COL).getDefaultNullValue());
+ Assert.assertEquals(row.getValue(NEW_INT_COL),
+ _extendedSchema.getFieldSpecFor(NEW_INT_COL).getDefaultNullValue());
+ }
+ } finally {
+ segment.destroy();
+ }
+ }
+
+ private void validateColumnData(ImmutableSegment segment1, ImmutableSegment segment2, String columnName) {
+ int numDocs = segment1.getSegmentMetadata().getTotalDocs();
+
+ GenericRow row1 = new GenericRow();
+ GenericRow row2 = new GenericRow();
+
+ for (int docId = 0; docId < numDocs; docId++) {
+ segment1.getRecord(docId, row1);
+ segment2.getRecord(docId, row2);
+
+ Object value1 = row1.getValue(columnName);
+ Object value2 = row2.getValue(columnName);
+
+ Assert.assertEquals(value1, value2,
+ String.format("Column %s differs at docId %d: %s vs %s", columnName, docId, value1, value2));
+ }
+ }
+
+ private List<GenericRow> generateTestData(int numRows) {
+ List<GenericRow> data = new ArrayList<>();
+
+ for (int i = 0; i < numRows; i++) {
+ GenericRow row = new GenericRow();
+ row.putValue(STRING_COL_1, "string1_" + i);
+ row.putValue(STRING_COL_2, "string2_" + (i % 10));
+ row.putValue(INT_COL_1, i);
Review Comment:
sorting is not handled in columnar segment build process for immutable to immutable transformation. It is only done in realtime segment conversion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2468681542
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.stats;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Stats container that efficiently collects statistics from columnar data using ColumnReader instances.
+ *
+ * <p>This implementation collects statistics by iterating column-wise instead of row-wise,
+ * which is more efficient for columnar data sources. It supports:
+ * <ul>
+ * <li>Column-wise statistics collection</li>
+ * <li>Existing columns from source data</li>
+ * <li>New columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ *
+ * <p>The statistics are collected using the same underlying collectors as the row-based approach
+ * (SegmentPreIndexStatsCollectorImpl) but with more efficient column-wise iteration.
+ */
+public class ColumnarSegmentPreIndexStatsContainer implements SegmentPreIndexStatsCollector {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentPreIndexStatsContainer.class);
+
+ private final Map<String, ColumnReader> _columnReaders;
+ private final StatsCollectorConfig _statsCollectorConfig;
+ private final Schema _targetSchema;
+ private final Map<String, AbstractColumnStatisticsCollector> _columnStatsCollectorMap;
+ private int _totalDocCount;
+
+ /**
+ * Create a ColumnarSegmentPreIndexStatsContainer.
+ *
+ * @param columnReaders Map of column name to ColumnReader instances
+ * @param statsCollectorConfig Configuration for statistics collection
+ */
+ public ColumnarSegmentPreIndexStatsContainer(Map<String, ColumnReader> columnReaders,
+ StatsCollectorConfig statsCollectorConfig) {
+ _columnReaders = columnReaders;
+ _statsCollectorConfig = statsCollectorConfig;
+ _targetSchema = statsCollectorConfig.getSchema();
+ _columnStatsCollectorMap = new HashMap<>();
+ _totalDocCount = -1; // indicates unset
+
+ initializeStatsCollectors();
Review Comment:
updating, I don't see any benefit in delaying the initialization by calling it immediately after object creation. These are not service start up objects but more to be used in the segment generation flow where there is not much benefit by split contructor logic into init method and calling it immediately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2356052703
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReaderImpl.java:
##########
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.readers;
+
+import java.io.IOException;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.utils.PinotDataType;
+import org.apache.pinot.segment.local.recordtransformer.DataTypeTransformer;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Implementation of ColumnReader for Pinot segments.
+ *
+ * <p>This class wraps the existing PinotSegmentColumnReader and provides the ColumnReader interface
+ * for columnar segment building. It handles:
+ * <ul>
+ * <li>Reading column values from Pinot segments</li>
+ * <li>Data type conversions when target schema differs from source</li>
+ * <li>Null value detection</li>
+ * <li>Resource cleanup</li>
+ * </ul>
+ */
+public class PinotSegmentColumnReaderImpl implements ColumnReader {
+ private static final Logger LOGGER = LoggerFactory.getLogger(PinotSegmentColumnReaderImpl.class);
+
+ private final PinotSegmentColumnReader _segmentColumnReader;
+ private final String _columnName;
+ private final int _numDocs;
+ private final boolean _needsConversion;
+ private final FieldSpec _targetFieldSpec;
+
+ private int _currentIndex;
+ private Object _currentValue;
+ private boolean _currentIsNull;
+
+ /**
+ * Create a PinotSegmentColumnReaderImpl for an existing column in the segment.
+ *
+ * @param indexSegment Source segment to read from
+ * @param columnName Name of the column
+ * @param targetFieldSpec Target field specification (may differ from source for data type conversion)
+ */
+ public PinotSegmentColumnReaderImpl(IndexSegment indexSegment, String columnName, FieldSpec targetFieldSpec) {
+ _segmentColumnReader = new PinotSegmentColumnReader(indexSegment, columnName);
+ _columnName = columnName;
+ _numDocs = indexSegment.getSegmentMetadata().getTotalDocs();
+ _targetFieldSpec = targetFieldSpec;
+ _currentIndex = 0;
+
+ // Check if data type conversion is needed
+ FieldSpec sourceFieldSpec = indexSegment.getSegmentMetadata().getSchema().getFieldSpecFor(columnName);
+ _needsConversion = sourceFieldSpec != null
Review Comment:
This might not be an allowed change like data type change so it is safe to skip. The forward index files for sv and mv will need to be rewritten in Pinot for existing data while the data type change does not require in reload. Pinot still might allow this change to make with force schema update API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2356156792
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReader.java:
##########
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.spi.data.readers;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.Serializable;
+import javax.annotation.Nullable;
+
+
+/**
+ * The <code>ColumnReader</code> interface is used to read column data from various data sources
+ * for columnar segment building. Unlike RecordReader which reads row-by-row, ColumnReader provides
+ * column-wise access to data, enabling efficient columnar segment creation.
+ *
+ * <p>This interface follows an iterator pattern similar to RecordReader:
+ * <ul>
+ * <li>Sequential iteration over all values in a column using hasNext() and next()</li>
+ * <li>Null value detection for the current value</li>
+ * <li>Rewind capability to restart iteration</li>
+ * <li>Resource cleanup</li>
+ * </ul>
+ *
+ * <p>Implementations should handle data type conversions, default values for new columns,
+ * and efficient column-wise data access patterns.
+ */
+public interface ColumnReader extends Closeable, Serializable {
Review Comment:
This sounds good, I will have a follow up PR to apply this feature in minino refresh job. Will handle the enhancements in a separate PR for upsert merger tasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2356677290
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/recordtransformer/DataTypeTransformer.java:
##########
@@ -128,6 +89,62 @@ public void transform(GenericRow record) {
}
}
+ /**
+ * Convert a single value to the target PinotDataType.
+ * This method can be reused by other components that need single value conversion.
+ *
+ * @param value The value to convert
+ * @param dest The target PinotDataType
+ * @param columnName The column name (for error reporting)
+ * @return The converted value
+ */
+ public static Object convertValue(@Nullable Object value, PinotDataType dest, String columnName) {
Review Comment:
(minor) `value` is never `null` (caller ensures that). Let's remove the `@Nullable` and null check. The return value could actually be `null`, so let's annotate the return value as `@Nullable`. Also suggest renaming `dest` to `targetType`
```suggestion
@Nullable
public static Object convertValue(Object value, PinotDataType targetType, String columnName) {
```
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReaderImpl.java:
##########
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.readers;
+
+import java.io.IOException;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.utils.PinotDataType;
+import org.apache.pinot.segment.local.recordtransformer.DataTypeTransformer;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Implementation of ColumnReader for Pinot segments.
+ *
+ * <p>This class wraps the existing PinotSegmentColumnReader and provides the ColumnReader interface
+ * for columnar segment building. It handles:
+ * <ul>
+ * <li>Reading column values from Pinot segments</li>
+ * <li>Data type conversions when target schema differs from source</li>
+ * <li>Null value detection</li>
+ * <li>Resource cleanup</li>
+ * </ul>
+ */
+public class PinotSegmentColumnReaderImpl implements ColumnReader {
+ private static final Logger LOGGER = LoggerFactory.getLogger(PinotSegmentColumnReaderImpl.class);
+
+ private final PinotSegmentColumnReader _segmentColumnReader;
+ private final String _columnName;
+ private final int _numDocs;
+ private final boolean _needsConversion;
+ private final FieldSpec _targetFieldSpec;
+
+ private int _currentIndex;
+ private Object _currentValue;
+ private boolean _currentIsNull;
+
+ /**
+ * Create a PinotSegmentColumnReaderImpl for an existing column in the segment.
+ *
+ * @param indexSegment Source segment to read from
+ * @param columnName Name of the column
+ * @param targetFieldSpec Target field specification (may differ from source for data type conversion)
+ */
+ public PinotSegmentColumnReaderImpl(IndexSegment indexSegment, String columnName, FieldSpec targetFieldSpec) {
+ _segmentColumnReader = new PinotSegmentColumnReader(indexSegment, columnName);
+ _columnName = columnName;
+ _numDocs = indexSegment.getSegmentMetadata().getTotalDocs();
+ _targetFieldSpec = targetFieldSpec;
+ _currentIndex = 0;
+
+ // Check if data type conversion is needed
+ FieldSpec sourceFieldSpec = indexSegment.getSegmentMetadata().getSchema().getFieldSpecFor(columnName);
+ _needsConversion = sourceFieldSpec != null
Review Comment:
How about other `RecordTransformer`? Seems the current implementation can only support `DataTypeTransformer`
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReader.java:
##########
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.spi.data.readers;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.Serializable;
+import javax.annotation.Nullable;
+
+
+/**
+ * The <code>ColumnReader</code> interface is used to read column data from various data sources
+ * for columnar segment building. Unlike RecordReader which reads row-by-row, ColumnReader provides
+ * column-wise access to data, enabling efficient columnar segment creation.
+ *
+ * <p>This interface follows an iterator pattern similar to RecordReader:
+ * <ul>
+ * <li>Sequential iteration over all values in a column using hasNext() and next()</li>
+ * <li>Null value detection for the current value</li>
+ * <li>Rewind capability to restart iteration</li>
+ * <li>Resource cleanup</li>
+ * </ul>
+ *
+ * <p>Implementations should handle data type conversions, default values for new columns,
+ * and efficient column-wise data access patterns.
+ */
+public interface ColumnReader extends Closeable, Serializable {
Review Comment:
This is mostly useful when reading from a column major file format such as Parquet. Can you check if this interface can be used for Parquet reader?
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReader.java:
##########
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.spi.data.readers;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.Serializable;
+import javax.annotation.Nullable;
+
+
+/**
+ * The <code>ColumnReader</code> interface is used to read column data from various data sources
+ * for columnar segment building. Unlike RecordReader which reads row-by-row, ColumnReader provides
+ * column-wise access to data, enabling efficient columnar segment creation.
+ *
+ * <p>This interface follows an iterator pattern similar to RecordReader:
+ * <ul>
+ * <li>Sequential iteration over all values in a column using hasNext() and next()</li>
+ * <li>Null value detection for the current value</li>
+ * <li>Rewind capability to restart iteration</li>
+ * <li>Resource cleanup</li>
+ * </ul>
+ *
+ * <p>Implementations should handle data type conversions, default values for new columns,
+ * and efficient column-wise data access patterns.
+ */
+public interface ColumnReader extends Closeable, Serializable {
+
+ /**
+ * Return <code>true</code> if more values remain to be read in this column.
+ * <p>This method should not throw exception. Caller is not responsible for handling exceptions from this method.
+ */
+ boolean hasNext();
+
+ /**
+ * Get the next value in the column.
+ * <p>This method should be called only if {@link #hasNext()} returns <code>true</code>. Caller is responsible for
+ * handling exceptions from this method and skip the value if user wants to continue reading the remaining values.
+ *
+ * @return Next column value, or null if the value is null
+ * @throws IOException If an I/O error occurs while reading
+ */
+ @Nullable
+ Object next() throws IOException;
+
+ /**
+ * Check if the current value (last returned by next()) is null.
+ * <p>This method should be called after {@link #next()} to check if the returned value represents a null.
+ *
+ * @return true if the current value is null, false otherwise
+ */
+ boolean isNull();
Review Comment:
Why do we need this when `next()` can return `null`?
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReader.java:
##########
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.spi.data.readers;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.Serializable;
+import javax.annotation.Nullable;
+
+
+/**
+ * The <code>ColumnReader</code> interface is used to read column data from various data sources
+ * for columnar segment building. Unlike RecordReader which reads row-by-row, ColumnReader provides
+ * column-wise access to data, enabling efficient columnar segment creation.
+ *
+ * <p>This interface follows an iterator pattern similar to RecordReader:
+ * <ul>
+ * <li>Sequential iteration over all values in a column using hasNext() and next()</li>
+ * <li>Null value detection for the current value</li>
+ * <li>Rewind capability to restart iteration</li>
+ * <li>Resource cleanup</li>
+ * </ul>
+ *
+ * <p>Implementations should handle data type conversions, default values for new columns,
+ * and efficient column-wise data access patterns.
+ */
+public interface ColumnReader extends Closeable, Serializable {
+
+ /**
+ * Return <code>true</code> if more values remain to be read in this column.
+ * <p>This method should not throw exception. Caller is not responsible for handling exceptions from this method.
+ */
+ boolean hasNext();
+
+ /**
+ * Get the next value in the column.
+ * <p>This method should be called only if {@link #hasNext()} returns <code>true</code>. Caller is responsible for
+ * handling exceptions from this method and skip the value if user wants to continue reading the remaining values.
+ *
+ * @return Next column value, or null if the value is null
+ * @throws IOException If an I/O error occurs while reading
+ */
+ @Nullable
+ Object next() throws IOException;
Review Comment:
(format) Reformat the changes with [Pinot Style](https://docs.pinot.apache.org/developers/developers-and-contributors/code-setup#set-up-ide)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] build segment columnarly from a column major datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2318120825
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java:
##########
@@ -169,31 +170,73 @@ public void init(SegmentGeneratorConfig config, RecordReader recordReader)
new TransformPipeline(config.getTableConfig(), config.getSchema()));
}
+ /**
+ * Initialize the driver for columnar segment building using a ColumnReaderFactory.
+ * This method sets up the driver to use column-wise input data access instead of row-wise.
+ *
+ * @param config Segment generator configuration
+ * @param columnReaderFactory Factory for creating column readers
+ * @throws Exception if initialization fails
+ */
+ public void init(SegmentGeneratorConfig config, ColumnReaderFactory columnReaderFactory)
+ throws Exception {
+ // Initialize the column reader factory with target schema
+ columnReaderFactory.init(config.getSchema());
+
+ // Create all column readers for the target schema
+ Map<String, ColumnReader> columnReaders = columnReaderFactory.getAllColumnReaders();
+
+ // Create columnar data source
+ ColumnarSegmentCreationDataSource columnarDataSource =
+ new ColumnarSegmentCreationDataSource(columnReaderFactory, columnReaders);
+
+ // Use the existing init method with columnar data source and no transform pipeline
+ init(config, columnarDataSource, null);
+
+ LOGGER.info("Initialized SegmentIndexCreationDriverImpl for columnar building with {} columns",
+ columnReaders.size());
+ }
+
public void init(SegmentGeneratorConfig config, SegmentCreationDataSource dataSource,
TransformPipeline transformPipeline)
throws Exception {
_config = config;
- _recordReader = dataSource.getRecordReader();
_dataSchema = config.getSchema();
_continueOnError = config.isContinueOnError();
+ String readerClassName = null;
+
+ // Handle columnar data sources differently
+ if (dataSource instanceof ColumnarSegmentCreationDataSource) {
+ // For columnar data sources, we don't have a record reader
+ _recordReader = null;
+ _transformPipeline = null; // No transform pipeline for columnar mode
+ _dataSource = dataSource;
+ // Initialize common components with columnar-specific reader class name
Review Comment:
self review: remove stale comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2356052703
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReaderImpl.java:
##########
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.readers;
+
+import java.io.IOException;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.utils.PinotDataType;
+import org.apache.pinot.segment.local.recordtransformer.DataTypeTransformer;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Implementation of ColumnReader for Pinot segments.
+ *
+ * <p>This class wraps the existing PinotSegmentColumnReader and provides the ColumnReader interface
+ * for columnar segment building. It handles:
+ * <ul>
+ * <li>Reading column values from Pinot segments</li>
+ * <li>Data type conversions when target schema differs from source</li>
+ * <li>Null value detection</li>
+ * <li>Resource cleanup</li>
+ * </ul>
+ */
+public class PinotSegmentColumnReaderImpl implements ColumnReader {
+ private static final Logger LOGGER = LoggerFactory.getLogger(PinotSegmentColumnReaderImpl.class);
+
+ private final PinotSegmentColumnReader _segmentColumnReader;
+ private final String _columnName;
+ private final int _numDocs;
+ private final boolean _needsConversion;
+ private final FieldSpec _targetFieldSpec;
+
+ private int _currentIndex;
+ private Object _currentValue;
+ private boolean _currentIsNull;
+
+ /**
+ * Create a PinotSegmentColumnReaderImpl for an existing column in the segment.
+ *
+ * @param indexSegment Source segment to read from
+ * @param columnName Name of the column
+ * @param targetFieldSpec Target field specification (may differ from source for data type conversion)
+ */
+ public PinotSegmentColumnReaderImpl(IndexSegment indexSegment, String columnName, FieldSpec targetFieldSpec) {
+ _segmentColumnReader = new PinotSegmentColumnReader(indexSegment, columnName);
+ _columnName = columnName;
+ _numDocs = indexSegment.getSegmentMetadata().getTotalDocs();
+ _targetFieldSpec = targetFieldSpec;
+ _currentIndex = 0;
+
+ // Check if data type conversion is needed
+ FieldSpec sourceFieldSpec = indexSegment.getSegmentMetadata().getSchema().getFieldSpecFor(columnName);
+ _needsConversion = sourceFieldSpec != null
Review Comment:
This might not be an allowed change like data type change so it is safe to skip.
The forward index files for sv and mv will need to be rewritten in Pinot for existing data while the data type change does not require this. Pinot still might allow this change to make with force schema update API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 merged PR #16727:
URL: https://github.com/apache/pinot/pull/16727
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "krishan1390 (via GitHub)" <gi...@apache.org>.
krishan1390 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2447699470
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReaderFactory.java:
##########
@@ -0,0 +1,154 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.readers;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.ColumnReaderFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * ColumnReaderFactory implementation for immutable Pinot segments.
+ *
+ * <p>This factory creates ColumnReader instances for reading data from Pinot segments
+ * in a columnar fashion. It handles:
+ * <ul>
+ * <li>Creating readers for existing columns in the segment</li>
+ * <li>Creating default value readers for new columns</li>
+ * <li>Resource management for all created readers</li>
+ * </ul>
+ */
+public class PinotSegmentColumnReaderFactory implements ColumnReaderFactory {
Review Comment:
I don't think createAllColumnReaders() should be public.
Rather init() should call createColumnReader() and createColumnReader() should be private. and getAllColumnReaders() should just get existing objects without creating new objects
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2468689810
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReaderFactory.java:
##########
@@ -0,0 +1,154 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.readers;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.ColumnReaderFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * ColumnReaderFactory implementation for immutable Pinot segments.
+ *
+ * <p>This factory creates ColumnReader instances for reading data from Pinot segments
+ * in a columnar fashion. It handles:
+ * <ul>
+ * <li>Creating readers for existing columns in the segment</li>
+ * <li>Creating default value readers for new columns</li>
+ * <li>Resource management for all created readers</li>
+ * </ul>
+ */
+public class PinotSegmentColumnReaderFactory implements ColumnReaderFactory {
Review Comment:
Revisited the interface for ColumnReaderFactory. There will be no create* methods exposed. The getMethods will return cached Readers and in future if needed one can add create methods if every time a new object needs to be created from the Factory. Same documentation in the interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "krishan1390 (via GitHub)" <gi...@apache.org>.
krishan1390 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2431747397
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java:
##########
@@ -171,31 +170,72 @@ public void init(SegmentGeneratorConfig config, RecordReader recordReader)
new TransformPipeline(config.getTableConfig(), config.getSchema()));
}
+ /**
+ * Initialize the driver for columnar segment building using a ColumnReaderFactory.
+ * This method sets up the driver to use column-wise input data access instead of row-wise.
+ *
+ * @param config Segment generator configuration
+ * @param columnReaderFactory Factory for creating column readers
+ * @throws Exception if initialization fails
+ */
+ public void init(SegmentGeneratorConfig config, ColumnReaderFactory columnReaderFactory)
+ throws Exception {
+ // Initialize the column reader factory with target schema
+ columnReaderFactory.init(config.getSchema());
+
+ // Create all column readers for the target schema
+ Map<String, ColumnReader> columnReaders = columnReaderFactory.getAllColumnReaders();
+
+ // Create columnar data source
+ ColumnarSegmentCreationDataSource columnarDataSource =
+ new ColumnarSegmentCreationDataSource(columnReaderFactory, columnReaders);
+
+ // Use the existing init method with columnar data source and no transform pipeline
+ init(config, columnarDataSource, null);
Review Comment:
so its not OK to skip it in all cases.
can you document use cases where columnar segment build can't be used ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2431176573
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReaderFactory.java:
##########
@@ -0,0 +1,92 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.spi.data.readers;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Map;
+import java.util.Set;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+
+
+/**
+ * Factory interface for creating ColumnReader instances for different data sources.
+ *
+ * <p>This factory provides a generic way to create column readers for various data sources
+ * such as Pinot segments, Parquet files, etc. The factory handles:
+ * <ul>
+ * <li>Creating column readers for existing columns</li>
+ * <li>Creating default value readers for new columns</li>
+ * <li>Data type conversions between source and target schemas</li>
+ * <li>Resource management</li>
+ * </ul>
+ */
+public interface ColumnReaderFactory extends Closeable, Serializable {
+
+ /**
+ * Initialize the factory with the data source and target schema.
+ *
+ * @param targetSchema Target schema for the output segment
+ * @throws IOException If initialization fails
+ */
+ void init(Schema targetSchema) throws IOException;
+
+ /**
+ * Get the set of column names available in the source data.
+ *
+ * @return Set of available column names in the source
+ */
+ Set<String> getAvailableColumns();
+
+ /**
+ * Get the total number of documents/rows in the data source.
+ *
+ * @return Total number of documents
+ */
+ int getNumDocs();
+
+ /**
+ * Create a column reader for the specified column.
+ *
+ * @param columnName Name of the column to read
+ * @param targetFieldSpec Target field specification from the output schema
+ * @return ColumnReader instance for the specified column
+ * @throws IOException If the column reader cannot be created
+ */
+ ColumnReader createColumnReader(String columnName, FieldSpec targetFieldSpec) throws IOException;
+
+ /**
+ * Get all column readers for the target schema.
+ * This is a convenience method that creates readers for all columns in the target schema.
+ *
+ * @return Map of column name to ColumnReader
+ * @throws IOException If any column reader cannot be created
+ */
+ Map<String, ColumnReader> getAllColumnReaders() throws IOException;
+
+ /**
+ * Check if the specified column exists in the source data.
+ *
+ * @param columnName Column name to check
+ * @return true if the column exists in source, false otherwise
+ */
+ boolean hasColumn(String columnName);
Review Comment:
Removing these from the ColumnReaderFactory interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2356156792
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReader.java:
##########
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.spi.data.readers;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.Serializable;
+import javax.annotation.Nullable;
+
+
+/**
+ * The <code>ColumnReader</code> interface is used to read column data from various data sources
+ * for columnar segment building. Unlike RecordReader which reads row-by-row, ColumnReader provides
+ * column-wise access to data, enabling efficient columnar segment creation.
+ *
+ * <p>This interface follows an iterator pattern similar to RecordReader:
+ * <ul>
+ * <li>Sequential iteration over all values in a column using hasNext() and next()</li>
+ * <li>Null value detection for the current value</li>
+ * <li>Rewind capability to restart iteration</li>
+ * <li>Resource cleanup</li>
+ * </ul>
+ *
+ * <p>Implementations should handle data type conversions, default values for new columns,
+ * and efficient column-wise data access patterns.
+ */
+public interface ColumnReader extends Closeable, Serializable {
Review Comment:
This sounds good, I will have a follow up PR to apply this feature in minion refresh job. Will handle the enhancements in a separate PR for upsert merger tasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on PR #16727:
URL: https://github.com/apache/pinot/pull/16727#issuecomment-3345391271
Offline discussion:
- Remove any transformation including data type which is not part of schema/index update rather ingestion transformation.
- Follow up: Build columnar ingestion transformation and later improve columnar build to support transformation. Relates to requirements in https://github.com/apache/pinot/issues/16643
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2432546141
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java:
##########
@@ -677,4 +706,63 @@ public int getSkippedRowsFound() {
public int getSanitizedRowsFound() {
return _sanitizedRowsFound;
}
+
+ /**
+ * Build segment using columnar approach.
+ * This method builds the segment by processing data column-wise instead of row-wise.
+ *
+ * @throws Exception if segment building fails
+ */
+ public void buildColumnar() throws Exception {
+ if (!(_dataSource instanceof ColumnarSegmentCreationDataSource)) {
+ throw new IllegalStateException("buildColumnar() can only be called after initColumnar()");
+ }
+
+ ColumnarSegmentCreationDataSource columnarDataSource = (ColumnarSegmentCreationDataSource) _dataSource;
+ Map<String, ColumnReader> columnReaders = columnarDataSource.getColumnReaders();
+
+ LOGGER.info("Starting columnar segment building for {} columns", columnReaders.size());
+
+ // Reuse existing stats collection and index creation info logic
+ LOGGER.debug("Start building StatsCollector!");
+ collectStatsAndIndexCreationInfo();
+ LOGGER.info("Finished building StatsCollector!");
+ LOGGER.info("Collected stats for {} documents", _totalDocs);
+
+ if (_totalDocs == 0) {
+ LOGGER.warn("No documents found in data source");
+ handlePostCreation();
+ return;
+ }
+
+ try {
+ // Initialize the index creation using the per-column statistics information
+ _indexCreator.init(_config, _segmentIndexCreationInfo, _indexCreationInfoMap, _dataSchema, _tempIndexDir, null);
+
+ // Build the indexes column-wise (true column-major approach)
+ LOGGER.info("Start building Index using columnar approach");
+ long indexStartTime = System.nanoTime();
+
+ TreeSet<String> columns = _dataSchema.getPhysicalColumnNames();
+ for (String columnName : columns) {
+ LOGGER.debug("Indexing column: {}", columnName);
+ ColumnReader columnReader = columnReaders.get(columnName);
+ if (columnReader == null) {
+ throw new IllegalStateException("No column reader found for column: " + columnName);
+ }
+
+ // Index each column independently using true column-major approach
+ // This is similar to how buildByColumn works but uses ColumnReader instead of IndexSegment
+ ((SegmentColumnarIndexCreator) _indexCreator).indexColumn(columnName, columnReader);
+ }
+
+ _totalIndexTimeNs = System.nanoTime() - indexStartTime;
+ } catch (Exception e) {
+ _indexCreator.close();
+ throw e;
+ }
Review Comment:
makes sense, added close method to ColumnarSegmentCreationDataSource and invoked here which will close all columnReaders.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2366374083
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReader.java:
##########
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.spi.data.readers;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.Serializable;
+import javax.annotation.Nullable;
+
+
+/**
+ * The <code>ColumnReader</code> interface is used to read column data from various data sources
+ * for columnar segment building. Unlike RecordReader which reads row-by-row, ColumnReader provides
+ * column-wise access to data, enabling efficient columnar segment creation.
+ *
+ * <p>This interface follows an iterator pattern similar to RecordReader:
+ * <ul>
+ * <li>Sequential iteration over all values in a column using hasNext() and next()</li>
+ * <li>Null value detection for the current value</li>
+ * <li>Rewind capability to restart iteration</li>
+ * <li>Resource cleanup</li>
+ * </ul>
+ *
+ * <p>Implementations should handle data type conversions, default values for new columns,
+ * and efficient column-wise data access patterns.
+ */
+public interface ColumnReader extends Closeable, Serializable {
Review Comment:
I do plan to implement this in future, but not an immediate priority. I can create an issue for this as follow up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "krishan1390 (via GitHub)" <gi...@apache.org>.
krishan1390 commented on PR #16727:
URL: https://github.com/apache/pinot/pull/16727#issuecomment-3454922169
@rohityadav1993 whats the plan for the open comments and failing tests and merge conflicts ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "noob-se7en (via GitHub)" <gi...@apache.org>.
noob-se7en commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2440711113
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java:
##########
@@ -677,4 +706,63 @@ public int getSkippedRowsFound() {
public int getSanitizedRowsFound() {
return _sanitizedRowsFound;
}
+
+ /**
+ * Build segment using columnar approach.
+ * This method builds the segment by processing data column-wise instead of row-wise.
+ *
+ * @throws Exception if segment building fails
+ */
+ public void buildColumnar() throws Exception {
+ if (!(_dataSource instanceof ColumnarSegmentCreationDataSource)) {
+ throw new IllegalStateException("buildColumnar() can only be called after initColumnar()");
+ }
+
+ ColumnarSegmentCreationDataSource columnarDataSource = (ColumnarSegmentCreationDataSource) _dataSource;
+ Map<String, ColumnReader> columnReaders = columnarDataSource.getColumnReaders();
+
+ LOGGER.info("Starting columnar segment building for {} columns", columnReaders.size());
+
+ // Reuse existing stats collection and index creation info logic
+ LOGGER.debug("Start building StatsCollector!");
+ collectStatsAndIndexCreationInfo();
+ LOGGER.info("Finished building StatsCollector!");
+ LOGGER.info("Collected stats for {} documents", _totalDocs);
+
+ if (_totalDocs == 0) {
+ LOGGER.warn("No documents found in data source");
+ handlePostCreation();
+ return;
+ }
+
+ try {
+ // Initialize the index creation using the per-column statistics information
+ _indexCreator.init(_config, _segmentIndexCreationInfo, _indexCreationInfoMap, _dataSchema, _tempIndexDir, null);
+
+ // Build the indexes column-wise (true column-major approach)
+ LOGGER.info("Start building Index using columnar approach");
+ long indexStartTime = System.nanoTime();
+
+ TreeSet<String> columns = _dataSchema.getPhysicalColumnNames();
+ for (String columnName : columns) {
+ LOGGER.debug("Indexing column: {}", columnName);
+ ColumnReader columnReader = columnReaders.get(columnName);
+ if (columnReader == null) {
+ throw new IllegalStateException("No column reader found for column: " + columnName);
+ }
+
+ // Index each column independently using true column-major approach
+ // This is similar to how buildByColumn works but uses ColumnReader instead of IndexSegment
+ ((SegmentColumnarIndexCreator) _indexCreator).indexColumn(columnName, columnReader);
+ }
+
+ _totalIndexTimeNs = System.nanoTime() - indexStartTime;
+ } catch (Exception e) {
+ _indexCreator.close();
+ throw e;
+ }
Review Comment:
col readers and dataSource also need to be closed outside of catch block?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2430150403
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java:
##########
@@ -677,4 +706,63 @@ public int getSkippedRowsFound() {
public int getSanitizedRowsFound() {
return _sanitizedRowsFound;
}
+
+ /**
+ * Build segment using columnar approach.
+ * This method builds the segment by processing data column-wise instead of row-wise.
+ *
+ * @throws Exception if segment building fails
+ */
+ public void buildColumnar() throws Exception {
Review Comment:
I think the segment creation should fail even if single columnReader errors which is happening currently. We are not doing transformation which is where continueOnError is handled in row major building.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on PR #16727:
URL: https://github.com/apache/pinot/pull/16727#issuecomment-3473881405
@Jackie-Jiang , @KKcorps please review and help merge this soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Columnar segment build clean [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #16727:
URL: https://github.com/apache/pinot/pull/16727#issuecomment-3241559257
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/16727?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `6.78571%` with `261 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 56.43%. Comparing base ([`777936a`](https://app.codecov.io/gh/apache/pinot/commit/777936a28e50fc075b934b4c038b29a2e6832e86?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`770a6ad`](https://app.codecov.io/gh/apache/pinot/commit/770a6adae56fce8545ae83f9ffd259d97c8b3f7f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 34 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/16727?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...l/stats/ColumnarSegmentPreIndexStatsContainer.java](https://app.codecov.io/gh/apache/pinot/pull/16727?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fsegment%2Fcreator%2Fimpl%2Fstats%2FColumnarSegmentPreIndexStatsContainer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9zdGF0cy9Db2x1bW5hclNlZ21lbnRQcmVJbmRleFN0YXRzQ29udGFpbmVyLmphdmE=) | 0.00% | [71 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/16727?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...t/creator/impl/SegmentIndexCreationDriverImpl.java](https://app.codecov.io/gh/apache/pinot/pull/16727?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fsegment%2Fcreator%2Fimpl%2FSegmentIndexCreationDriverImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50SW5kZXhDcmVhdGlvbkRyaXZlckltcGwuamF2YQ==) | 11.66% | [48 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/16727?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...gment/readers/PinotSegmentColumnReaderFactory.java](https://app.codecov.io/gh/apache/pinot/pull/16727?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fsegment%2Freaders%2FPinotSegmentColumnReaderFactory.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L3JlYWRlcnMvUGlub3RTZWdtZW50Q29sdW1uUmVhZGVyRmFjdG9yeS5qYXZh) | 0.00% | [44 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/16727?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../segment/readers/PinotSegmentColumnReaderImpl.java](https://app.codecov.io/gh/apache/pinot/pull/16727?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fsegment%2Freaders%2FPinotSegmentColumnReaderImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L3JlYWRlcnMvUGlub3RTZWdtZW50Q29sdW1uUmVhZGVySW1wbC5qYXZh) | 0.00% | [39 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/16727?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://app.codecov.io/gh/apache/pinot/pull/16727?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fsegment%2Fcreator%2Fimpl%2FSegmentColumnarIndexCreator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | 0.00% | [19 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/16727?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ocal/segment/readers/DefaultValueColumnReader.java](https://app.codecov.io/gh/apache/pinot/pull/16727?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fsegment%2Freaders%2FDefaultValueColumnReader.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L3JlYWRlcnMvRGVmYXVsdFZhbHVlQ29sdW1uUmVhZGVyLmphdmE=) | 0.00% | [19 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/16727?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ent/creator/ColumnarSegmentCreationDataSource.java](https://app.codecov.io/gh/apache/pinot/pull/16727?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fsegment%2Fcreator%2FColumnarSegmentCreationDataSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvQ29sdW1uYXJTZWdtZW50Q3JlYXRpb25EYXRhU291cmNlLmphdmE=) | 0.00% | [10 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/16727?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...t/local/recordtransformer/DataTypeTransformer.java](https://app.codecov.io/gh/apache/pinot/pull/16727?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Frecordtransformer%2FDataTypeTransformer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWNvcmR0cmFuc2Zvcm1lci9EYXRhVHlwZVRyYW5zZm9ybWVyLmphdmE=) | 66.66% | [3 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/16727?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (777936a) and HEAD (770a6ad). Click for more details.
>
> <details><summary>HEAD has 28 uploads less than BASE</summary>
>
>| Flag | BASE (777936a) | HEAD (770a6ad) |
>|------|------|------|
>|java-21|4|1|
>|unittests1|2|1|
>|unittests|4|1|
>|temurin|8|1|
>|unittests2|2|0|
>|integration|4|0|
>|integration1|2|0|
>|java-11|4|0|
>|custom-integration1|2|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #16727 +/- ##
============================================
- Coverage 63.33% 56.43% -6.90%
+ Complexity 1379 699 -680
============================================
Files 3036 2388 -648
Lines 176785 132350 -44435
Branches 27131 21070 -6061
============================================
- Hits 111971 74695 -37276
+ Misses 56227 51452 -4775
+ Partials 8587 6203 -2384
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/16727/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/16727/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/16727/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/16727/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/16727/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/16727/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.43% <6.78%> (-6.88%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/16727/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.43% <6.78%> (-6.90%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/16727/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.43% <6.78%> (-6.90%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/16727/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.43% <6.78%> (+<0.01%)` | :arrow_up: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/16727/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/16727?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on PR #16727:
URL: https://github.com/apache/pinot/pull/16727#issuecomment-3473865607
Merge conflicts resolved and comments addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "krishan1390 (via GitHub)" <gi...@apache.org>.
krishan1390 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2447694277
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.stats;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Stats container that efficiently collects statistics from columnar data using ColumnReader instances.
+ *
+ * <p>This implementation collects statistics by iterating column-wise instead of row-wise,
+ * which is more efficient for columnar data sources. It supports:
+ * <ul>
+ * <li>Column-wise statistics collection</li>
+ * <li>Existing columns from source data</li>
+ * <li>New columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ *
+ * <p>The statistics are collected using the same underlying collectors as the row-based approach
+ * (SegmentPreIndexStatsCollectorImpl) but with more efficient column-wise iteration.
+ */
+public class ColumnarSegmentPreIndexStatsContainer implements SegmentPreIndexStatsCollector {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentPreIndexStatsContainer.class);
+
+ private final Map<String, ColumnReader> _columnReaders;
+ private final StatsCollectorConfig _statsCollectorConfig;
+ private final Schema _targetSchema;
+ private final Map<String, AbstractColumnStatisticsCollector> _columnStatsCollectorMap;
+ private int _totalDocCount;
+
+ /**
+ * Create a ColumnarSegmentPreIndexStatsContainer.
+ *
+ * @param columnReaders Map of column name to ColumnReader instances
+ * @param statsCollectorConfig Configuration for statistics collection
+ */
+ public ColumnarSegmentPreIndexStatsContainer(Map<String, ColumnReader> columnReaders,
+ StatsCollectorConfig statsCollectorConfig) {
+ _columnReaders = columnReaders;
+ _statsCollectorConfig = statsCollectorConfig;
+ _targetSchema = statsCollectorConfig.getSchema();
+ _columnStatsCollectorMap = new HashMap<>();
+ _totalDocCount = -1; // indicates unset
+
+ initializeStatsCollectors();
+ collectColumnStats();
+ }
+
+ /**
+ * Initialize stats collectors for all columns in the target schema.
+ */
+ private void initializeStatsCollectors() {
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
Review Comment:
@rohityadav1993 we should do this change before merging the PR
cc @KKcorps @noob-se7en
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "tarun11Mavani (via GitHub)" <gi...@apache.org>.
tarun11Mavani commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2354407791
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReaderFactory.java:
##########
@@ -0,0 +1,151 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.readers;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.ColumnReaderFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * ColumnReaderFactory implementation for immutable Pinot segments.
+ *
+ * <p>This factory creates ColumnReader instances for reading data from Pinot segments
+ * in a columnar fashion. It handles:
+ * <ul>
+ * <li>Creating readers for existing columns in the segment</li>
+ * <li>Creating default value readers for new columns</li>
+ * <li>Data type conversions between source and target schemas</li>
+ * <li>Resource management for all created readers</li>
+ * </ul>
+ */
+public class PinotSegmentColumnReaderFactory implements ColumnReaderFactory {
+ private static final Logger LOGGER = LoggerFactory.getLogger(PinotSegmentColumnReaderFactory.class);
+
+ private final IndexSegment _indexSegment;
+ private Schema _targetSchema;
+ private final Map<String, ColumnReader> _columnReaders;
+
+ /**
+ * Create a PinotSegmentColumnReaderFactory.
+ *
+ * @param indexSegment Source segment to read from
+ */
+ public PinotSegmentColumnReaderFactory(IndexSegment indexSegment) {
+ _indexSegment = indexSegment;
+ _columnReaders = new HashMap<>();
+ }
+
+ @Override
+ public void init(Schema targetSchema) throws IOException {
+ _targetSchema = targetSchema;
+ LOGGER.info("Initialized PinotSegmentColumnReaderFactory with target schema containing {} columns",
+ targetSchema.getPhysicalColumnNames().size());
+ }
+
+ @Override
+ public Set<String> getAvailableColumns() {
+ return _indexSegment.getPhysicalColumnNames();
+ }
+
+ @Override
+ public int getNumDocs() {
+ return _indexSegment.getSegmentMetadata().getTotalDocs();
+ }
+
+ @Override
+ public ColumnReader createColumnReader(String columnName, FieldSpec targetFieldSpec) throws IOException {
+ if (_targetSchema == null) {
+ throw new IllegalStateException("Factory not initialized. Call init() first.");
+ }
+
+ // Check if we already have a reader for this column
+ ColumnReader existingReader = _columnReaders.get(columnName);
+ if (existingReader != null) {
+ return existingReader;
+ }
+
+ ColumnReader columnReader;
+
+ if (hasColumn(columnName)) {
Review Comment:
We are skipping VirtualColumns in getAllColumnReaders but this can still be called for virtual column from somewhere else. We should skip VirtualColumns here as well.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReaderImpl.java:
##########
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.readers;
+
+import java.io.IOException;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.utils.PinotDataType;
+import org.apache.pinot.segment.local.recordtransformer.DataTypeTransformer;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Implementation of ColumnReader for Pinot segments.
+ *
+ * <p>This class wraps the existing PinotSegmentColumnReader and provides the ColumnReader interface
+ * for columnar segment building. It handles:
+ * <ul>
+ * <li>Reading column values from Pinot segments</li>
+ * <li>Data type conversions when target schema differs from source</li>
+ * <li>Null value detection</li>
+ * <li>Resource cleanup</li>
+ * </ul>
+ */
+public class PinotSegmentColumnReaderImpl implements ColumnReader {
+ private static final Logger LOGGER = LoggerFactory.getLogger(PinotSegmentColumnReaderImpl.class);
+
+ private final PinotSegmentColumnReader _segmentColumnReader;
+ private final String _columnName;
+ private final int _numDocs;
+ private final boolean _needsConversion;
+ private final FieldSpec _targetFieldSpec;
+
+ private int _currentIndex;
+ private Object _currentValue;
+ private boolean _currentIsNull;
+
+ /**
+ * Create a PinotSegmentColumnReaderImpl for an existing column in the segment.
+ *
+ * @param indexSegment Source segment to read from
+ * @param columnName Name of the column
+ * @param targetFieldSpec Target field specification (may differ from source for data type conversion)
+ */
+ public PinotSegmentColumnReaderImpl(IndexSegment indexSegment, String columnName, FieldSpec targetFieldSpec) {
+ _segmentColumnReader = new PinotSegmentColumnReader(indexSegment, columnName);
+ _columnName = columnName;
+ _numDocs = indexSegment.getSegmentMetadata().getTotalDocs();
+ _targetFieldSpec = targetFieldSpec;
+ _currentIndex = 0;
+
+ // Check if data type conversion is needed
+ FieldSpec sourceFieldSpec = indexSegment.getSegmentMetadata().getSchema().getFieldSpecFor(columnName);
+ _needsConversion = sourceFieldSpec != null
Review Comment:
what if the data type is same but we are transforming from SV to MV?
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -383,6 +384,46 @@ public void indexColumn(String columnName, @Nullable int[] sortedDocIds, IndexSe
}
}
+ /**
+ * Index a column using a ColumnReader (column-major approach).
+ * This method processes the column values using the iterator pattern from ColumnReader.
+ *
+ * @param columnName Name of the column to index
+ * @param columnReader ColumnReader for the column data
+ * @throws IOException if indexing fails
+ */
+ public void indexColumn(String columnName, ColumnReader columnReader) throws IOException {
Review Comment:
Can add this method signature to the interface SegmentCreator.
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReader.java:
##########
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.spi.data.readers;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.Serializable;
+import javax.annotation.Nullable;
+
+
+/**
+ * The <code>ColumnReader</code> interface is used to read column data from various data sources
+ * for columnar segment building. Unlike RecordReader which reads row-by-row, ColumnReader provides
+ * column-wise access to data, enabling efficient columnar segment creation.
+ *
+ * <p>This interface follows an iterator pattern similar to RecordReader:
+ * <ul>
+ * <li>Sequential iteration over all values in a column using hasNext() and next()</li>
+ * <li>Null value detection for the current value</li>
+ * <li>Rewind capability to restart iteration</li>
+ * <li>Resource cleanup</li>
+ * </ul>
+ *
+ * <p>Implementations should handle data type conversions, default values for new columns,
+ * and efficient column-wise data access patterns.
+ */
+public interface ColumnReader extends Closeable, Serializable {
Review Comment:
Can define another implementation(CompactedColumnReader) of this in future to include validDocIds and sortedColumn to support compaction. This can be used to improve compact segment build time for UpsertCompaction and UpsertMergeCompact task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2430149193
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.stats;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Stats container that efficiently collects statistics from columnar data using ColumnReader instances.
+ *
+ * <p>This implementation collects statistics by iterating column-wise instead of row-wise,
+ * which is more efficient for columnar data sources. It supports:
+ * <ul>
+ * <li>Column-wise statistics collection</li>
+ * <li>Existing columns from source data</li>
+ * <li>New columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ *
+ * <p>The statistics are collected using the same underlying collectors as the row-based approach
+ * (SegmentPreIndexStatsCollectorImpl) but with more efficient column-wise iteration.
+ */
+public class ColumnarSegmentPreIndexStatsContainer implements SegmentPreIndexStatsCollector {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentPreIndexStatsContainer.class);
+
+ private final Map<String, ColumnReader> _columnReaders;
+ private final StatsCollectorConfig _statsCollectorConfig;
+ private final Schema _targetSchema;
+ private final Map<String, AbstractColumnStatisticsCollector> _columnStatsCollectorMap;
+ private int _totalDocCount;
+
+ /**
+ * Create a ColumnarSegmentPreIndexStatsContainer.
+ *
+ * @param columnReaders Map of column name to ColumnReader instances
+ * @param statsCollectorConfig Configuration for statistics collection
+ */
+ public ColumnarSegmentPreIndexStatsContainer(Map<String, ColumnReader> columnReaders,
+ StatsCollectorConfig statsCollectorConfig) {
+ _columnReaders = columnReaders;
+ _statsCollectorConfig = statsCollectorConfig;
+ _targetSchema = statsCollectorConfig.getSchema();
+ _columnStatsCollectorMap = new HashMap<>();
+ _totalDocCount = -1; // indicates unset
+
+ initializeStatsCollectors();
+ collectColumnStats();
+ }
+
+ /**
+ * Initialize stats collectors for all columns in the target schema.
+ */
+ private void initializeStatsCollectors() {
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
+ switch (fieldSpec.getDataType().getStoredType()) {
+ case INT:
+ _columnStatsCollectorMap.put(columnName,
+ new IntColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case LONG:
+ _columnStatsCollectorMap.put(columnName,
+ new LongColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case FLOAT:
+ _columnStatsCollectorMap.put(columnName,
+ new FloatColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case DOUBLE:
+ _columnStatsCollectorMap.put(columnName,
+ new DoubleColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BIG_DECIMAL:
+ _columnStatsCollectorMap.put(columnName,
+ new BigDecimalColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case STRING:
+ _columnStatsCollectorMap.put(columnName,
+ new StringColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BYTES:
+ _columnStatsCollectorMap.put(columnName,
+ new BytesColumnPredIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case MAP:
+ _columnStatsCollectorMap.put(columnName,
+ new MapColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ default:
+ throw new IllegalStateException("Unsupported data type: " + fieldSpec.getDataType());
+ }
+ }
+ }
+
+ /**
+ * Collect stats by iterating column-wise using the provided ColumnReader instances.
+ */
+ private void collectColumnStats() {
+ LOGGER.info("Collecting stats for {} columns using column-wise iteration", _columnReaders.size());
+
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
+ AbstractColumnStatisticsCollector statsCollector = _columnStatsCollectorMap.get(columnName);
+ ColumnReader columnReader = _columnReaders.get(columnName);
+
+ if (columnReader == null) {
+ throw new RuntimeException("Column reader for column " + columnName + " not found");
+ }
+
+ LOGGER.debug("Collecting stats for column: {}", columnName);
+ collectStatsFromColumnReader(columnName, columnReader, statsCollector);
+
+ // Seal the stats collector
+ statsCollector.seal();
+ }
+ }
+
+ /**
+ * Collect stats from a column reader by iterating over all values using the iterator pattern.
+ */
+ private void collectStatsFromColumnReader(String columnName, ColumnReader columnReader,
+ AbstractColumnStatisticsCollector statsCollector) {
+ try {
+ // Reset the column reader to start from the beginning
+ columnReader.rewind();
+
+ int docCount = 0;
+ while (columnReader.hasNext()) {
+ Object value = columnReader.next();
+ statsCollector.collect(value);
Review Comment:
What exceptions should be handled here? Failing fast should be fine in case of runtime exceptions, this is not used in any realtime ingestion flow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2430160637
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/creator/impl/ColumnarSegmentBuildingTest.java:
##########
@@ -0,0 +1,500 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl;
+
+import com.google.common.collect.Lists;
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.segment.readers.PinotSegmentColumnReaderFactory;
+import org.apache.pinot.segment.spi.ImmutableSegment;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.data.readers.RecordReader;
+import org.apache.pinot.spi.data.readers.RecordReaderConfig;
+import org.apache.pinot.spi.utils.ReadMode;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+
+/**
+ * Tests for columnar segment building functionality.
+ *
+ * <p>This test class validates the new columnar segment building capability including:
+ * <ul>
+ * <li>ColumnReader interface implementations</li>
+ * <li>ColumnReaderFactory for different data sources</li>
+ * <li>Columnar segment building vs row-major building equivalence</li>
+ * <li>Support for new columns with default values</li>
+ * </ul>
+ */
+public class ColumnarSegmentBuildingTest {
+ private static final String TEMP_DIR = System.getProperty("java.io.tmpdir");
+ private static final String TABLE_NAME = "testTable";
+ private static final String SEGMENT_NAME = "testSegment";
+
+ // Test columns
+ private static final String STRING_COL_1 = "stringCol1";
+ private static final String STRING_COL_2 = "stringCol2";
+ private static final String INT_COL_1 = "intCol1";
+ private static final String INT_COL_2 = "intCol2";
+ private static final String LONG_COL = "longCol";
+ private static final String FLOAT_COL = "floatCol";
+ private static final String DOUBLE_COL = "doubleCol";
+ private static final String BIG_DECIMAL_COL = "bigDecimalCol";
+ private static final String BYTES_COL = "bytesCol";
+ private static final String TIME_COL = "timeCol";
+ private static final String MV_INT_COL = "mvIntCol";
+ private static final String MV_STRING_COL = "mvStringCol";
+
+ // New column for testing default value handling
+ private static final String NEW_STRING_COL = "newStringCol";
+ private static final String NEW_INT_COL = "newIntCol";
+
+ private File _tempDir;
+ private Schema _originalSchema;
+ private Schema _extendedSchema; // Schema with additional columns
+ private TableConfig _tableConfig;
+ private List<GenericRow> _testData;
+
+ @BeforeClass
+ public void setUp() throws IOException {
+ _tempDir = new File(TEMP_DIR, "ColumnarSegmentBuildingTest");
+ FileUtils.deleteQuietly(_tempDir);
+ _tempDir.mkdirs();
+
+ // Create original schema
+ _originalSchema = new Schema.SchemaBuilder()
+ .addSingleValueDimension(STRING_COL_1, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(STRING_COL_2, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(INT_COL_1, FieldSpec.DataType.INT)
+ .addSingleValueDimension(INT_COL_2, FieldSpec.DataType.INT)
+ .addSingleValueDimension(LONG_COL, FieldSpec.DataType.LONG)
+ .addSingleValueDimension(FLOAT_COL, FieldSpec.DataType.FLOAT)
+ .addSingleValueDimension(DOUBLE_COL, FieldSpec.DataType.DOUBLE)
+ .addSingleValueDimension(BIG_DECIMAL_COL, FieldSpec.DataType.BIG_DECIMAL)
+ .addSingleValueDimension(BYTES_COL, FieldSpec.DataType.BYTES)
+ .addMultiValueDimension(MV_INT_COL, FieldSpec.DataType.INT)
+ .addMultiValueDimension(MV_STRING_COL, FieldSpec.DataType.STRING)
+ .addDateTime(TIME_COL, FieldSpec.DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS")
+ .build();
+
+ // Create extended schema with additional columns
+ _extendedSchema = new Schema.SchemaBuilder()
+ .addSingleValueDimension(STRING_COL_1, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(STRING_COL_2, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(INT_COL_1, FieldSpec.DataType.INT)
+ .addSingleValueDimension(INT_COL_2, FieldSpec.DataType.INT)
+ .addSingleValueDimension(LONG_COL, FieldSpec.DataType.LONG)
+ .addSingleValueDimension(FLOAT_COL, FieldSpec.DataType.FLOAT)
+ .addSingleValueDimension(DOUBLE_COL, FieldSpec.DataType.DOUBLE)
+ .addSingleValueDimension(BIG_DECIMAL_COL, FieldSpec.DataType.BIG_DECIMAL)
+ .addSingleValueDimension(BYTES_COL, FieldSpec.DataType.BYTES)
+ .addMultiValueDimension(MV_INT_COL, FieldSpec.DataType.INT)
+ .addMultiValueDimension(MV_STRING_COL, FieldSpec.DataType.STRING)
+ .addDateTime(TIME_COL, FieldSpec.DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS")
+ .addSingleValueDimension(NEW_STRING_COL, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(NEW_INT_COL, FieldSpec.DataType.INT)
+ .build();
+
+ // Create table config
+ _tableConfig = new TableConfigBuilder(TableType.OFFLINE)
+ .setTableName(TABLE_NAME)
+ .setTimeColumnName(TIME_COL)
+ .setInvertedIndexColumns(Lists.newArrayList(STRING_COL_1, INT_COL_1))
+ .setSortedColumn(INT_COL_1)
+ .build();
+
+ // Generate test data
+ _testData = generateTestData(100);
+ }
+
+ @AfterClass
+ public void tearDown() throws IOException {
+ FileUtils.deleteQuietly(_tempDir);
+ }
+
+ @Test
+ public void testBasicColumnarBuilding() throws Exception {
+ // First create a segment using traditional row-major approach
+ File rowMajorSegmentDir = createRowMajorSegment();
+
+ // Then create a segment using columnar approach from the row-major segment
+ File columnarSegmentDir = createColumnarSegment(rowMajorSegmentDir);
+
+ // Validate that both segments have identical data
+ validateSegmentsIdentical(rowMajorSegmentDir, columnarSegmentDir);
+ }
+
+ @Test
+ public void testColumnarBuildingWithNewColumns() throws Exception {
+ // Create original segment with original schema
+ File originalSegmentDir = createRowMajorSegment();
+
+ // Create new segment with extended schema (has additional columns)
+ File newSegmentDir = createColumnarSegmentWithNewColumns(originalSegmentDir);
+
+ // Validate that the new segment has the additional columns with default values
+ validateSegmentWithNewColumns(newSegmentDir);
+ }
+
+ @Test
+ public void testColumnReaderFactory() throws Exception {
+ // Create a segment to test column reader factory with
+ File segmentDir = createRowMajorSegment();
+ ImmutableSegment segment = ImmutableSegmentLoader.load(segmentDir, ReadMode.mmap);
+
+ try {
+ // Test PinotSegmentColumnReaderFactory
+ try (PinotSegmentColumnReaderFactory factory = new PinotSegmentColumnReaderFactory(segment)) {
+ factory.init(_originalSchema);
+
+ // Test that all expected columns are available
+ Set<String> availableColumns = factory.getAvailableColumns();
+ Assert.assertTrue(availableColumns.contains(STRING_COL_1));
+ Assert.assertTrue(availableColumns.contains(INT_COL_1));
+ Assert.assertTrue(availableColumns.contains(MV_INT_COL));
+
+ // Test creating individual column readers
+ ColumnReader stringReader = factory.createColumnReader(STRING_COL_1,
+ _originalSchema.getFieldSpecFor(STRING_COL_1));
+ Assert.assertEquals(stringReader.getColumnName(), STRING_COL_1);
+
+ ColumnReader mvReader = factory.createColumnReader(MV_INT_COL,
+ _originalSchema.getFieldSpecFor(MV_INT_COL));
+ Assert.assertEquals(mvReader.getColumnName(), MV_INT_COL);
+
+ // Test reading values using iterator pattern
+ Assert.assertTrue(stringReader.hasNext());
+ Object firstStringValue = stringReader.next();
+ Assert.assertEquals(firstStringValue, "string1_0");
+
+ // Test that we can continue reading
+ Assert.assertTrue(stringReader.hasNext());
+ Object secondStringValue = stringReader.next();
+ Assert.assertEquals(secondStringValue, "string1_1");
+
+ // Reset and test again
+ stringReader.rewind();
+ Assert.assertTrue(stringReader.hasNext());
+ firstStringValue = stringReader.next();
+ Assert.assertEquals(firstStringValue, "string1_0");
+
+ // Test creating all column readers
+ Map<String, ColumnReader> allReaders = factory.getAllColumnReaders();
+ Assert.assertEquals(allReaders.size(), _originalSchema.getPhysicalColumnNames().size());
+ }
+ } finally {
+ segment.destroy();
+ }
+ }
+
+ @Test
+ public void testColumnReaderWithNewColumns() throws Exception {
+ // Create a segment to test with
+ File segmentDir = createRowMajorSegment();
+ ImmutableSegment segment = ImmutableSegmentLoader.load(segmentDir, ReadMode.mmap);
+
+ try {
+ // Test creating readers for new columns (should return default value readers)
+ try (PinotSegmentColumnReaderFactory factory = new PinotSegmentColumnReaderFactory(segment)) {
+ factory.init(_extendedSchema);
+
+ // Test creating reader for new column
+ ColumnReader newStringReader = factory.createColumnReader(NEW_STRING_COL,
+ _extendedSchema.getFieldSpecFor(NEW_STRING_COL));
+ Assert.assertEquals(newStringReader.getColumnName(), NEW_STRING_COL);
+
+ // Verify it returns default values using iterator pattern
+ Assert.assertTrue(newStringReader.hasNext());
+ Object defaultValue = newStringReader.next();
+ Assert.assertEquals(defaultValue, _extendedSchema.getFieldSpecFor(NEW_STRING_COL).getDefaultNullValue());
+
+ // Test that all values are the same (default)
+ int valueCount = 1; // We already read one value
+ while (newStringReader.hasNext()) {
+ Object value = newStringReader.next();
+ Assert.assertEquals(value, defaultValue);
+ valueCount++;
+ }
+ Assert.assertEquals(valueCount, _testData.size());
+ }
+ } finally {
+ segment.destroy();
+ }
+ }
+
+ @Test
+ public void testAllDataTypes() throws Exception {
+ // This test validates that all supported data types work correctly with columnar building
+ File rowMajorSegmentDir = createRowMajorSegment();
+ File columnarSegmentDir = createColumnarSegment(rowMajorSegmentDir);
+
+ // Validate that both segments have identical data for all data types
+ validateSegmentsIdentical(rowMajorSegmentDir, columnarSegmentDir);
+
+ // Additionally validate that all expected columns and data types are present
+ ImmutableSegment segment = ImmutableSegmentLoader.load(columnarSegmentDir, ReadMode.mmap);
+ try {
+ Set<String> columnNames = segment.getPhysicalColumnNames();
+
+ // Validate all data types are present
+ Assert.assertTrue(columnNames.contains(STRING_COL_1), "STRING column missing");
+ Assert.assertTrue(columnNames.contains(INT_COL_1), "INT column missing");
+ Assert.assertTrue(columnNames.contains(LONG_COL), "LONG column missing");
+ Assert.assertTrue(columnNames.contains(FLOAT_COL), "FLOAT column missing");
+ Assert.assertTrue(columnNames.contains(DOUBLE_COL), "DOUBLE column missing");
+ Assert.assertTrue(columnNames.contains(BIG_DECIMAL_COL), "BIG_DECIMAL column missing");
+ Assert.assertTrue(columnNames.contains(BYTES_COL), "BYTES column missing");
+ Assert.assertTrue(columnNames.contains(MV_INT_COL), "Multi-value INT column missing");
+ Assert.assertTrue(columnNames.contains(MV_STRING_COL), "Multi-value STRING column missing");
+ Assert.assertTrue(columnNames.contains(TIME_COL), "TIME column missing");
+ } finally {
+ segment.destroy();
+ }
+ }
+
+
+
+ private File createRowMajorSegment() throws Exception {
+ File outputDir = new File(_tempDir, "rowMajorSegment");
+ FileUtils.deleteQuietly(outputDir);
+ outputDir.mkdirs();
+
+ SegmentGeneratorConfig config = new SegmentGeneratorConfig(_tableConfig, _originalSchema);
+ config.setOutDir(outputDir.getAbsolutePath());
+ config.setSegmentName(SEGMENT_NAME + "_rowMajor");
+
+ SegmentIndexCreationDriverImpl driver = new SegmentIndexCreationDriverImpl();
+ TestRecordReader recordReader = new TestRecordReader(_testData);
+ driver.init(config, recordReader);
+ driver.build();
+
+ return new File(outputDir, SEGMENT_NAME + "_rowMajor");
+ }
+
+ private File createColumnarSegment(File sourceSegmentDir) throws Exception {
+ File outputDir = new File(_tempDir, "columnarSegment");
+ FileUtils.deleteQuietly(outputDir);
+ outputDir.mkdirs();
+
+ // Load the source segment
+ ImmutableSegment sourceSegment = ImmutableSegmentLoader.load(sourceSegmentDir, ReadMode.mmap);
+
+ try {
+ SegmentGeneratorConfig config = new SegmentGeneratorConfig(_tableConfig, _originalSchema);
+ config.setOutDir(outputDir.getAbsolutePath());
+ config.setSegmentName(SEGMENT_NAME + "_columnar");
+
+ SegmentIndexCreationDriverImpl driver = new SegmentIndexCreationDriverImpl();
+
+ // Use the new columnar building approach with config
+ try (PinotSegmentColumnReaderFactory factory = new PinotSegmentColumnReaderFactory(sourceSegment)) {
+ driver.init(config, factory);
+ driver.build();
+ }
+
+ return new File(outputDir, SEGMENT_NAME + "_columnar");
+ } finally {
+ sourceSegment.destroy();
+ }
+ }
+
+ private File createColumnarSegmentWithNewColumns(File sourceSegmentDir) throws Exception {
+ File outputDir = new File(_tempDir, "columnarSegmentWithNewColumns");
+ FileUtils.deleteQuietly(outputDir);
+ outputDir.mkdirs();
+
+ // Load the source segment
+ ImmutableSegment sourceSegment = ImmutableSegmentLoader.load(sourceSegmentDir, ReadMode.mmap);
+
+ try {
+ // Use extended schema with new columns
+ SegmentGeneratorConfig config = new SegmentGeneratorConfig(_tableConfig, _extendedSchema);
+ config.setOutDir(outputDir.getAbsolutePath());
+ config.setSegmentName(SEGMENT_NAME + "_withNewColumns");
+
+ SegmentIndexCreationDriverImpl driver = new SegmentIndexCreationDriverImpl();
+
+ // Use columnar building with extended schema
+ try (PinotSegmentColumnReaderFactory factory = new PinotSegmentColumnReaderFactory(sourceSegment)) {
+ driver.init(config, factory);
+ driver.build();
+ }
+
+ return new File(outputDir, SEGMENT_NAME + "_withNewColumns");
+ } finally {
+ sourceSegment.destroy();
+ }
+ }
+
+ private void validateSegmentsIdentical(File segment1Dir, File segment2Dir) throws Exception {
+ ImmutableSegment segment1 = ImmutableSegmentLoader.load(segment1Dir, ReadMode.mmap);
+ ImmutableSegment segment2 = ImmutableSegmentLoader.load(segment2Dir, ReadMode.mmap);
+
+ try {
+ // Validate metadata
+ Assert.assertEquals(segment1.getSegmentMetadata().getTotalDocs(), segment2.getSegmentMetadata().getTotalDocs());
+ Assert.assertEquals(segment1.getSegmentMetadata().getAllColumns(), segment2.getSegmentMetadata().getAllColumns());
+
+ // Validate data for each column
+ for (String columnName : segment1.getPhysicalColumnNames()) {
+ validateColumnData(segment1, segment2, columnName);
+ }
+ } finally {
+ segment1.destroy();
+ segment2.destroy();
+ }
+ }
+
+ private void validateSegmentWithNewColumns(File segmentDir) throws Exception {
+ ImmutableSegment segment = ImmutableSegmentLoader.load(segmentDir, ReadMode.mmap);
+
+ try {
+ // Validate that new columns exist
+ Assert.assertTrue(segment.getPhysicalColumnNames().contains(NEW_STRING_COL));
+ Assert.assertTrue(segment.getPhysicalColumnNames().contains(NEW_INT_COL));
+
+ // Validate that new columns have default values
+ GenericRow row = new GenericRow();
+ for (int docId = 0; docId < segment.getSegmentMetadata().getTotalDocs(); docId++) {
+ segment.getRecord(docId, row);
+
+ // Check default values
+ Assert.assertEquals(row.getValue(NEW_STRING_COL),
+ _extendedSchema.getFieldSpecFor(NEW_STRING_COL).getDefaultNullValue());
+ Assert.assertEquals(row.getValue(NEW_INT_COL),
+ _extendedSchema.getFieldSpecFor(NEW_INT_COL).getDefaultNullValue());
+ }
+ } finally {
+ segment.destroy();
+ }
+ }
+
+ private void validateColumnData(ImmutableSegment segment1, ImmutableSegment segment2, String columnName) {
+ int numDocs = segment1.getSegmentMetadata().getTotalDocs();
+
+ GenericRow row1 = new GenericRow();
+ GenericRow row2 = new GenericRow();
+
+ for (int docId = 0; docId < numDocs; docId++) {
+ segment1.getRecord(docId, row1);
+ segment2.getRecord(docId, row2);
+
+ Object value1 = row1.getValue(columnName);
+ Object value2 = row2.getValue(columnName);
+
+ Assert.assertEquals(value1, value2,
+ String.format("Column %s differs at docId %d: %s vs %s", columnName, docId, value1, value2));
+ }
+ }
+
+ private List<GenericRow> generateTestData(int numRows) {
+ List<GenericRow> data = new ArrayList<>();
+
+ for (int i = 0; i < numRows; i++) {
+ GenericRow row = new GenericRow();
+ row.putValue(STRING_COL_1, "string1_" + i);
+ row.putValue(STRING_COL_2, "string2_" + (i % 10));
+ row.putValue(INT_COL_1, i);
Review Comment:
sorting is not handled in columnar segment build process for mutable to mutable transformation. It is only done in realtime segment conversion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on PR #16727:
URL: https://github.com/apache/pinot/pull/16727#issuecomment-3345391935
@Jackie-Jiang, this is ready to review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "noob-se7en (via GitHub)" <gi...@apache.org>.
noob-se7en commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2414761599
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java:
##########
@@ -171,31 +170,72 @@ public void init(SegmentGeneratorConfig config, RecordReader recordReader)
new TransformPipeline(config.getTableConfig(), config.getSchema()));
}
+ /**
+ * Initialize the driver for columnar segment building using a ColumnReaderFactory.
+ * This method sets up the driver to use column-wise input data access instead of row-wise.
+ *
+ * @param config Segment generator configuration
+ * @param columnReaderFactory Factory for creating column readers
+ * @throws Exception if initialization fails
+ */
+ public void init(SegmentGeneratorConfig config, ColumnReaderFactory columnReaderFactory)
+ throws Exception {
+ // Initialize the column reader factory with target schema
+ columnReaderFactory.init(config.getSchema());
+
+ // Create all column readers for the target schema
+ Map<String, ColumnReader> columnReaders = columnReaderFactory.getAllColumnReaders();
+
+ // Create columnar data source
+ ColumnarSegmentCreationDataSource columnarDataSource =
+ new ColumnarSegmentCreationDataSource(columnReaderFactory, columnReaders);
+
+ // Use the existing init method with columnar data source and no transform pipeline
+ init(config, columnarDataSource, null);
+
+ LOGGER.info("Initialized SegmentIndexCreationDriverImpl for columnar building with {} columns",
Review Comment:
nit:
```suggestion
LOGGER.info("Initialized SegmentIndexCreationDriverImpl for columnar data source building with {} columns",
```
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReaderFactory.java:
##########
@@ -0,0 +1,92 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.spi.data.readers;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Map;
+import java.util.Set;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+
+
+/**
+ * Factory interface for creating ColumnReader instances for different data sources.
+ *
+ * <p>This factory provides a generic way to create column readers for various data sources
+ * such as Pinot segments, Parquet files, etc. The factory handles:
+ * <ul>
+ * <li>Creating column readers for existing columns</li>
+ * <li>Creating default value readers for new columns</li>
+ * <li>Data type conversions between source and target schemas</li>
+ * <li>Resource management</li>
+ * </ul>
+ */
+public interface ColumnReaderFactory extends Closeable, Serializable {
+
+ /**
+ * Initialize the factory with the data source and target schema.
+ *
+ * @param targetSchema Target schema for the output segment
+ * @throws IOException If initialization fails
+ */
+ void init(Schema targetSchema) throws IOException;
+
+ /**
+ * Get the set of column names available in the source data.
+ *
+ * @return Set of available column names in the source
+ */
+ Set<String> getAvailableColumns();
+
+ /**
+ * Get the total number of documents/rows in the data source.
+ *
+ * @return Total number of documents
+ */
+ int getNumDocs();
+
+ /**
+ * Create a column reader for the specified column.
+ *
+ * @param columnName Name of the column to read
+ * @param targetFieldSpec Target field specification from the output schema
+ * @return ColumnReader instance for the specified column
+ * @throws IOException If the column reader cannot be created
+ */
+ ColumnReader createColumnReader(String columnName, FieldSpec targetFieldSpec) throws IOException;
+
+ /**
+ * Get all column readers for the target schema.
+ * This is a convenience method that creates readers for all columns in the target schema.
+ *
+ * @return Map of column name to ColumnReader
+ * @throws IOException If any column reader cannot be created
+ */
+ Map<String, ColumnReader> getAllColumnReaders() throws IOException;
+
+ /**
+ * Check if the specified column exists in the source data.
+ *
+ * @param columnName Column name to check
+ * @return true if the column exists in source, false otherwise
+ */
+ boolean hasColumn(String columnName);
Review Comment:
nit: Does this need to be publicly exposed? Applicable for other methods like `getNumDocs()` as well.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java:
##########
@@ -677,4 +706,63 @@ public int getSkippedRowsFound() {
public int getSanitizedRowsFound() {
return _sanitizedRowsFound;
}
+
+ /**
+ * Build segment using columnar approach.
+ * This method builds the segment by processing data column-wise instead of row-wise.
+ *
+ * @throws Exception if segment building fails
+ */
+ public void buildColumnar() throws Exception {
+ if (!(_dataSource instanceof ColumnarSegmentCreationDataSource)) {
+ throw new IllegalStateException("buildColumnar() can only be called after initColumnar()");
+ }
+
+ ColumnarSegmentCreationDataSource columnarDataSource = (ColumnarSegmentCreationDataSource) _dataSource;
+ Map<String, ColumnReader> columnReaders = columnarDataSource.getColumnReaders();
+
+ LOGGER.info("Starting columnar segment building for {} columns", columnReaders.size());
+
+ // Reuse existing stats collection and index creation info logic
+ LOGGER.debug("Start building StatsCollector!");
+ collectStatsAndIndexCreationInfo();
+ LOGGER.info("Finished building StatsCollector!");
+ LOGGER.info("Collected stats for {} documents", _totalDocs);
+
+ if (_totalDocs == 0) {
+ LOGGER.warn("No documents found in data source");
+ handlePostCreation();
+ return;
+ }
+
+ try {
+ // Initialize the index creation using the per-column statistics information
+ _indexCreator.init(_config, _segmentIndexCreationInfo, _indexCreationInfoMap, _dataSchema, _tempIndexDir, null);
+
+ // Build the indexes column-wise (true column-major approach)
+ LOGGER.info("Start building Index using columnar approach");
+ long indexStartTime = System.nanoTime();
+
+ TreeSet<String> columns = _dataSchema.getPhysicalColumnNames();
+ for (String columnName : columns) {
+ LOGGER.debug("Indexing column: {}", columnName);
+ ColumnReader columnReader = columnReaders.get(columnName);
+ if (columnReader == null) {
+ throw new IllegalStateException("No column reader found for column: " + columnName);
+ }
+
+ // Index each column independently using true column-major approach
+ // This is similar to how buildByColumn works but uses ColumnReader instead of IndexSegment
+ ((SegmentColumnarIndexCreator) _indexCreator).indexColumn(columnName, columnReader);
Review Comment:
nit: reduntant cast?
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReaderFactory.java:
##########
@@ -0,0 +1,154 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.readers;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.ColumnReaderFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * ColumnReaderFactory implementation for immutable Pinot segments.
+ *
+ * <p>This factory creates ColumnReader instances for reading data from Pinot segments
+ * in a columnar fashion. It handles:
+ * <ul>
+ * <li>Creating readers for existing columns in the segment</li>
+ * <li>Creating default value readers for new columns</li>
+ * <li>Resource management for all created readers</li>
+ * </ul>
+ */
+public class PinotSegmentColumnReaderFactory implements ColumnReaderFactory {
+ private static final Logger LOGGER = LoggerFactory.getLogger(PinotSegmentColumnReaderFactory.class);
+
+ private final IndexSegment _indexSegment;
+ private Schema _targetSchema;
+ private final Map<String, ColumnReader> _columnReaders;
+
+ /**
+ * Create a PinotSegmentColumnReaderFactory.
+ *
+ * @param indexSegment Source segment to read from
+ */
+ public PinotSegmentColumnReaderFactory(IndexSegment indexSegment) {
+ _indexSegment = indexSegment;
+ _columnReaders = new HashMap<>();
+ }
+
+ @Override
+ public void init(Schema targetSchema) throws IOException {
+ _targetSchema = targetSchema;
+ LOGGER.info("Initialized PinotSegmentColumnReaderFactory with target schema containing {} columns",
+ targetSchema.getPhysicalColumnNames().size());
+ }
+
+ @Override
+ public Set<String> getAvailableColumns() {
+ return _indexSegment.getPhysicalColumnNames();
+ }
+
+ @Override
+ public int getNumDocs() {
+ return _indexSegment.getSegmentMetadata().getTotalDocs();
+ }
+
+ @Override
+ public ColumnReader createColumnReader(String columnName, FieldSpec targetFieldSpec) throws IOException {
Review Comment:
nit: 'IOException' is never thrown in the method
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.stats;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Stats container that efficiently collects statistics from columnar data using ColumnReader instances.
+ *
+ * <p>This implementation collects statistics by iterating column-wise instead of row-wise,
+ * which is more efficient for columnar data sources. It supports:
+ * <ul>
+ * <li>Column-wise statistics collection</li>
+ * <li>Existing columns from source data</li>
+ * <li>New columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ *
+ * <p>The statistics are collected using the same underlying collectors as the row-based approach
+ * (SegmentPreIndexStatsCollectorImpl) but with more efficient column-wise iteration.
+ */
+public class ColumnarSegmentPreIndexStatsContainer implements SegmentPreIndexStatsCollector {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentPreIndexStatsContainer.class);
+
+ private final Map<String, ColumnReader> _columnReaders;
+ private final StatsCollectorConfig _statsCollectorConfig;
+ private final Schema _targetSchema;
+ private final Map<String, AbstractColumnStatisticsCollector> _columnStatsCollectorMap;
+ private int _totalDocCount;
+
+ /**
+ * Create a ColumnarSegmentPreIndexStatsContainer.
+ *
+ * @param columnReaders Map of column name to ColumnReader instances
+ * @param statsCollectorConfig Configuration for statistics collection
+ */
+ public ColumnarSegmentPreIndexStatsContainer(Map<String, ColumnReader> columnReaders,
+ StatsCollectorConfig statsCollectorConfig) {
+ _columnReaders = columnReaders;
+ _statsCollectorConfig = statsCollectorConfig;
+ _targetSchema = statsCollectorConfig.getSchema();
+ _columnStatsCollectorMap = new HashMap<>();
Review Comment:
nit:
```suggestion
_columnStatsCollectorMap = new HashMap<>(columnReaders.size());
```
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.stats;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Stats container that efficiently collects statistics from columnar data using ColumnReader instances.
+ *
+ * <p>This implementation collects statistics by iterating column-wise instead of row-wise,
+ * which is more efficient for columnar data sources. It supports:
+ * <ul>
+ * <li>Column-wise statistics collection</li>
+ * <li>Existing columns from source data</li>
+ * <li>New columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ *
+ * <p>The statistics are collected using the same underlying collectors as the row-based approach
+ * (SegmentPreIndexStatsCollectorImpl) but with more efficient column-wise iteration.
+ */
+public class ColumnarSegmentPreIndexStatsContainer implements SegmentPreIndexStatsCollector {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentPreIndexStatsContainer.class);
+
+ private final Map<String, ColumnReader> _columnReaders;
+ private final StatsCollectorConfig _statsCollectorConfig;
+ private final Schema _targetSchema;
+ private final Map<String, AbstractColumnStatisticsCollector> _columnStatsCollectorMap;
+ private int _totalDocCount;
+
+ /**
+ * Create a ColumnarSegmentPreIndexStatsContainer.
+ *
+ * @param columnReaders Map of column name to ColumnReader instances
+ * @param statsCollectorConfig Configuration for statistics collection
+ */
+ public ColumnarSegmentPreIndexStatsContainer(Map<String, ColumnReader> columnReaders,
+ StatsCollectorConfig statsCollectorConfig) {
+ _columnReaders = columnReaders;
+ _statsCollectorConfig = statsCollectorConfig;
+ _targetSchema = statsCollectorConfig.getSchema();
+ _columnStatsCollectorMap = new HashMap<>();
+ _totalDocCount = -1; // indicates unset
+
+ initializeStatsCollectors();
+ collectColumnStats();
+ }
+
+ /**
+ * Initialize stats collectors for all columns in the target schema.
+ */
+ private void initializeStatsCollectors() {
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
+ switch (fieldSpec.getDataType().getStoredType()) {
+ case INT:
+ _columnStatsCollectorMap.put(columnName,
+ new IntColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case LONG:
+ _columnStatsCollectorMap.put(columnName,
+ new LongColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case FLOAT:
+ _columnStatsCollectorMap.put(columnName,
+ new FloatColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case DOUBLE:
+ _columnStatsCollectorMap.put(columnName,
+ new DoubleColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BIG_DECIMAL:
+ _columnStatsCollectorMap.put(columnName,
+ new BigDecimalColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case STRING:
+ _columnStatsCollectorMap.put(columnName,
+ new StringColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BYTES:
+ _columnStatsCollectorMap.put(columnName,
+ new BytesColumnPredIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case MAP:
+ _columnStatsCollectorMap.put(columnName,
+ new MapColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ default:
+ throw new IllegalStateException("Unsupported data type: " + fieldSpec.getDataType());
+ }
+ }
+ }
+
+ /**
+ * Collect stats by iterating column-wise using the provided ColumnReader instances.
+ */
+ private void collectColumnStats() {
+ LOGGER.info("Collecting stats for {} columns using column-wise iteration", _columnReaders.size());
+
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
+ AbstractColumnStatisticsCollector statsCollector = _columnStatsCollectorMap.get(columnName);
+ ColumnReader columnReader = _columnReaders.get(columnName);
+
+ if (columnReader == null) {
+ throw new RuntimeException("Column reader for column " + columnName + " not found");
+ }
+
+ LOGGER.debug("Collecting stats for column: {}", columnName);
+ collectStatsFromColumnReader(columnName, columnReader, statsCollector);
+
+ // Seal the stats collector
+ statsCollector.seal();
+ }
+ }
+
+ /**
+ * Collect stats from a column reader by iterating over all values using the iterator pattern.
+ */
+ private void collectStatsFromColumnReader(String columnName, ColumnReader columnReader,
+ AbstractColumnStatisticsCollector statsCollector) {
Review Comment:
nit: lint fix
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.stats;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Stats container that efficiently collects statistics from columnar data using ColumnReader instances.
+ *
+ * <p>This implementation collects statistics by iterating column-wise instead of row-wise,
+ * which is more efficient for columnar data sources. It supports:
+ * <ul>
+ * <li>Column-wise statistics collection</li>
+ * <li>Existing columns from source data</li>
+ * <li>New columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ *
+ * <p>The statistics are collected using the same underlying collectors as the row-based approach
+ * (SegmentPreIndexStatsCollectorImpl) but with more efficient column-wise iteration.
+ */
+public class ColumnarSegmentPreIndexStatsContainer implements SegmentPreIndexStatsCollector {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentPreIndexStatsContainer.class);
+
+ private final Map<String, ColumnReader> _columnReaders;
+ private final StatsCollectorConfig _statsCollectorConfig;
+ private final Schema _targetSchema;
+ private final Map<String, AbstractColumnStatisticsCollector> _columnStatsCollectorMap;
+ private int _totalDocCount;
+
+ /**
+ * Create a ColumnarSegmentPreIndexStatsContainer.
+ *
+ * @param columnReaders Map of column name to ColumnReader instances
+ * @param statsCollectorConfig Configuration for statistics collection
+ */
+ public ColumnarSegmentPreIndexStatsContainer(Map<String, ColumnReader> columnReaders,
+ StatsCollectorConfig statsCollectorConfig) {
+ _columnReaders = columnReaders;
+ _statsCollectorConfig = statsCollectorConfig;
+ _targetSchema = statsCollectorConfig.getSchema();
+ _columnStatsCollectorMap = new HashMap<>();
+ _totalDocCount = -1; // indicates unset
+
+ initializeStatsCollectors();
+ collectColumnStats();
+ }
+
+ /**
+ * Initialize stats collectors for all columns in the target schema.
+ */
+ private void initializeStatsCollectors() {
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
+ switch (fieldSpec.getDataType().getStoredType()) {
+ case INT:
+ _columnStatsCollectorMap.put(columnName,
+ new IntColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case LONG:
+ _columnStatsCollectorMap.put(columnName,
+ new LongColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case FLOAT:
+ _columnStatsCollectorMap.put(columnName,
+ new FloatColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case DOUBLE:
+ _columnStatsCollectorMap.put(columnName,
+ new DoubleColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BIG_DECIMAL:
+ _columnStatsCollectorMap.put(columnName,
+ new BigDecimalColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case STRING:
+ _columnStatsCollectorMap.put(columnName,
+ new StringColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BYTES:
+ _columnStatsCollectorMap.put(columnName,
+ new BytesColumnPredIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case MAP:
+ _columnStatsCollectorMap.put(columnName,
+ new MapColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ default:
+ throw new IllegalStateException("Unsupported data type: " + fieldSpec.getDataType());
+ }
+ }
+ }
+
+ /**
+ * Collect stats by iterating column-wise using the provided ColumnReader instances.
+ */
+ private void collectColumnStats() {
+ LOGGER.info("Collecting stats for {} columns using column-wise iteration", _columnReaders.size());
+
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
Review Comment:
We can iterate on just `_columnStatsCollectorMap` to avoid this
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java:
##########
@@ -677,4 +706,63 @@ public int getSkippedRowsFound() {
public int getSanitizedRowsFound() {
return _sanitizedRowsFound;
}
+
+ /**
+ * Build segment using columnar approach.
+ * This method builds the segment by processing data column-wise instead of row-wise.
+ *
+ * @throws Exception if segment building fails
+ */
+ public void buildColumnar() throws Exception {
Review Comment:
nit: should be public?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2436922482
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java:
##########
@@ -171,31 +170,72 @@ public void init(SegmentGeneratorConfig config, RecordReader recordReader)
new TransformPipeline(config.getTableConfig(), config.getSchema()));
}
+ /**
+ * Initialize the driver for columnar segment building using a ColumnReaderFactory.
+ * This method sets up the driver to use column-wise input data access instead of row-wise.
Review Comment:
added code comments in the buildColumnar method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] build segment columnarly from a column major datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2318111183
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java:
##########
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.stats;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Stats container that efficiently collects statistics from columnar data using ColumnReader instances.
+ *
+ * <p>This implementation collects statistics by iterating column-wise instead of row-wise,
+ * which is more efficient for columnar data sources. It supports:
+ * <ul>
+ * <li>Column-wise statistics collection</li>
+ * <li>Existing columns from source data</li>
+ * <li>New columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ *
+ * <p>The statistics are collected using the same underlying collectors as the row-based approach
+ * (SegmentPreIndexStatsCollectorImpl) but with more efficient column-wise iteration.
+ */
+public class ColumnarSegmentPreIndexStatsContainer implements SegmentPreIndexStatsCollector {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentPreIndexStatsContainer.class);
+
+ private final Map<String, ColumnReader> _columnReaders;
+ private final StatsCollectorConfig _statsCollectorConfig;
+ private final Schema _targetSchema;
+ private final Map<String, AbstractColumnStatisticsCollector> _columnStatsCollectorMap;
+ private int _totalDocCount;
+
+ /**
+ * Create a ColumnarSegmentPreIndexStatsContainer.
+ *
+ * @param columnReaders Map of column name to ColumnReader instances
+ * @param statsCollectorConfig Configuration for statistics collection
+ */
+ public ColumnarSegmentPreIndexStatsContainer(Map<String, ColumnReader> columnReaders,
+ StatsCollectorConfig statsCollectorConfig) {
+ _columnReaders = columnReaders;
+ _statsCollectorConfig = statsCollectorConfig;
+ _targetSchema = statsCollectorConfig.getSchema();
+ _columnStatsCollectorMap = new HashMap<>();
+ _totalDocCount = 0;
Review Comment:
self review: initialise with -1 to identify uninitialized scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2408955927
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/ColumnarSegmentCreationDataSource.java:
##########
@@ -0,0 +1,96 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator;
+
+import java.util.Map;
+import org.apache.pinot.segment.local.segment.creator.impl.stats.ColumnarSegmentPreIndexStatsContainer;
+import org.apache.pinot.segment.spi.creator.SegmentCreationDataSource;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.ColumnReaderFactory;
+import org.apache.pinot.spi.data.readers.RecordReader;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * SegmentCreationDataSource implementation that uses ColumnReaderFactory for columnar data access.
+ *
+ * <p>This data source enables columnar segment building by providing access to column readers
+ * instead of row-based record readers. It supports:
+ * <ul>
+ * <li>Columnar statistics collection</li>
+ * <li>Column-wise data access through ColumnReaderFactory</li>
+ * <li>Efficient handling of new columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ */
+public class ColumnarSegmentCreationDataSource implements SegmentCreationDataSource {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentCreationDataSource.class);
+
+ private final ColumnReaderFactory _columnReaderFactory;
+ private final Map<String, ColumnReader> _columnReaders;
+
+ /**
+ * Create a ColumnarSegmentCreationDataSource.
+ *
+ * @param columnReaderFactory Factory for creating column readers
+ * @param columnReaders Map of column name to ColumnReader instances
+ */
+ public ColumnarSegmentCreationDataSource(ColumnReaderFactory columnReaderFactory,
+ Map<String, ColumnReader> columnReaders) {
Review Comment:
(code format) Can you reformat all the changes with [Pinot Style](https://docs.pinot.apache.org/developers/developers-and-contributors/code-setup#set-up-ide)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "krishan1390 (via GitHub)" <gi...@apache.org>.
krishan1390 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2447695595
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.stats;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Stats container that efficiently collects statistics from columnar data using ColumnReader instances.
+ *
+ * <p>This implementation collects statistics by iterating column-wise instead of row-wise,
+ * which is more efficient for columnar data sources. It supports:
+ * <ul>
+ * <li>Column-wise statistics collection</li>
+ * <li>Existing columns from source data</li>
+ * <li>New columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ *
+ * <p>The statistics are collected using the same underlying collectors as the row-based approach
+ * (SegmentPreIndexStatsCollectorImpl) but with more efficient column-wise iteration.
+ */
+public class ColumnarSegmentPreIndexStatsContainer implements SegmentPreIndexStatsCollector {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentPreIndexStatsContainer.class);
+
+ private final Map<String, ColumnReader> _columnReaders;
+ private final StatsCollectorConfig _statsCollectorConfig;
+ private final Schema _targetSchema;
+ private final Map<String, AbstractColumnStatisticsCollector> _columnStatsCollectorMap;
+ private int _totalDocCount;
+
+ /**
+ * Create a ColumnarSegmentPreIndexStatsContainer.
+ *
+ * @param columnReaders Map of column name to ColumnReader instances
+ * @param statsCollectorConfig Configuration for statistics collection
+ */
+ public ColumnarSegmentPreIndexStatsContainer(Map<String, ColumnReader> columnReaders,
+ StatsCollectorConfig statsCollectorConfig) {
+ _columnReaders = columnReaders;
+ _statsCollectorConfig = statsCollectorConfig;
+ _targetSchema = statsCollectorConfig.getSchema();
+ _columnStatsCollectorMap = new HashMap<>();
+ _totalDocCount = -1; // indicates unset
+
+ initializeStatsCollectors();
+ collectColumnStats();
+ }
+
+ /**
+ * Initialize stats collectors for all columns in the target schema.
+ */
+ private void initializeStatsCollectors() {
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
+ switch (fieldSpec.getDataType().getStoredType()) {
+ case INT:
+ _columnStatsCollectorMap.put(columnName,
+ new IntColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case LONG:
+ _columnStatsCollectorMap.put(columnName,
+ new LongColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case FLOAT:
+ _columnStatsCollectorMap.put(columnName,
+ new FloatColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case DOUBLE:
+ _columnStatsCollectorMap.put(columnName,
+ new DoubleColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BIG_DECIMAL:
+ _columnStatsCollectorMap.put(columnName,
+ new BigDecimalColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case STRING:
+ _columnStatsCollectorMap.put(columnName,
+ new StringColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BYTES:
+ _columnStatsCollectorMap.put(columnName,
+ new BytesColumnPredIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case MAP:
+ _columnStatsCollectorMap.put(columnName,
+ new MapColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ default:
+ throw new IllegalStateException("Unsupported data type: " + fieldSpec.getDataType());
+ }
+ }
+ }
+
+ /**
+ * Collect stats by iterating column-wise using the provided ColumnReader instances.
+ */
+ private void collectColumnStats() {
+ LOGGER.info("Collecting stats for {} columns using column-wise iteration", _columnReaders.size());
+
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
+ AbstractColumnStatisticsCollector statsCollector = _columnStatsCollectorMap.get(columnName);
+ ColumnReader columnReader = _columnReaders.get(columnName);
+
+ if (columnReader == null) {
+ throw new RuntimeException("Column reader for column " + columnName + " not found");
+ }
+
+ LOGGER.debug("Collecting stats for column: {}", columnName);
+ collectStatsFromColumnReader(columnName, columnReader, statsCollector);
+
+ // Seal the stats collector
+ statsCollector.seal();
+ }
+ }
+
+ /**
+ * Collect stats from a column reader by iterating over all values using the iterator pattern.
+ */
+ private void collectStatsFromColumnReader(String columnName, ColumnReader columnReader,
+ AbstractColumnStatisticsCollector statsCollector) {
+ try {
+ // Reset the column reader to start from the beginning
+ columnReader.rewind();
+
+ int docCount = 0;
+ while (columnReader.hasNext()) {
+ Object value = columnReader.next();
+ statsCollector.collect(value);
+ docCount++;
+ }
+
+ // if totalDocCount is unset then set total doc count from the first column
+ if (_totalDocCount == -1) {
+ _totalDocCount = docCount;
+ } else if (_totalDocCount != docCount) {
+ // all columns should have same count
+ LOGGER.warn("Column {} has {} documents, but expected {} documents",
+ columnName, docCount, _totalDocCount);
+ }
+ } catch (IOException e) {
+ LOGGER.error("Failed to collect stats for column: {}", columnName, e);
+ throw new RuntimeException("Failed to collect stats for column: " + columnName, e);
+ }
+ }
+
+ @Override
+ public ColumnStatistics getColumnProfileFor(String column) {
+ return _columnStatsCollectorMap.get(column);
+ }
+
+ @Override
+ public int getTotalDocCount() {
+ return _totalDocCount;
+ }
+
+ @Override
+ public void init() {
+ // Already initialized in constructor
+ }
+
+ @Override
+ public void build() {
+ // Stats are already collected in constructor
+ }
+
+ @Override
+ public void logStats() {
+ LOGGER.info("Columnar segment stats collection completed for {} columns with {} documents",
Review Comment:
@rohityadav1993 this needs to be done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "krishan1390 (via GitHub)" <gi...@apache.org>.
krishan1390 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2447692881
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.stats;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Stats container that efficiently collects statistics from columnar data using ColumnReader instances.
+ *
+ * <p>This implementation collects statistics by iterating column-wise instead of row-wise,
+ * which is more efficient for columnar data sources. It supports:
+ * <ul>
+ * <li>Column-wise statistics collection</li>
+ * <li>Existing columns from source data</li>
+ * <li>New columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ *
+ * <p>The statistics are collected using the same underlying collectors as the row-based approach
+ * (SegmentPreIndexStatsCollectorImpl) but with more efficient column-wise iteration.
+ */
+public class ColumnarSegmentPreIndexStatsContainer implements SegmentPreIndexStatsCollector {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentPreIndexStatsContainer.class);
+
+ private final Map<String, ColumnReader> _columnReaders;
+ private final StatsCollectorConfig _statsCollectorConfig;
+ private final Schema _targetSchema;
+ private final Map<String, AbstractColumnStatisticsCollector> _columnStatsCollectorMap;
+ private int _totalDocCount;
+
+ /**
+ * Create a ColumnarSegmentPreIndexStatsContainer.
+ *
+ * @param columnReaders Map of column name to ColumnReader instances
+ * @param statsCollectorConfig Configuration for statistics collection
+ */
+ public ColumnarSegmentPreIndexStatsContainer(Map<String, ColumnReader> columnReaders,
+ StatsCollectorConfig statsCollectorConfig) {
+ _columnReaders = columnReaders;
+ _statsCollectorConfig = statsCollectorConfig;
+ _targetSchema = statsCollectorConfig.getSchema();
+ _columnStatsCollectorMap = new HashMap<>();
+ _totalDocCount = -1; // indicates unset
+
+ initializeStatsCollectors();
Review Comment:
Its cleaner and more easier to maintain to keep constructor light and have the methods do what they're supposed to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2366373366
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReader.java:
##########
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.spi.data.readers;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.Serializable;
+import javax.annotation.Nullable;
+
+
+/**
+ * The <code>ColumnReader</code> interface is used to read column data from various data sources
+ * for columnar segment building. Unlike RecordReader which reads row-by-row, ColumnReader provides
+ * column-wise access to data, enabling efficient columnar segment creation.
+ *
+ * <p>This interface follows an iterator pattern similar to RecordReader:
+ * <ul>
+ * <li>Sequential iteration over all values in a column using hasNext() and next()</li>
+ * <li>Null value detection for the current value</li>
+ * <li>Rewind capability to restart iteration</li>
+ * <li>Resource cleanup</li>
+ * </ul>
+ *
+ * <p>Implementations should handle data type conversions, default values for new columns,
+ * and efficient column-wise data access patterns.
+ */
+public interface ColumnReader extends Closeable, Serializable {
Review Comment:
The interface looks good, but one will need to create a `ParquetSegmentColumnReaderFactory` similar to `PinotSegmentColumnReaderFactory` which should be able to correctly initialise a `ParquetSegmentColumnReaderImpl` for a given parquet file to be read.
On a high level Parquet files are analogous to a Pinot segment as collection of columns and there is a similar low level columnar reader: https://github.com/apache/parquet-java/blob/f50dd6cb4b526cf4b585993c1b69a838cd8151f3/parquet-column/src/main/java/org/apache/parquet/column/ColumnReader.java#L40
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2431419671
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReaderFactory.java:
##########
@@ -0,0 +1,154 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.readers;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.ColumnReaderFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * ColumnReaderFactory implementation for immutable Pinot segments.
+ *
+ * <p>This factory creates ColumnReader instances for reading data from Pinot segments
+ * in a columnar fashion. It handles:
+ * <ul>
+ * <li>Creating readers for existing columns in the segment</li>
+ * <li>Creating default value readers for new columns</li>
+ * <li>Resource management for all created readers</li>
+ * </ul>
+ */
+public class PinotSegmentColumnReaderFactory implements ColumnReaderFactory {
Review Comment:
init is taking schema as it is trying to standardise how factory should be initialised. One is free to use any constructor signature needed. e.g. if data source is segment, the constructor is `PinotSegmentColumnReaderFactory(IndexSegment indexSegment)` and in case it was parquet, it can be `ParquetColumnReaderFactory(File parquetFile)`
renaming `getAllColumnReaders()` to `createAllColumnReaders()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "krishan1390 (via GitHub)" <gi...@apache.org>.
krishan1390 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2431726494
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.stats;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Stats container that efficiently collects statistics from columnar data using ColumnReader instances.
+ *
+ * <p>This implementation collects statistics by iterating column-wise instead of row-wise,
+ * which is more efficient for columnar data sources. It supports:
+ * <ul>
+ * <li>Column-wise statistics collection</li>
+ * <li>Existing columns from source data</li>
+ * <li>New columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ *
+ * <p>The statistics are collected using the same underlying collectors as the row-based approach
+ * (SegmentPreIndexStatsCollectorImpl) but with more efficient column-wise iteration.
+ */
+public class ColumnarSegmentPreIndexStatsContainer implements SegmentPreIndexStatsCollector {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentPreIndexStatsContainer.class);
+
+ private final Map<String, ColumnReader> _columnReaders;
+ private final StatsCollectorConfig _statsCollectorConfig;
+ private final Schema _targetSchema;
+ private final Map<String, AbstractColumnStatisticsCollector> _columnStatsCollectorMap;
+ private int _totalDocCount;
+
+ /**
+ * Create a ColumnarSegmentPreIndexStatsContainer.
+ *
+ * @param columnReaders Map of column name to ColumnReader instances
+ * @param statsCollectorConfig Configuration for statistics collection
+ */
+ public ColumnarSegmentPreIndexStatsContainer(Map<String, ColumnReader> columnReaders,
+ StatsCollectorConfig statsCollectorConfig) {
+ _columnReaders = columnReaders;
+ _statsCollectorConfig = statsCollectorConfig;
+ _targetSchema = statsCollectorConfig.getSchema();
+ _columnStatsCollectorMap = new HashMap<>();
+ _totalDocCount = -1; // indicates unset
+
+ initializeStatsCollectors();
+ collectColumnStats();
+ }
+
+ /**
+ * Initialize stats collectors for all columns in the target schema.
+ */
+ private void initializeStatsCollectors() {
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
Review Comment:
yes. but avoiding redundant code will be helpful to maintain the code until it exists and keep them consistent for common logic. eg: NoDictColumnStatisticsCollector was missing in this class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2431885542
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java:
##########
@@ -209,7 +249,7 @@ public void init(SegmentGeneratorConfig config, SegmentCreationDataSource dataSo
}
_ingestionSchemaValidator =
- SchemaValidatorFactory.getSchemaValidator(_dataSchema, _recordReader.getClass().getName(),
+ SchemaValidatorFactory.getSchemaValidator(_dataSchema, readerClassName,
Review Comment:
I found no use of `getIngestionSchemaValidator()` outside test classes. There is no change being made in `getSchemaValidator()` which is creating a SchemaValidator
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2430146924
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.stats;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Stats container that efficiently collects statistics from columnar data using ColumnReader instances.
+ *
+ * <p>This implementation collects statistics by iterating column-wise instead of row-wise,
+ * which is more efficient for columnar data sources. It supports:
+ * <ul>
+ * <li>Column-wise statistics collection</li>
+ * <li>Existing columns from source data</li>
+ * <li>New columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ *
+ * <p>The statistics are collected using the same underlying collectors as the row-based approach
+ * (SegmentPreIndexStatsCollectorImpl) but with more efficient column-wise iteration.
+ */
+public class ColumnarSegmentPreIndexStatsContainer implements SegmentPreIndexStatsCollector {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentPreIndexStatsContainer.class);
+
+ private final Map<String, ColumnReader> _columnReaders;
+ private final StatsCollectorConfig _statsCollectorConfig;
+ private final Schema _targetSchema;
+ private final Map<String, AbstractColumnStatisticsCollector> _columnStatsCollectorMap;
+ private int _totalDocCount;
+
+ /**
+ * Create a ColumnarSegmentPreIndexStatsContainer.
+ *
+ * @param columnReaders Map of column name to ColumnReader instances
+ * @param statsCollectorConfig Configuration for statistics collection
+ */
+ public ColumnarSegmentPreIndexStatsContainer(Map<String, ColumnReader> columnReaders,
+ StatsCollectorConfig statsCollectorConfig) {
+ _columnReaders = columnReaders;
+ _statsCollectorConfig = statsCollectorConfig;
+ _targetSchema = statsCollectorConfig.getSchema();
+ _columnStatsCollectorMap = new HashMap<>();
+ _totalDocCount = -1; // indicates unset
+
+ initializeStatsCollectors();
+ collectColumnStats();
+ }
+
+ /**
+ * Initialize stats collectors for all columns in the target schema.
+ */
+ private void initializeStatsCollectors() {
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
Review Comment:
virtaulColumns are generated at the time of segment load. These are not handled in segment build operation as it isn't present in the physical datasource
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.stats;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Stats container that efficiently collects statistics from columnar data using ColumnReader instances.
+ *
+ * <p>This implementation collects statistics by iterating column-wise instead of row-wise,
+ * which is more efficient for columnar data sources. It supports:
+ * <ul>
+ * <li>Column-wise statistics collection</li>
+ * <li>Existing columns from source data</li>
+ * <li>New columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ *
+ * <p>The statistics are collected using the same underlying collectors as the row-based approach
+ * (SegmentPreIndexStatsCollectorImpl) but with more efficient column-wise iteration.
+ */
+public class ColumnarSegmentPreIndexStatsContainer implements SegmentPreIndexStatsCollector {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentPreIndexStatsContainer.class);
+
+ private final Map<String, ColumnReader> _columnReaders;
+ private final StatsCollectorConfig _statsCollectorConfig;
+ private final Schema _targetSchema;
+ private final Map<String, AbstractColumnStatisticsCollector> _columnStatsCollectorMap;
+ private int _totalDocCount;
+
+ /**
+ * Create a ColumnarSegmentPreIndexStatsContainer.
+ *
+ * @param columnReaders Map of column name to ColumnReader instances
+ * @param statsCollectorConfig Configuration for statistics collection
+ */
+ public ColumnarSegmentPreIndexStatsContainer(Map<String, ColumnReader> columnReaders,
+ StatsCollectorConfig statsCollectorConfig) {
+ _columnReaders = columnReaders;
+ _statsCollectorConfig = statsCollectorConfig;
+ _targetSchema = statsCollectorConfig.getSchema();
+ _columnStatsCollectorMap = new HashMap<>();
+ _totalDocCount = -1; // indicates unset
+
+ initializeStatsCollectors();
+ collectColumnStats();
+ }
+
+ /**
+ * Initialize stats collectors for all columns in the target schema.
+ */
+ private void initializeStatsCollectors() {
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
Review Comment:
SegmentPreIndexStatsCollectorImpl is heavily assumes row major iteration using method `collectRow()`
We are now using `buildByColumn` as default for mutable segment to immutable segment conversion and this is a new addition `buildColumnar` for immutable datasource to immutableSegment conversion in column major order so the above should get phased out after we are completely columnar with support for transformations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2468621883
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.stats;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Stats container that efficiently collects statistics from columnar data using ColumnReader instances.
+ *
+ * <p>This implementation collects statistics by iterating column-wise instead of row-wise,
+ * which is more efficient for columnar data sources. It supports:
+ * <ul>
+ * <li>Column-wise statistics collection</li>
+ * <li>Existing columns from source data</li>
+ * <li>New columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ *
+ * <p>The statistics are collected using the same underlying collectors as the row-based approach
+ * (SegmentPreIndexStatsCollectorImpl) but with more efficient column-wise iteration.
+ */
+public class ColumnarSegmentPreIndexStatsContainer implements SegmentPreIndexStatsCollector {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentPreIndexStatsContainer.class);
+
+ private final Map<String, ColumnReader> _columnReaders;
+ private final StatsCollectorConfig _statsCollectorConfig;
+ private final Schema _targetSchema;
+ private final Map<String, AbstractColumnStatisticsCollector> _columnStatsCollectorMap;
+ private int _totalDocCount;
+
+ /**
+ * Create a ColumnarSegmentPreIndexStatsContainer.
+ *
+ * @param columnReaders Map of column name to ColumnReader instances
+ * @param statsCollectorConfig Configuration for statistics collection
+ */
+ public ColumnarSegmentPreIndexStatsContainer(Map<String, ColumnReader> columnReaders,
+ StatsCollectorConfig statsCollectorConfig) {
+ _columnReaders = columnReaders;
+ _statsCollectorConfig = statsCollectorConfig;
+ _targetSchema = statsCollectorConfig.getSchema();
+ _columnStatsCollectorMap = new HashMap<>();
+ _totalDocCount = -1; // indicates unset
+
+ initializeStatsCollectors();
+ collectColumnStats();
+ }
+
+ /**
+ * Initialize stats collectors for all columns in the target schema.
+ */
+ private void initializeStatsCollectors() {
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
Review Comment:
Aligned on reducing redundancy, extracting the common logic in a class called StatsCollectorUtils.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2355107792
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReaderFactory.java:
##########
@@ -0,0 +1,151 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.readers;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.ColumnReaderFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * ColumnReaderFactory implementation for immutable Pinot segments.
+ *
+ * <p>This factory creates ColumnReader instances for reading data from Pinot segments
+ * in a columnar fashion. It handles:
+ * <ul>
+ * <li>Creating readers for existing columns in the segment</li>
+ * <li>Creating default value readers for new columns</li>
+ * <li>Data type conversions between source and target schemas</li>
+ * <li>Resource management for all created readers</li>
+ * </ul>
+ */
+public class PinotSegmentColumnReaderFactory implements ColumnReaderFactory {
+ private static final Logger LOGGER = LoggerFactory.getLogger(PinotSegmentColumnReaderFactory.class);
+
+ private final IndexSegment _indexSegment;
+ private Schema _targetSchema;
+ private final Map<String, ColumnReader> _columnReaders;
+
+ /**
+ * Create a PinotSegmentColumnReaderFactory.
+ *
+ * @param indexSegment Source segment to read from
+ */
+ public PinotSegmentColumnReaderFactory(IndexSegment indexSegment) {
+ _indexSegment = indexSegment;
+ _columnReaders = new HashMap<>();
+ }
+
+ @Override
+ public void init(Schema targetSchema) throws IOException {
+ _targetSchema = targetSchema;
+ LOGGER.info("Initialized PinotSegmentColumnReaderFactory with target schema containing {} columns",
+ targetSchema.getPhysicalColumnNames().size());
+ }
+
+ @Override
+ public Set<String> getAvailableColumns() {
+ return _indexSegment.getPhysicalColumnNames();
+ }
+
+ @Override
+ public int getNumDocs() {
+ return _indexSegment.getSegmentMetadata().getTotalDocs();
+ }
+
+ @Override
+ public ColumnReader createColumnReader(String columnName, FieldSpec targetFieldSpec) throws IOException {
+ if (_targetSchema == null) {
+ throw new IllegalStateException("Factory not initialized. Call init() first.");
+ }
+
+ // Check if we already have a reader for this column
+ ColumnReader existingReader = _columnReaders.get(columnName);
+ if (existingReader != null) {
+ return existingReader;
+ }
+
+ ColumnReader columnReader;
+
+ if (hasColumn(columnName)) {
Review Comment:
makes sense, adding an error scenario if invoked for virtual columns
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] build segment columnarly from a column major datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2318104949
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java:
##########
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.stats;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Stats container that efficiently collects statistics from columnar data using ColumnReader instances.
+ *
+ * <p>This implementation collects statistics by iterating column-wise instead of row-wise,
+ * which is more efficient for columnar data sources. It supports:
+ * <ul>
+ * <li>Column-wise statistics collection</li>
+ * <li>Existing columns from source data</li>
+ * <li>New columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ *
+ * <p>The statistics are collected using the same underlying collectors as the row-based approach
+ * (SegmentPreIndexStatsCollectorImpl) but with more efficient column-wise iteration.
+ */
+public class ColumnarSegmentPreIndexStatsContainer implements SegmentPreIndexStatsCollector {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentPreIndexStatsContainer.class);
+
+ private final Map<String, ColumnReader> _columnReaders;
+ private final StatsCollectorConfig _statsCollectorConfig;
+ private final Schema _targetSchema;
+ private final Map<String, AbstractColumnStatisticsCollector> _columnStatsCollectorMap;
+ private int _totalDocCount;
+
+ /**
+ * Create a ColumnarSegmentPreIndexStatsContainer.
+ *
+ * @param columnReaders Map of column name to ColumnReader instances
+ * @param statsCollectorConfig Configuration for statistics collection
+ */
+ public ColumnarSegmentPreIndexStatsContainer(Map<String, ColumnReader> columnReaders,
+ StatsCollectorConfig statsCollectorConfig) {
+ _columnReaders = columnReaders;
+ _statsCollectorConfig = statsCollectorConfig;
+ _targetSchema = statsCollectorConfig.getSchema();
+ _columnStatsCollectorMap = new HashMap<>();
+ _totalDocCount = 0;
+
+ initializeStatsCollectors();
+ collectColumnStats();
+ }
+
+ /**
+ * Initialize stats collectors for all columns in the target schema.
+ */
+ private void initializeStatsCollectors() {
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
+ switch (fieldSpec.getDataType().getStoredType()) {
+ case INT:
+ _columnStatsCollectorMap.put(columnName,
+ new IntColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case LONG:
+ _columnStatsCollectorMap.put(columnName,
+ new LongColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case FLOAT:
+ _columnStatsCollectorMap.put(columnName,
+ new FloatColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case DOUBLE:
+ _columnStatsCollectorMap.put(columnName,
+ new DoubleColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BIG_DECIMAL:
+ _columnStatsCollectorMap.put(columnName,
+ new BigDecimalColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case STRING:
+ _columnStatsCollectorMap.put(columnName,
+ new StringColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BYTES:
Review Comment:
Self review: handle map data type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2366358075
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReader.java:
##########
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.spi.data.readers;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.Serializable;
+import javax.annotation.Nullable;
+
+
+/**
+ * The <code>ColumnReader</code> interface is used to read column data from various data sources
+ * for columnar segment building. Unlike RecordReader which reads row-by-row, ColumnReader provides
+ * column-wise access to data, enabling efficient columnar segment creation.
+ *
+ * <p>This interface follows an iterator pattern similar to RecordReader:
+ * <ul>
+ * <li>Sequential iteration over all values in a column using hasNext() and next()</li>
+ * <li>Null value detection for the current value</li>
+ * <li>Rewind capability to restart iteration</li>
+ * <li>Resource cleanup</li>
+ * </ul>
+ *
+ * <p>Implementations should handle data type conversions, default values for new columns,
+ * and efficient column-wise data access patterns.
+ */
+public interface ColumnReader extends Closeable, Serializable {
+
+ /**
+ * Return <code>true</code> if more values remain to be read in this column.
+ * <p>This method should not throw exception. Caller is not responsible for handling exceptions from this method.
+ */
+ boolean hasNext();
+
+ /**
+ * Get the next value in the column.
+ * <p>This method should be called only if {@link #hasNext()} returns <code>true</code>. Caller is responsible for
+ * handling exceptions from this method and skip the value if user wants to continue reading the remaining values.
+ *
+ * @return Next column value, or null if the value is null
+ * @throws IOException If an I/O error occurs while reading
+ */
+ @Nullable
+ Object next() throws IOException;
+
+ /**
+ * Check if the current value (last returned by next()) is null.
+ * <p>This method should be called after {@link #next()} to check if the returned value represents a null.
+ *
+ * @return true if the current value is null, false otherwise
+ */
+ boolean isNull();
Review Comment:
Can be handled in `next()`, let me update. Originally, idea was to add custom null check handling in `isNull()`, e.g. "null" string --> null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2430147579
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.stats;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Stats container that efficiently collects statistics from columnar data using ColumnReader instances.
+ *
+ * <p>This implementation collects statistics by iterating column-wise instead of row-wise,
+ * which is more efficient for columnar data sources. It supports:
+ * <ul>
+ * <li>Column-wise statistics collection</li>
+ * <li>Existing columns from source data</li>
+ * <li>New columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ *
+ * <p>The statistics are collected using the same underlying collectors as the row-based approach
+ * (SegmentPreIndexStatsCollectorImpl) but with more efficient column-wise iteration.
+ */
+public class ColumnarSegmentPreIndexStatsContainer implements SegmentPreIndexStatsCollector {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentPreIndexStatsContainer.class);
+
+ private final Map<String, ColumnReader> _columnReaders;
+ private final StatsCollectorConfig _statsCollectorConfig;
+ private final Schema _targetSchema;
+ private final Map<String, AbstractColumnStatisticsCollector> _columnStatsCollectorMap;
+ private int _totalDocCount;
+
+ /**
+ * Create a ColumnarSegmentPreIndexStatsContainer.
+ *
+ * @param columnReaders Map of column name to ColumnReader instances
+ * @param statsCollectorConfig Configuration for statistics collection
+ */
+ public ColumnarSegmentPreIndexStatsContainer(Map<String, ColumnReader> columnReaders,
+ StatsCollectorConfig statsCollectorConfig) {
+ _columnReaders = columnReaders;
+ _statsCollectorConfig = statsCollectorConfig;
+ _targetSchema = statsCollectorConfig.getSchema();
+ _columnStatsCollectorMap = new HashMap<>();
+ _totalDocCount = -1; // indicates unset
+
+ initializeStatsCollectors();
+ collectColumnStats();
+ }
+
+ /**
+ * Initialize stats collectors for all columns in the target schema.
+ */
+ private void initializeStatsCollectors() {
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
+ switch (fieldSpec.getDataType().getStoredType()) {
+ case INT:
+ _columnStatsCollectorMap.put(columnName,
+ new IntColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case LONG:
+ _columnStatsCollectorMap.put(columnName,
+ new LongColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case FLOAT:
+ _columnStatsCollectorMap.put(columnName,
+ new FloatColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case DOUBLE:
+ _columnStatsCollectorMap.put(columnName,
+ new DoubleColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BIG_DECIMAL:
+ _columnStatsCollectorMap.put(columnName,
+ new BigDecimalColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case STRING:
+ _columnStatsCollectorMap.put(columnName,
+ new StringColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BYTES:
+ _columnStatsCollectorMap.put(columnName,
+ new BytesColumnPredIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case MAP:
+ _columnStatsCollectorMap.put(columnName,
+ new MapColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ default:
+ throw new IllegalStateException("Unsupported data type: " + fieldSpec.getDataType());
+ }
+ }
+ }
+
+ /**
+ * Collect stats by iterating column-wise using the provided ColumnReader instances.
+ */
+ private void collectColumnStats() {
+ LOGGER.info("Collecting stats for {} columns using column-wise iteration", _columnReaders.size());
+
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
+ AbstractColumnStatisticsCollector statsCollector = _columnStatsCollectorMap.get(columnName);
+ ColumnReader columnReader = _columnReaders.get(columnName);
+
+ if (columnReader == null) {
+ throw new RuntimeException("Column reader for column " + columnName + " not found");
+ }
+
+ LOGGER.debug("Collecting stats for column: {}", columnName);
+ collectStatsFromColumnReader(columnName, columnReader, statsCollector);
+
+ // Seal the stats collector
+ statsCollector.seal();
+ }
+ }
+
+ /**
+ * Collect stats from a column reader by iterating over all values using the iterator pattern.
+ */
+ private void collectStatsFromColumnReader(String columnName, ColumnReader columnReader,
+ AbstractColumnStatisticsCollector statsCollector) {
+ try {
+ // Reset the column reader to start from the beginning
+ columnReader.rewind();
+
+ int docCount = 0;
+ while (columnReader.hasNext()) {
+ Object value = columnReader.next();
+ statsCollector.collect(value);
+ docCount++;
+ }
+
+ // if totalDocCount is unset then set total doc count from the first column
+ if (_totalDocCount == -1) {
+ _totalDocCount = docCount;
+ } else if (_totalDocCount != docCount) {
+ // all columns should have same count
+ LOGGER.warn("Column {} has {} documents, but expected {} documents",
Review Comment:
Putting assumptions with max might be confusing and we need to handle missing data for smaller columns. Can throw an error here to fail early.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.stats;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Stats container that efficiently collects statistics from columnar data using ColumnReader instances.
+ *
+ * <p>This implementation collects statistics by iterating column-wise instead of row-wise,
+ * which is more efficient for columnar data sources. It supports:
+ * <ul>
+ * <li>Column-wise statistics collection</li>
+ * <li>Existing columns from source data</li>
+ * <li>New columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ *
+ * <p>The statistics are collected using the same underlying collectors as the row-based approach
+ * (SegmentPreIndexStatsCollectorImpl) but with more efficient column-wise iteration.
+ */
+public class ColumnarSegmentPreIndexStatsContainer implements SegmentPreIndexStatsCollector {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentPreIndexStatsContainer.class);
+
+ private final Map<String, ColumnReader> _columnReaders;
+ private final StatsCollectorConfig _statsCollectorConfig;
+ private final Schema _targetSchema;
+ private final Map<String, AbstractColumnStatisticsCollector> _columnStatsCollectorMap;
+ private int _totalDocCount;
+
+ /**
+ * Create a ColumnarSegmentPreIndexStatsContainer.
+ *
+ * @param columnReaders Map of column name to ColumnReader instances
+ * @param statsCollectorConfig Configuration for statistics collection
+ */
+ public ColumnarSegmentPreIndexStatsContainer(Map<String, ColumnReader> columnReaders,
+ StatsCollectorConfig statsCollectorConfig) {
+ _columnReaders = columnReaders;
+ _statsCollectorConfig = statsCollectorConfig;
+ _targetSchema = statsCollectorConfig.getSchema();
+ _columnStatsCollectorMap = new HashMap<>();
+ _totalDocCount = -1; // indicates unset
+
+ initializeStatsCollectors();
+ collectColumnStats();
+ }
+
+ /**
+ * Initialize stats collectors for all columns in the target schema.
+ */
+ private void initializeStatsCollectors() {
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
+ switch (fieldSpec.getDataType().getStoredType()) {
+ case INT:
+ _columnStatsCollectorMap.put(columnName,
+ new IntColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case LONG:
+ _columnStatsCollectorMap.put(columnName,
+ new LongColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case FLOAT:
+ _columnStatsCollectorMap.put(columnName,
+ new FloatColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case DOUBLE:
+ _columnStatsCollectorMap.put(columnName,
+ new DoubleColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BIG_DECIMAL:
+ _columnStatsCollectorMap.put(columnName,
+ new BigDecimalColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case STRING:
+ _columnStatsCollectorMap.put(columnName,
+ new StringColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BYTES:
+ _columnStatsCollectorMap.put(columnName,
+ new BytesColumnPredIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case MAP:
+ _columnStatsCollectorMap.put(columnName,
+ new MapColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ default:
+ throw new IllegalStateException("Unsupported data type: " + fieldSpec.getDataType());
+ }
+ }
+ }
+
+ /**
+ * Collect stats by iterating column-wise using the provided ColumnReader instances.
+ */
+ private void collectColumnStats() {
+ LOGGER.info("Collecting stats for {} columns using column-wise iteration", _columnReaders.size());
+
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
+ AbstractColumnStatisticsCollector statsCollector = _columnStatsCollectorMap.get(columnName);
+ ColumnReader columnReader = _columnReaders.get(columnName);
+
+ if (columnReader == null) {
+ throw new RuntimeException("Column reader for column " + columnName + " not found");
+ }
+
+ LOGGER.debug("Collecting stats for column: {}", columnName);
+ collectStatsFromColumnReader(columnName, columnReader, statsCollector);
+
+ // Seal the stats collector
+ statsCollector.seal();
+ }
+ }
+
+ /**
+ * Collect stats from a column reader by iterating over all values using the iterator pattern.
+ */
+ private void collectStatsFromColumnReader(String columnName, ColumnReader columnReader,
+ AbstractColumnStatisticsCollector statsCollector) {
+ try {
+ // Reset the column reader to start from the beginning
+ columnReader.rewind();
+
+ int docCount = 0;
+ while (columnReader.hasNext()) {
+ Object value = columnReader.next();
+ statsCollector.collect(value);
+ docCount++;
+ }
+
+ // if totalDocCount is unset then set total doc count from the first column
+ if (_totalDocCount == -1) {
+ _totalDocCount = docCount;
+ } else if (_totalDocCount != docCount) {
+ // all columns should have same count
+ LOGGER.warn("Column {} has {} documents, but expected {} documents",
+ columnName, docCount, _totalDocCount);
+ }
+ } catch (IOException e) {
+ LOGGER.error("Failed to collect stats for column: {}", columnName, e);
+ throw new RuntimeException("Failed to collect stats for column: " + columnName, e);
+ }
+ }
+
+ @Override
+ public ColumnStatistics getColumnProfileFor(String column) {
+ return _columnStatsCollectorMap.get(column);
+ }
+
+ @Override
+ public int getTotalDocCount() {
+ return _totalDocCount;
+ }
+
+ @Override
+ public void init() {
+ // Already initialized in constructor
+ }
+
+ @Override
+ public void build() {
+ // Stats are already collected in constructor
+ }
+
+ @Override
+ public void logStats() {
+ LOGGER.info("Columnar segment stats collection completed for {} columns with {} documents",
Review Comment:
column stats get persisted as segment metadata in segment file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2430150403
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java:
##########
@@ -677,4 +706,63 @@ public int getSkippedRowsFound() {
public int getSanitizedRowsFound() {
return _sanitizedRowsFound;
}
+
+ /**
+ * Build segment using columnar approach.
+ * This method builds the segment by processing data column-wise instead of row-wise.
+ *
+ * @throws Exception if segment building fails
+ */
+ public void buildColumnar() throws Exception {
Review Comment:
I think the segment creation should fail even if single columnReader errors which is happening currently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "krishan1390 (via GitHub)" <gi...@apache.org>.
krishan1390 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2431734397
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.stats;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Stats container that efficiently collects statistics from columnar data using ColumnReader instances.
+ *
+ * <p>This implementation collects statistics by iterating column-wise instead of row-wise,
+ * which is more efficient for columnar data sources. It supports:
+ * <ul>
+ * <li>Column-wise statistics collection</li>
+ * <li>Existing columns from source data</li>
+ * <li>New columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ *
+ * <p>The statistics are collected using the same underlying collectors as the row-based approach
+ * (SegmentPreIndexStatsCollectorImpl) but with more efficient column-wise iteration.
+ */
+public class ColumnarSegmentPreIndexStatsContainer implements SegmentPreIndexStatsCollector {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentPreIndexStatsContainer.class);
+
+ private final Map<String, ColumnReader> _columnReaders;
+ private final StatsCollectorConfig _statsCollectorConfig;
+ private final Schema _targetSchema;
+ private final Map<String, AbstractColumnStatisticsCollector> _columnStatsCollectorMap;
+ private int _totalDocCount;
+
+ /**
+ * Create a ColumnarSegmentPreIndexStatsContainer.
+ *
+ * @param columnReaders Map of column name to ColumnReader instances
+ * @param statsCollectorConfig Configuration for statistics collection
+ */
+ public ColumnarSegmentPreIndexStatsContainer(Map<String, ColumnReader> columnReaders,
+ StatsCollectorConfig statsCollectorConfig) {
+ _columnReaders = columnReaders;
+ _statsCollectorConfig = statsCollectorConfig;
+ _targetSchema = statsCollectorConfig.getSchema();
+ _columnStatsCollectorMap = new HashMap<>();
+ _totalDocCount = -1; // indicates unset
+
+ initializeStatsCollectors();
+ collectColumnStats();
+ }
+
+ /**
+ * Initialize stats collectors for all columns in the target schema.
+ */
+ private void initializeStatsCollectors() {
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
+ switch (fieldSpec.getDataType().getStoredType()) {
+ case INT:
+ _columnStatsCollectorMap.put(columnName,
+ new IntColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case LONG:
+ _columnStatsCollectorMap.put(columnName,
+ new LongColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case FLOAT:
+ _columnStatsCollectorMap.put(columnName,
+ new FloatColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case DOUBLE:
+ _columnStatsCollectorMap.put(columnName,
+ new DoubleColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BIG_DECIMAL:
+ _columnStatsCollectorMap.put(columnName,
+ new BigDecimalColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case STRING:
+ _columnStatsCollectorMap.put(columnName,
+ new StringColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BYTES:
+ _columnStatsCollectorMap.put(columnName,
+ new BytesColumnPredIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case MAP:
+ _columnStatsCollectorMap.put(columnName,
+ new MapColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ default:
+ throw new IllegalStateException("Unsupported data type: " + fieldSpec.getDataType());
+ }
+ }
+ }
+
+ /**
+ * Collect stats by iterating column-wise using the provided ColumnReader instances.
+ */
+ private void collectColumnStats() {
+ LOGGER.info("Collecting stats for {} columns using column-wise iteration", _columnReaders.size());
+
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
+ AbstractColumnStatisticsCollector statsCollector = _columnStatsCollectorMap.get(columnName);
+ ColumnReader columnReader = _columnReaders.get(columnName);
+
+ if (columnReader == null) {
+ throw new RuntimeException("Column reader for column " + columnName + " not found");
+ }
+
+ LOGGER.debug("Collecting stats for column: {}", columnName);
+ collectStatsFromColumnReader(columnName, columnReader, statsCollector);
+
+ // Seal the stats collector
+ statsCollector.seal();
+ }
+ }
+
+ /**
+ * Collect stats from a column reader by iterating over all values using the iterator pattern.
+ */
+ private void collectStatsFromColumnReader(String columnName, ColumnReader columnReader,
+ AbstractColumnStatisticsCollector statsCollector) {
+ try {
+ // Reset the column reader to start from the beginning
+ columnReader.rewind();
+
+ int docCount = 0;
+ while (columnReader.hasNext()) {
+ Object value = columnReader.next();
+ statsCollector.collect(value);
+ docCount++;
+ }
+
+ // if totalDocCount is unset then set total doc count from the first column
+ if (_totalDocCount == -1) {
+ _totalDocCount = docCount;
+ } else if (_totalDocCount != docCount) {
+ // all columns should have same count
+ LOGGER.warn("Column {} has {} documents, but expected {} documents",
+ columnName, docCount, _totalDocCount);
+ }
+ } catch (IOException e) {
+ LOGGER.error("Failed to collect stats for column: {}", columnName, e);
+ throw new RuntimeException("Failed to collect stats for column: " + columnName, e);
+ }
+ }
+
+ @Override
+ public ColumnStatistics getColumnProfileFor(String column) {
+ return _columnStatsCollectorMap.get(column);
+ }
+
+ @Override
+ public int getTotalDocCount() {
+ return _totalDocCount;
+ }
+
+ @Override
+ public void init() {
+ // Already initialized in constructor
+ }
+
+ @Override
+ public void build() {
+ // Stats are already collected in constructor
+ }
+
+ @Override
+ public void logStats() {
+ LOGGER.info("Columnar segment stats collection completed for {} columns with {} documents",
Review Comment:
segment metadata won't be available on minion hosts. and might be updated on server if the segment is refreshed multiple times. and will be unavailable if segment is deleted ,etc
for retrospective debugging in such cases, logging the stats will be helpful. its currently done in SegmentPreIndexStatsCollectorImpl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "krishan1390 (via GitHub)" <gi...@apache.org>.
krishan1390 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2431742202
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.stats;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Stats container that efficiently collects statistics from columnar data using ColumnReader instances.
+ *
+ * <p>This implementation collects statistics by iterating column-wise instead of row-wise,
+ * which is more efficient for columnar data sources. It supports:
+ * <ul>
+ * <li>Column-wise statistics collection</li>
+ * <li>Existing columns from source data</li>
+ * <li>New columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ *
+ * <p>The statistics are collected using the same underlying collectors as the row-based approach
+ * (SegmentPreIndexStatsCollectorImpl) but with more efficient column-wise iteration.
+ */
+public class ColumnarSegmentPreIndexStatsContainer implements SegmentPreIndexStatsCollector {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentPreIndexStatsContainer.class);
+
+ private final Map<String, ColumnReader> _columnReaders;
+ private final StatsCollectorConfig _statsCollectorConfig;
+ private final Schema _targetSchema;
+ private final Map<String, AbstractColumnStatisticsCollector> _columnStatsCollectorMap;
+ private int _totalDocCount;
+
+ /**
+ * Create a ColumnarSegmentPreIndexStatsContainer.
+ *
+ * @param columnReaders Map of column name to ColumnReader instances
+ * @param statsCollectorConfig Configuration for statistics collection
+ */
+ public ColumnarSegmentPreIndexStatsContainer(Map<String, ColumnReader> columnReaders,
+ StatsCollectorConfig statsCollectorConfig) {
+ _columnReaders = columnReaders;
+ _statsCollectorConfig = statsCollectorConfig;
+ _targetSchema = statsCollectorConfig.getSchema();
+ _columnStatsCollectorMap = new HashMap<>();
+ _totalDocCount = -1; // indicates unset
+
+ initializeStatsCollectors();
+ collectColumnStats();
+ }
+
+ /**
+ * Initialize stats collectors for all columns in the target schema.
+ */
+ private void initializeStatsCollectors() {
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
+ switch (fieldSpec.getDataType().getStoredType()) {
+ case INT:
+ _columnStatsCollectorMap.put(columnName,
+ new IntColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case LONG:
+ _columnStatsCollectorMap.put(columnName,
+ new LongColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case FLOAT:
+ _columnStatsCollectorMap.put(columnName,
+ new FloatColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case DOUBLE:
+ _columnStatsCollectorMap.put(columnName,
+ new DoubleColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BIG_DECIMAL:
+ _columnStatsCollectorMap.put(columnName,
+ new BigDecimalColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case STRING:
+ _columnStatsCollectorMap.put(columnName,
+ new StringColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BYTES:
+ _columnStatsCollectorMap.put(columnName,
+ new BytesColumnPredIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case MAP:
+ _columnStatsCollectorMap.put(columnName,
+ new MapColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ default:
+ throw new IllegalStateException("Unsupported data type: " + fieldSpec.getDataType());
+ }
+ }
+ }
+
+ /**
+ * Collect stats by iterating column-wise using the provided ColumnReader instances.
+ */
+ private void collectColumnStats() {
+ LOGGER.info("Collecting stats for {} columns using column-wise iteration", _columnReaders.size());
+
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
+ AbstractColumnStatisticsCollector statsCollector = _columnStatsCollectorMap.get(columnName);
+ ColumnReader columnReader = _columnReaders.get(columnName);
+
+ if (columnReader == null) {
+ throw new RuntimeException("Column reader for column " + columnName + " not found");
+ }
+
+ LOGGER.debug("Collecting stats for column: {}", columnName);
+ collectStatsFromColumnReader(columnName, columnReader, statsCollector);
+
+ // Seal the stats collector
+ statsCollector.seal();
+ }
+ }
+
+ /**
+ * Collect stats from a column reader by iterating over all values using the iterator pattern.
+ */
+ private void collectStatsFromColumnReader(String columnName, ColumnReader columnReader,
+ AbstractColumnStatisticsCollector statsCollector) {
+ try {
+ // Reset the column reader to start from the beginning
+ columnReader.rewind();
+
+ int docCount = 0;
+ while (columnReader.hasNext()) {
+ Object value = columnReader.next();
+ statsCollector.collect(value);
Review Comment:
ideally rather than failing the entire segment build, only the problematic rows should be removed. exceptions can be of any time like data type mismatch or any other runtime exception.
you will see that row based build handles such cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "krishan1390 (via GitHub)" <gi...@apache.org>.
krishan1390 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2431727379
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/ColumnarSegmentPreIndexStatsContainer.java:
##########
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.stats;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
+import org.apache.pinot.segment.spi.creator.SegmentPreIndexStatsCollector;
+import org.apache.pinot.segment.spi.creator.StatsCollectorConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Stats container that efficiently collects statistics from columnar data using ColumnReader instances.
+ *
+ * <p>This implementation collects statistics by iterating column-wise instead of row-wise,
+ * which is more efficient for columnar data sources. It supports:
+ * <ul>
+ * <li>Column-wise statistics collection</li>
+ * <li>Existing columns from source data</li>
+ * <li>New columns with default values</li>
+ * <li>Data type conversions during schema evolution</li>
+ * </ul>
+ *
+ * <p>The statistics are collected using the same underlying collectors as the row-based approach
+ * (SegmentPreIndexStatsCollectorImpl) but with more efficient column-wise iteration.
+ */
+public class ColumnarSegmentPreIndexStatsContainer implements SegmentPreIndexStatsCollector {
+ private static final Logger LOGGER = LoggerFactory.getLogger(ColumnarSegmentPreIndexStatsContainer.class);
+
+ private final Map<String, ColumnReader> _columnReaders;
+ private final StatsCollectorConfig _statsCollectorConfig;
+ private final Schema _targetSchema;
+ private final Map<String, AbstractColumnStatisticsCollector> _columnStatsCollectorMap;
+ private int _totalDocCount;
+
+ /**
+ * Create a ColumnarSegmentPreIndexStatsContainer.
+ *
+ * @param columnReaders Map of column name to ColumnReader instances
+ * @param statsCollectorConfig Configuration for statistics collection
+ */
+ public ColumnarSegmentPreIndexStatsContainer(Map<String, ColumnReader> columnReaders,
+ StatsCollectorConfig statsCollectorConfig) {
+ _columnReaders = columnReaders;
+ _statsCollectorConfig = statsCollectorConfig;
+ _targetSchema = statsCollectorConfig.getSchema();
+ _columnStatsCollectorMap = new HashMap<>();
+ _totalDocCount = -1; // indicates unset
+
+ initializeStatsCollectors();
+ collectColumnStats();
+ }
+
+ /**
+ * Initialize stats collectors for all columns in the target schema.
+ */
+ private void initializeStatsCollectors() {
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
+ switch (fieldSpec.getDataType().getStoredType()) {
+ case INT:
+ _columnStatsCollectorMap.put(columnName,
+ new IntColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case LONG:
+ _columnStatsCollectorMap.put(columnName,
+ new LongColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case FLOAT:
+ _columnStatsCollectorMap.put(columnName,
+ new FloatColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case DOUBLE:
+ _columnStatsCollectorMap.put(columnName,
+ new DoubleColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BIG_DECIMAL:
+ _columnStatsCollectorMap.put(columnName,
+ new BigDecimalColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case STRING:
+ _columnStatsCollectorMap.put(columnName,
+ new StringColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case BYTES:
+ _columnStatsCollectorMap.put(columnName,
+ new BytesColumnPredIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ case MAP:
+ _columnStatsCollectorMap.put(columnName,
+ new MapColumnPreIndexStatsCollector(columnName, _statsCollectorConfig));
+ break;
+ default:
+ throw new IllegalStateException("Unsupported data type: " + fieldSpec.getDataType());
+ }
+ }
+ }
+
+ /**
+ * Collect stats by iterating column-wise using the provided ColumnReader instances.
+ */
+ private void collectColumnStats() {
+ LOGGER.info("Collecting stats for {} columns using column-wise iteration", _columnReaders.size());
+
+ for (FieldSpec fieldSpec : _targetSchema.getAllFieldSpecs()) {
+ if (fieldSpec.isVirtualColumn()) {
+ continue;
+ }
+
+ String columnName = fieldSpec.getName();
+ AbstractColumnStatisticsCollector statsCollector = _columnStatsCollectorMap.get(columnName);
+ ColumnReader columnReader = _columnReaders.get(columnName);
+
+ if (columnReader == null) {
+ throw new RuntimeException("Column reader for column " + columnName + " not found");
+ }
+
+ LOGGER.debug("Collecting stats for column: {}", columnName);
+ collectStatsFromColumnReader(columnName, columnReader, statsCollector);
+
+ // Seal the stats collector
+ statsCollector.seal();
+ }
+ }
+
+ /**
+ * Collect stats from a column reader by iterating over all values using the iterator pattern.
+ */
+ private void collectStatsFromColumnReader(String columnName, ColumnReader columnReader,
+ AbstractColumnStatisticsCollector statsCollector) {
+ try {
+ // Reset the column reader to start from the beginning
+ columnReader.rewind();
+
+ int docCount = 0;
+ while (columnReader.hasNext()) {
+ Object value = columnReader.next();
+ statsCollector.collect(value);
+ docCount++;
+ }
+
+ // if totalDocCount is unset then set total doc count from the first column
+ if (_totalDocCount == -1) {
+ _totalDocCount = docCount;
+ } else if (_totalDocCount != docCount) {
+ // all columns should have same count
+ LOGGER.warn("Column {} has {} documents, but expected {} documents",
Review Comment:
yes agree we need to throw an error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "krishan1390 (via GitHub)" <gi...@apache.org>.
krishan1390 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2425437446
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentColumnReaderImpl.java:
##########
@@ -0,0 +1,100 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.readers;
+
+import java.io.IOException;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+
+
+/**
+ * Implementation of ColumnReader for Pinot segments.
+ *
+ * <p>This class wraps the existing PinotSegmentColumnReader and provides the ColumnReader interface
+ * for columnar segment building. It handles:
+ * <ul>
+ * <li>Reading column values from Pinot segments</li>
+ * <li>Resource cleanup</li>
+ * </ul>
+ */
+public class PinotSegmentColumnReaderImpl implements ColumnReader {
+ private final PinotSegmentColumnReader _segmentColumnReader;
+ private final String _columnName;
+ private final int _numDocs;
+
+ private int _currentIndex;
+
+ // Reusable variables to avoid garbage collection on every next() call
+ private Object _reuseValue;
+
+ /**
+ * Create a PinotSegmentColumnReaderImpl for an existing column in the segment.
+ *
+ * @param indexSegment Source segment to read from
+ * @param columnName Name of the column
+ */
+ public PinotSegmentColumnReaderImpl(IndexSegment indexSegment, String columnName) {
+ _segmentColumnReader = new PinotSegmentColumnReader(indexSegment, columnName);
+ _columnName = columnName;
+ _numDocs = indexSegment.getSegmentMetadata().getTotalDocs();
+ _currentIndex = 0;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return _currentIndex < _numDocs;
+ }
+
+ @Override
+ @Nullable
+ public Object next() throws IOException {
+ if (!hasNext()) {
+ throw new IllegalStateException("No more values available");
+ }
+
+ _reuseValue = _segmentColumnReader.getValue(_currentIndex);
+ _currentIndex++;
+
+ // Return null if the value is null
+ if (_segmentColumnReader.isNull(_currentIndex) || _reuseValue == null) {
Review Comment:
we should call _segmentColumnReader.isNull(_currentIndex) before _segmentColumnReader.getValue(_currentIndex) ?
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/creator/impl/ColumnarSegmentBuildingTest.java:
##########
@@ -0,0 +1,500 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl;
+
+import com.google.common.collect.Lists;
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.segment.readers.PinotSegmentColumnReaderFactory;
+import org.apache.pinot.segment.spi.ImmutableSegment;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.data.readers.RecordReader;
+import org.apache.pinot.spi.data.readers.RecordReaderConfig;
+import org.apache.pinot.spi.utils.ReadMode;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+
+/**
+ * Tests for columnar segment building functionality.
+ *
+ * <p>This test class validates the new columnar segment building capability including:
+ * <ul>
+ * <li>ColumnReader interface implementations</li>
+ * <li>ColumnReaderFactory for different data sources</li>
+ * <li>Columnar segment building vs row-major building equivalence</li>
+ * <li>Support for new columns with default values</li>
+ * </ul>
+ */
+public class ColumnarSegmentBuildingTest {
+ private static final String TEMP_DIR = System.getProperty("java.io.tmpdir");
+ private static final String TABLE_NAME = "testTable";
+ private static final String SEGMENT_NAME = "testSegment";
+
+ // Test columns
+ private static final String STRING_COL_1 = "stringCol1";
+ private static final String STRING_COL_2 = "stringCol2";
+ private static final String INT_COL_1 = "intCol1";
+ private static final String INT_COL_2 = "intCol2";
+ private static final String LONG_COL = "longCol";
+ private static final String FLOAT_COL = "floatCol";
+ private static final String DOUBLE_COL = "doubleCol";
+ private static final String BIG_DECIMAL_COL = "bigDecimalCol";
+ private static final String BYTES_COL = "bytesCol";
+ private static final String TIME_COL = "timeCol";
+ private static final String MV_INT_COL = "mvIntCol";
+ private static final String MV_STRING_COL = "mvStringCol";
+
+ // New column for testing default value handling
+ private static final String NEW_STRING_COL = "newStringCol";
+ private static final String NEW_INT_COL = "newIntCol";
+
+ private File _tempDir;
+ private Schema _originalSchema;
+ private Schema _extendedSchema; // Schema with additional columns
+ private TableConfig _tableConfig;
+ private List<GenericRow> _testData;
+
+ @BeforeClass
+ public void setUp() throws IOException {
+ _tempDir = new File(TEMP_DIR, "ColumnarSegmentBuildingTest");
+ FileUtils.deleteQuietly(_tempDir);
+ _tempDir.mkdirs();
+
+ // Create original schema
+ _originalSchema = new Schema.SchemaBuilder()
+ .addSingleValueDimension(STRING_COL_1, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(STRING_COL_2, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(INT_COL_1, FieldSpec.DataType.INT)
+ .addSingleValueDimension(INT_COL_2, FieldSpec.DataType.INT)
+ .addSingleValueDimension(LONG_COL, FieldSpec.DataType.LONG)
+ .addSingleValueDimension(FLOAT_COL, FieldSpec.DataType.FLOAT)
+ .addSingleValueDimension(DOUBLE_COL, FieldSpec.DataType.DOUBLE)
+ .addSingleValueDimension(BIG_DECIMAL_COL, FieldSpec.DataType.BIG_DECIMAL)
+ .addSingleValueDimension(BYTES_COL, FieldSpec.DataType.BYTES)
+ .addMultiValueDimension(MV_INT_COL, FieldSpec.DataType.INT)
+ .addMultiValueDimension(MV_STRING_COL, FieldSpec.DataType.STRING)
+ .addDateTime(TIME_COL, FieldSpec.DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS")
+ .build();
+
+ // Create extended schema with additional columns
+ _extendedSchema = new Schema.SchemaBuilder()
+ .addSingleValueDimension(STRING_COL_1, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(STRING_COL_2, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(INT_COL_1, FieldSpec.DataType.INT)
+ .addSingleValueDimension(INT_COL_2, FieldSpec.DataType.INT)
+ .addSingleValueDimension(LONG_COL, FieldSpec.DataType.LONG)
+ .addSingleValueDimension(FLOAT_COL, FieldSpec.DataType.FLOAT)
+ .addSingleValueDimension(DOUBLE_COL, FieldSpec.DataType.DOUBLE)
+ .addSingleValueDimension(BIG_DECIMAL_COL, FieldSpec.DataType.BIG_DECIMAL)
+ .addSingleValueDimension(BYTES_COL, FieldSpec.DataType.BYTES)
+ .addMultiValueDimension(MV_INT_COL, FieldSpec.DataType.INT)
+ .addMultiValueDimension(MV_STRING_COL, FieldSpec.DataType.STRING)
+ .addDateTime(TIME_COL, FieldSpec.DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS")
+ .addSingleValueDimension(NEW_STRING_COL, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(NEW_INT_COL, FieldSpec.DataType.INT)
Review Comment:
lets add a multi value new column too to cover that scenario too
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/creator/impl/ColumnarSegmentBuildingTest.java:
##########
@@ -0,0 +1,500 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl;
+
+import com.google.common.collect.Lists;
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.segment.readers.PinotSegmentColumnReaderFactory;
+import org.apache.pinot.segment.spi.ImmutableSegment;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.data.readers.RecordReader;
+import org.apache.pinot.spi.data.readers.RecordReaderConfig;
+import org.apache.pinot.spi.utils.ReadMode;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+
+/**
+ * Tests for columnar segment building functionality.
+ *
+ * <p>This test class validates the new columnar segment building capability including:
+ * <ul>
+ * <li>ColumnReader interface implementations</li>
+ * <li>ColumnReaderFactory for different data sources</li>
+ * <li>Columnar segment building vs row-major building equivalence</li>
+ * <li>Support for new columns with default values</li>
Review Comment:
nit. probably can keep seperate test classes for
1. ColumnReader / ColumnReaderFactory tests / handling of new columns etc - eg: testColumnReaderFactory , testColumnReaderWithNewColumns, testAllDataTypes
2. Comparing column and row major segments
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/creator/impl/ColumnarSegmentBuildingTest.java:
##########
@@ -0,0 +1,500 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl;
+
+import com.google.common.collect.Lists;
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.segment.readers.PinotSegmentColumnReaderFactory;
+import org.apache.pinot.segment.spi.ImmutableSegment;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.data.readers.RecordReader;
+import org.apache.pinot.spi.data.readers.RecordReaderConfig;
+import org.apache.pinot.spi.utils.ReadMode;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+
+/**
+ * Tests for columnar segment building functionality.
+ *
+ * <p>This test class validates the new columnar segment building capability including:
+ * <ul>
+ * <li>ColumnReader interface implementations</li>
+ * <li>ColumnReaderFactory for different data sources</li>
+ * <li>Columnar segment building vs row-major building equivalence</li>
+ * <li>Support for new columns with default values</li>
+ * </ul>
+ */
+public class ColumnarSegmentBuildingTest {
+ private static final String TEMP_DIR = System.getProperty("java.io.tmpdir");
+ private static final String TABLE_NAME = "testTable";
+ private static final String SEGMENT_NAME = "testSegment";
+
+ // Test columns
+ private static final String STRING_COL_1 = "stringCol1";
+ private static final String STRING_COL_2 = "stringCol2";
+ private static final String INT_COL_1 = "intCol1";
+ private static final String INT_COL_2 = "intCol2";
+ private static final String LONG_COL = "longCol";
+ private static final String FLOAT_COL = "floatCol";
+ private static final String DOUBLE_COL = "doubleCol";
+ private static final String BIG_DECIMAL_COL = "bigDecimalCol";
+ private static final String BYTES_COL = "bytesCol";
+ private static final String TIME_COL = "timeCol";
+ private static final String MV_INT_COL = "mvIntCol";
+ private static final String MV_STRING_COL = "mvStringCol";
+
+ // New column for testing default value handling
+ private static final String NEW_STRING_COL = "newStringCol";
+ private static final String NEW_INT_COL = "newIntCol";
+
+ private File _tempDir;
+ private Schema _originalSchema;
+ private Schema _extendedSchema; // Schema with additional columns
+ private TableConfig _tableConfig;
+ private List<GenericRow> _testData;
+
+ @BeforeClass
+ public void setUp() throws IOException {
+ _tempDir = new File(TEMP_DIR, "ColumnarSegmentBuildingTest");
+ FileUtils.deleteQuietly(_tempDir);
+ _tempDir.mkdirs();
+
+ // Create original schema
+ _originalSchema = new Schema.SchemaBuilder()
+ .addSingleValueDimension(STRING_COL_1, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(STRING_COL_2, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(INT_COL_1, FieldSpec.DataType.INT)
+ .addSingleValueDimension(INT_COL_2, FieldSpec.DataType.INT)
+ .addSingleValueDimension(LONG_COL, FieldSpec.DataType.LONG)
+ .addSingleValueDimension(FLOAT_COL, FieldSpec.DataType.FLOAT)
+ .addSingleValueDimension(DOUBLE_COL, FieldSpec.DataType.DOUBLE)
+ .addSingleValueDimension(BIG_DECIMAL_COL, FieldSpec.DataType.BIG_DECIMAL)
+ .addSingleValueDimension(BYTES_COL, FieldSpec.DataType.BYTES)
+ .addMultiValueDimension(MV_INT_COL, FieldSpec.DataType.INT)
+ .addMultiValueDimension(MV_STRING_COL, FieldSpec.DataType.STRING)
+ .addDateTime(TIME_COL, FieldSpec.DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS")
+ .build();
+
+ // Create extended schema with additional columns
+ _extendedSchema = new Schema.SchemaBuilder()
+ .addSingleValueDimension(STRING_COL_1, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(STRING_COL_2, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(INT_COL_1, FieldSpec.DataType.INT)
+ .addSingleValueDimension(INT_COL_2, FieldSpec.DataType.INT)
+ .addSingleValueDimension(LONG_COL, FieldSpec.DataType.LONG)
+ .addSingleValueDimension(FLOAT_COL, FieldSpec.DataType.FLOAT)
+ .addSingleValueDimension(DOUBLE_COL, FieldSpec.DataType.DOUBLE)
+ .addSingleValueDimension(BIG_DECIMAL_COL, FieldSpec.DataType.BIG_DECIMAL)
+ .addSingleValueDimension(BYTES_COL, FieldSpec.DataType.BYTES)
+ .addMultiValueDimension(MV_INT_COL, FieldSpec.DataType.INT)
+ .addMultiValueDimension(MV_STRING_COL, FieldSpec.DataType.STRING)
+ .addDateTime(TIME_COL, FieldSpec.DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS")
+ .addSingleValueDimension(NEW_STRING_COL, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(NEW_INT_COL, FieldSpec.DataType.INT)
+ .build();
+
+ // Create table config
+ _tableConfig = new TableConfigBuilder(TableType.OFFLINE)
+ .setTableName(TABLE_NAME)
+ .setTimeColumnName(TIME_COL)
+ .setInvertedIndexColumns(Lists.newArrayList(STRING_COL_1, INT_COL_1))
+ .setSortedColumn(INT_COL_1)
+ .build();
+
+ // Generate test data
+ _testData = generateTestData(100);
+ }
+
+ @AfterClass
+ public void tearDown() throws IOException {
+ FileUtils.deleteQuietly(_tempDir);
+ }
+
+ @Test
+ public void testBasicColumnarBuilding() throws Exception {
+ // First create a segment using traditional row-major approach
+ File rowMajorSegmentDir = createRowMajorSegment();
+
+ // Then create a segment using columnar approach from the row-major segment
+ File columnarSegmentDir = createColumnarSegment(rowMajorSegmentDir);
+
+ // Validate that both segments have identical data
+ validateSegmentsIdentical(rowMajorSegmentDir, columnarSegmentDir);
+ }
+
+ @Test
+ public void testColumnarBuildingWithNewColumns() throws Exception {
+ // Create original segment with original schema
+ File originalSegmentDir = createRowMajorSegment();
+
+ // Create new segment with extended schema (has additional columns)
+ File newSegmentDir = createColumnarSegmentWithNewColumns(originalSegmentDir);
+
+ // Validate that the new segment has the additional columns with default values
Review Comment:
we can also build a row major segment with the updated schema and validate if that segment is identical with columnar segment
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java:
##########
@@ -171,31 +170,72 @@ public void init(SegmentGeneratorConfig config, RecordReader recordReader)
new TransformPipeline(config.getTableConfig(), config.getSchema()));
}
+ /**
+ * Initialize the driver for columnar segment building using a ColumnReaderFactory.
+ * This method sets up the driver to use column-wise input data access instead of row-wise.
Review Comment:
lets document cases where columnar build shouldn't be used even if input data is columnar.
Example : I guess we can't use this for cases where sort column has been modified ?
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/creator/impl/ColumnarSegmentBuildingTest.java:
##########
@@ -0,0 +1,500 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl;
+
+import com.google.common.collect.Lists;
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.segment.readers.PinotSegmentColumnReaderFactory;
+import org.apache.pinot.segment.spi.ImmutableSegment;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.data.readers.RecordReader;
+import org.apache.pinot.spi.data.readers.RecordReaderConfig;
+import org.apache.pinot.spi.utils.ReadMode;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+
+/**
+ * Tests for columnar segment building functionality.
+ *
+ * <p>This test class validates the new columnar segment building capability including:
+ * <ul>
+ * <li>ColumnReader interface implementations</li>
+ * <li>ColumnReaderFactory for different data sources</li>
+ * <li>Columnar segment building vs row-major building equivalence</li>
+ * <li>Support for new columns with default values</li>
+ * </ul>
+ */
+public class ColumnarSegmentBuildingTest {
+ private static final String TEMP_DIR = System.getProperty("java.io.tmpdir");
+ private static final String TABLE_NAME = "testTable";
+ private static final String SEGMENT_NAME = "testSegment";
+
+ // Test columns
+ private static final String STRING_COL_1 = "stringCol1";
+ private static final String STRING_COL_2 = "stringCol2";
+ private static final String INT_COL_1 = "intCol1";
+ private static final String INT_COL_2 = "intCol2";
+ private static final String LONG_COL = "longCol";
+ private static final String FLOAT_COL = "floatCol";
+ private static final String DOUBLE_COL = "doubleCol";
+ private static final String BIG_DECIMAL_COL = "bigDecimalCol";
+ private static final String BYTES_COL = "bytesCol";
+ private static final String TIME_COL = "timeCol";
+ private static final String MV_INT_COL = "mvIntCol";
+ private static final String MV_STRING_COL = "mvStringCol";
+
+ // New column for testing default value handling
+ private static final String NEW_STRING_COL = "newStringCol";
+ private static final String NEW_INT_COL = "newIntCol";
+
+ private File _tempDir;
+ private Schema _originalSchema;
+ private Schema _extendedSchema; // Schema with additional columns
+ private TableConfig _tableConfig;
+ private List<GenericRow> _testData;
+
+ @BeforeClass
+ public void setUp() throws IOException {
+ _tempDir = new File(TEMP_DIR, "ColumnarSegmentBuildingTest");
+ FileUtils.deleteQuietly(_tempDir);
+ _tempDir.mkdirs();
+
+ // Create original schema
+ _originalSchema = new Schema.SchemaBuilder()
+ .addSingleValueDimension(STRING_COL_1, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(STRING_COL_2, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(INT_COL_1, FieldSpec.DataType.INT)
+ .addSingleValueDimension(INT_COL_2, FieldSpec.DataType.INT)
+ .addSingleValueDimension(LONG_COL, FieldSpec.DataType.LONG)
+ .addSingleValueDimension(FLOAT_COL, FieldSpec.DataType.FLOAT)
+ .addSingleValueDimension(DOUBLE_COL, FieldSpec.DataType.DOUBLE)
+ .addSingleValueDimension(BIG_DECIMAL_COL, FieldSpec.DataType.BIG_DECIMAL)
+ .addSingleValueDimension(BYTES_COL, FieldSpec.DataType.BYTES)
+ .addMultiValueDimension(MV_INT_COL, FieldSpec.DataType.INT)
+ .addMultiValueDimension(MV_STRING_COL, FieldSpec.DataType.STRING)
+ .addDateTime(TIME_COL, FieldSpec.DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS")
+ .build();
+
+ // Create extended schema with additional columns
+ _extendedSchema = new Schema.SchemaBuilder()
+ .addSingleValueDimension(STRING_COL_1, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(STRING_COL_2, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(INT_COL_1, FieldSpec.DataType.INT)
+ .addSingleValueDimension(INT_COL_2, FieldSpec.DataType.INT)
+ .addSingleValueDimension(LONG_COL, FieldSpec.DataType.LONG)
+ .addSingleValueDimension(FLOAT_COL, FieldSpec.DataType.FLOAT)
+ .addSingleValueDimension(DOUBLE_COL, FieldSpec.DataType.DOUBLE)
+ .addSingleValueDimension(BIG_DECIMAL_COL, FieldSpec.DataType.BIG_DECIMAL)
+ .addSingleValueDimension(BYTES_COL, FieldSpec.DataType.BYTES)
+ .addMultiValueDimension(MV_INT_COL, FieldSpec.DataType.INT)
+ .addMultiValueDimension(MV_STRING_COL, FieldSpec.DataType.STRING)
+ .addDateTime(TIME_COL, FieldSpec.DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS")
+ .addSingleValueDimension(NEW_STRING_COL, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(NEW_INT_COL, FieldSpec.DataType.INT)
+ .build();
+
+ // Create table config
+ _tableConfig = new TableConfigBuilder(TableType.OFFLINE)
+ .setTableName(TABLE_NAME)
+ .setTimeColumnName(TIME_COL)
+ .setInvertedIndexColumns(Lists.newArrayList(STRING_COL_1, INT_COL_1))
+ .setSortedColumn(INT_COL_1)
+ .build();
+
+ // Generate test data
+ _testData = generateTestData(100);
+ }
+
+ @AfterClass
+ public void tearDown() throws IOException {
+ FileUtils.deleteQuietly(_tempDir);
+ }
+
+ @Test
+ public void testBasicColumnarBuilding() throws Exception {
+ // First create a segment using traditional row-major approach
+ File rowMajorSegmentDir = createRowMajorSegment();
+
+ // Then create a segment using columnar approach from the row-major segment
+ File columnarSegmentDir = createColumnarSegment(rowMajorSegmentDir);
+
+ // Validate that both segments have identical data
+ validateSegmentsIdentical(rowMajorSegmentDir, columnarSegmentDir);
+ }
+
+ @Test
+ public void testColumnarBuildingWithNewColumns() throws Exception {
+ // Create original segment with original schema
+ File originalSegmentDir = createRowMajorSegment();
+
+ // Create new segment with extended schema (has additional columns)
+ File newSegmentDir = createColumnarSegmentWithNewColumns(originalSegmentDir);
+
+ // Validate that the new segment has the additional columns with default values
+ validateSegmentWithNewColumns(newSegmentDir);
+ }
+
+ @Test
+ public void testColumnReaderFactory() throws Exception {
+ // Create a segment to test column reader factory with
+ File segmentDir = createRowMajorSegment();
+ ImmutableSegment segment = ImmutableSegmentLoader.load(segmentDir, ReadMode.mmap);
+
+ try {
+ // Test PinotSegmentColumnReaderFactory
+ try (PinotSegmentColumnReaderFactory factory = new PinotSegmentColumnReaderFactory(segment)) {
+ factory.init(_originalSchema);
+
+ // Test that all expected columns are available
+ Set<String> availableColumns = factory.getAvailableColumns();
+ Assert.assertTrue(availableColumns.contains(STRING_COL_1));
+ Assert.assertTrue(availableColumns.contains(INT_COL_1));
+ Assert.assertTrue(availableColumns.contains(MV_INT_COL));
+
+ // Test creating individual column readers
+ ColumnReader stringReader = factory.createColumnReader(STRING_COL_1,
+ _originalSchema.getFieldSpecFor(STRING_COL_1));
+ Assert.assertEquals(stringReader.getColumnName(), STRING_COL_1);
+
+ ColumnReader mvReader = factory.createColumnReader(MV_INT_COL,
+ _originalSchema.getFieldSpecFor(MV_INT_COL));
+ Assert.assertEquals(mvReader.getColumnName(), MV_INT_COL);
+
+ // Test reading values using iterator pattern
+ Assert.assertTrue(stringReader.hasNext());
+ Object firstStringValue = stringReader.next();
+ Assert.assertEquals(firstStringValue, "string1_0");
+
+ // Test that we can continue reading
+ Assert.assertTrue(stringReader.hasNext());
+ Object secondStringValue = stringReader.next();
+ Assert.assertEquals(secondStringValue, "string1_1");
+
+ // Reset and test again
+ stringReader.rewind();
+ Assert.assertTrue(stringReader.hasNext());
+ firstStringValue = stringReader.next();
+ Assert.assertEquals(firstStringValue, "string1_0");
+
+ // Test creating all column readers
+ Map<String, ColumnReader> allReaders = factory.getAllColumnReaders();
+ Assert.assertEquals(allReaders.size(), _originalSchema.getPhysicalColumnNames().size());
+ }
+ } finally {
+ segment.destroy();
+ }
+ }
+
+ @Test
+ public void testColumnReaderWithNewColumns() throws Exception {
+ // Create a segment to test with
+ File segmentDir = createRowMajorSegment();
+ ImmutableSegment segment = ImmutableSegmentLoader.load(segmentDir, ReadMode.mmap);
+
+ try {
+ // Test creating readers for new columns (should return default value readers)
+ try (PinotSegmentColumnReaderFactory factory = new PinotSegmentColumnReaderFactory(segment)) {
+ factory.init(_extendedSchema);
+
+ // Test creating reader for new column
+ ColumnReader newStringReader = factory.createColumnReader(NEW_STRING_COL,
+ _extendedSchema.getFieldSpecFor(NEW_STRING_COL));
+ Assert.assertEquals(newStringReader.getColumnName(), NEW_STRING_COL);
+
+ // Verify it returns default values using iterator pattern
+ Assert.assertTrue(newStringReader.hasNext());
+ Object defaultValue = newStringReader.next();
+ Assert.assertEquals(defaultValue, _extendedSchema.getFieldSpecFor(NEW_STRING_COL).getDefaultNullValue());
+
+ // Test that all values are the same (default)
+ int valueCount = 1; // We already read one value
+ while (newStringReader.hasNext()) {
+ Object value = newStringReader.next();
+ Assert.assertEquals(value, defaultValue);
+ valueCount++;
+ }
+ Assert.assertEquals(valueCount, _testData.size());
+ }
+ } finally {
+ segment.destroy();
+ }
+ }
+
+ @Test
+ public void testAllDataTypes() throws Exception {
+ // This test validates that all supported data types work correctly with columnar building
+ File rowMajorSegmentDir = createRowMajorSegment();
+ File columnarSegmentDir = createColumnarSegment(rowMajorSegmentDir);
+
+ // Validate that both segments have identical data for all data types
+ validateSegmentsIdentical(rowMajorSegmentDir, columnarSegmentDir);
+
+ // Additionally validate that all expected columns and data types are present
+ ImmutableSegment segment = ImmutableSegmentLoader.load(columnarSegmentDir, ReadMode.mmap);
+ try {
+ Set<String> columnNames = segment.getPhysicalColumnNames();
+
+ // Validate all data types are present
+ Assert.assertTrue(columnNames.contains(STRING_COL_1), "STRING column missing");
+ Assert.assertTrue(columnNames.contains(INT_COL_1), "INT column missing");
+ Assert.assertTrue(columnNames.contains(LONG_COL), "LONG column missing");
+ Assert.assertTrue(columnNames.contains(FLOAT_COL), "FLOAT column missing");
+ Assert.assertTrue(columnNames.contains(DOUBLE_COL), "DOUBLE column missing");
+ Assert.assertTrue(columnNames.contains(BIG_DECIMAL_COL), "BIG_DECIMAL column missing");
+ Assert.assertTrue(columnNames.contains(BYTES_COL), "BYTES column missing");
+ Assert.assertTrue(columnNames.contains(MV_INT_COL), "Multi-value INT column missing");
+ Assert.assertTrue(columnNames.contains(MV_STRING_COL), "Multi-value STRING column missing");
+ Assert.assertTrue(columnNames.contains(TIME_COL), "TIME column missing");
+ } finally {
+ segment.destroy();
+ }
+ }
+
+
+
+ private File createRowMajorSegment() throws Exception {
+ File outputDir = new File(_tempDir, "rowMajorSegment");
+ FileUtils.deleteQuietly(outputDir);
+ outputDir.mkdirs();
+
+ SegmentGeneratorConfig config = new SegmentGeneratorConfig(_tableConfig, _originalSchema);
+ config.setOutDir(outputDir.getAbsolutePath());
+ config.setSegmentName(SEGMENT_NAME + "_rowMajor");
+
+ SegmentIndexCreationDriverImpl driver = new SegmentIndexCreationDriverImpl();
+ TestRecordReader recordReader = new TestRecordReader(_testData);
+ driver.init(config, recordReader);
+ driver.build();
+
+ return new File(outputDir, SEGMENT_NAME + "_rowMajor");
+ }
+
+ private File createColumnarSegment(File sourceSegmentDir) throws Exception {
+ File outputDir = new File(_tempDir, "columnarSegment");
+ FileUtils.deleteQuietly(outputDir);
+ outputDir.mkdirs();
+
+ // Load the source segment
+ ImmutableSegment sourceSegment = ImmutableSegmentLoader.load(sourceSegmentDir, ReadMode.mmap);
+
+ try {
+ SegmentGeneratorConfig config = new SegmentGeneratorConfig(_tableConfig, _originalSchema);
+ config.setOutDir(outputDir.getAbsolutePath());
+ config.setSegmentName(SEGMENT_NAME + "_columnar");
+
+ SegmentIndexCreationDriverImpl driver = new SegmentIndexCreationDriverImpl();
+
+ // Use the new columnar building approach with config
+ try (PinotSegmentColumnReaderFactory factory = new PinotSegmentColumnReaderFactory(sourceSegment)) {
+ driver.init(config, factory);
+ driver.build();
+ }
+
+ return new File(outputDir, SEGMENT_NAME + "_columnar");
+ } finally {
+ sourceSegment.destroy();
+ }
+ }
+
+ private File createColumnarSegmentWithNewColumns(File sourceSegmentDir) throws Exception {
+ File outputDir = new File(_tempDir, "columnarSegmentWithNewColumns");
+ FileUtils.deleteQuietly(outputDir);
+ outputDir.mkdirs();
+
+ // Load the source segment
+ ImmutableSegment sourceSegment = ImmutableSegmentLoader.load(sourceSegmentDir, ReadMode.mmap);
+
+ try {
+ // Use extended schema with new columns
+ SegmentGeneratorConfig config = new SegmentGeneratorConfig(_tableConfig, _extendedSchema);
+ config.setOutDir(outputDir.getAbsolutePath());
+ config.setSegmentName(SEGMENT_NAME + "_withNewColumns");
+
+ SegmentIndexCreationDriverImpl driver = new SegmentIndexCreationDriverImpl();
+
+ // Use columnar building with extended schema
+ try (PinotSegmentColumnReaderFactory factory = new PinotSegmentColumnReaderFactory(sourceSegment)) {
+ driver.init(config, factory);
+ driver.build();
+ }
+
+ return new File(outputDir, SEGMENT_NAME + "_withNewColumns");
+ } finally {
+ sourceSegment.destroy();
+ }
+ }
+
+ private void validateSegmentsIdentical(File segment1Dir, File segment2Dir) throws Exception {
+ ImmutableSegment segment1 = ImmutableSegmentLoader.load(segment1Dir, ReadMode.mmap);
+ ImmutableSegment segment2 = ImmutableSegmentLoader.load(segment2Dir, ReadMode.mmap);
+
+ try {
+ // Validate metadata
+ Assert.assertEquals(segment1.getSegmentMetadata().getTotalDocs(), segment2.getSegmentMetadata().getTotalDocs());
+ Assert.assertEquals(segment1.getSegmentMetadata().getAllColumns(), segment2.getSegmentMetadata().getAllColumns());
Review Comment:
we should validate segment columnar stats too.
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/creator/impl/ColumnarSegmentBuildingTest.java:
##########
@@ -0,0 +1,500 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl;
+
+import com.google.common.collect.Lists;
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.segment.readers.PinotSegmentColumnReaderFactory;
+import org.apache.pinot.segment.spi.ImmutableSegment;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.data.readers.RecordReader;
+import org.apache.pinot.spi.data.readers.RecordReaderConfig;
+import org.apache.pinot.spi.utils.ReadMode;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+
+/**
+ * Tests for columnar segment building functionality.
+ *
+ * <p>This test class validates the new columnar segment building capability including:
+ * <ul>
+ * <li>ColumnReader interface implementations</li>
+ * <li>ColumnReaderFactory for different data sources</li>
+ * <li>Columnar segment building vs row-major building equivalence</li>
+ * <li>Support for new columns with default values</li>
+ * </ul>
+ */
+public class ColumnarSegmentBuildingTest {
+ private static final String TEMP_DIR = System.getProperty("java.io.tmpdir");
+ private static final String TABLE_NAME = "testTable";
+ private static final String SEGMENT_NAME = "testSegment";
+
+ // Test columns
+ private static final String STRING_COL_1 = "stringCol1";
+ private static final String STRING_COL_2 = "stringCol2";
+ private static final String INT_COL_1 = "intCol1";
+ private static final String INT_COL_2 = "intCol2";
+ private static final String LONG_COL = "longCol";
+ private static final String FLOAT_COL = "floatCol";
+ private static final String DOUBLE_COL = "doubleCol";
+ private static final String BIG_DECIMAL_COL = "bigDecimalCol";
+ private static final String BYTES_COL = "bytesCol";
+ private static final String TIME_COL = "timeCol";
+ private static final String MV_INT_COL = "mvIntCol";
+ private static final String MV_STRING_COL = "mvStringCol";
+
+ // New column for testing default value handling
+ private static final String NEW_STRING_COL = "newStringCol";
+ private static final String NEW_INT_COL = "newIntCol";
+
+ private File _tempDir;
+ private Schema _originalSchema;
+ private Schema _extendedSchema; // Schema with additional columns
+ private TableConfig _tableConfig;
+ private List<GenericRow> _testData;
+
+ @BeforeClass
+ public void setUp() throws IOException {
+ _tempDir = new File(TEMP_DIR, "ColumnarSegmentBuildingTest");
+ FileUtils.deleteQuietly(_tempDir);
+ _tempDir.mkdirs();
+
+ // Create original schema
+ _originalSchema = new Schema.SchemaBuilder()
+ .addSingleValueDimension(STRING_COL_1, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(STRING_COL_2, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(INT_COL_1, FieldSpec.DataType.INT)
+ .addSingleValueDimension(INT_COL_2, FieldSpec.DataType.INT)
+ .addSingleValueDimension(LONG_COL, FieldSpec.DataType.LONG)
+ .addSingleValueDimension(FLOAT_COL, FieldSpec.DataType.FLOAT)
+ .addSingleValueDimension(DOUBLE_COL, FieldSpec.DataType.DOUBLE)
+ .addSingleValueDimension(BIG_DECIMAL_COL, FieldSpec.DataType.BIG_DECIMAL)
+ .addSingleValueDimension(BYTES_COL, FieldSpec.DataType.BYTES)
+ .addMultiValueDimension(MV_INT_COL, FieldSpec.DataType.INT)
+ .addMultiValueDimension(MV_STRING_COL, FieldSpec.DataType.STRING)
+ .addDateTime(TIME_COL, FieldSpec.DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS")
+ .build();
+
+ // Create extended schema with additional columns
+ _extendedSchema = new Schema.SchemaBuilder()
+ .addSingleValueDimension(STRING_COL_1, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(STRING_COL_2, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(INT_COL_1, FieldSpec.DataType.INT)
+ .addSingleValueDimension(INT_COL_2, FieldSpec.DataType.INT)
+ .addSingleValueDimension(LONG_COL, FieldSpec.DataType.LONG)
+ .addSingleValueDimension(FLOAT_COL, FieldSpec.DataType.FLOAT)
+ .addSingleValueDimension(DOUBLE_COL, FieldSpec.DataType.DOUBLE)
+ .addSingleValueDimension(BIG_DECIMAL_COL, FieldSpec.DataType.BIG_DECIMAL)
+ .addSingleValueDimension(BYTES_COL, FieldSpec.DataType.BYTES)
+ .addMultiValueDimension(MV_INT_COL, FieldSpec.DataType.INT)
+ .addMultiValueDimension(MV_STRING_COL, FieldSpec.DataType.STRING)
+ .addDateTime(TIME_COL, FieldSpec.DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS")
+ .addSingleValueDimension(NEW_STRING_COL, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(NEW_INT_COL, FieldSpec.DataType.INT)
+ .build();
+
+ // Create table config
+ _tableConfig = new TableConfigBuilder(TableType.OFFLINE)
+ .setTableName(TABLE_NAME)
+ .setTimeColumnName(TIME_COL)
+ .setInvertedIndexColumns(Lists.newArrayList(STRING_COL_1, INT_COL_1))
+ .setSortedColumn(INT_COL_1)
+ .build();
+
+ // Generate test data
+ _testData = generateTestData(100);
+ }
+
+ @AfterClass
+ public void tearDown() throws IOException {
+ FileUtils.deleteQuietly(_tempDir);
+ }
+
+ @Test
+ public void testBasicColumnarBuilding() throws Exception {
+ // First create a segment using traditional row-major approach
+ File rowMajorSegmentDir = createRowMajorSegment();
+
+ // Then create a segment using columnar approach from the row-major segment
+ File columnarSegmentDir = createColumnarSegment(rowMajorSegmentDir);
+
+ // Validate that both segments have identical data
+ validateSegmentsIdentical(rowMajorSegmentDir, columnarSegmentDir);
+ }
+
+ @Test
+ public void testColumnarBuildingWithNewColumns() throws Exception {
+ // Create original segment with original schema
+ File originalSegmentDir = createRowMajorSegment();
+
+ // Create new segment with extended schema (has additional columns)
+ File newSegmentDir = createColumnarSegmentWithNewColumns(originalSegmentDir);
+
+ // Validate that the new segment has the additional columns with default values
+ validateSegmentWithNewColumns(newSegmentDir);
+ }
+
+ @Test
+ public void testColumnReaderFactory() throws Exception {
+ // Create a segment to test column reader factory with
+ File segmentDir = createRowMajorSegment();
+ ImmutableSegment segment = ImmutableSegmentLoader.load(segmentDir, ReadMode.mmap);
+
+ try {
+ // Test PinotSegmentColumnReaderFactory
+ try (PinotSegmentColumnReaderFactory factory = new PinotSegmentColumnReaderFactory(segment)) {
+ factory.init(_originalSchema);
+
+ // Test that all expected columns are available
+ Set<String> availableColumns = factory.getAvailableColumns();
+ Assert.assertTrue(availableColumns.contains(STRING_COL_1));
+ Assert.assertTrue(availableColumns.contains(INT_COL_1));
+ Assert.assertTrue(availableColumns.contains(MV_INT_COL));
+
+ // Test creating individual column readers
+ ColumnReader stringReader = factory.createColumnReader(STRING_COL_1,
+ _originalSchema.getFieldSpecFor(STRING_COL_1));
+ Assert.assertEquals(stringReader.getColumnName(), STRING_COL_1);
+
+ ColumnReader mvReader = factory.createColumnReader(MV_INT_COL,
+ _originalSchema.getFieldSpecFor(MV_INT_COL));
+ Assert.assertEquals(mvReader.getColumnName(), MV_INT_COL);
+
+ // Test reading values using iterator pattern
+ Assert.assertTrue(stringReader.hasNext());
+ Object firstStringValue = stringReader.next();
+ Assert.assertEquals(firstStringValue, "string1_0");
+
+ // Test that we can continue reading
+ Assert.assertTrue(stringReader.hasNext());
+ Object secondStringValue = stringReader.next();
+ Assert.assertEquals(secondStringValue, "string1_1");
+
+ // Reset and test again
+ stringReader.rewind();
+ Assert.assertTrue(stringReader.hasNext());
+ firstStringValue = stringReader.next();
+ Assert.assertEquals(firstStringValue, "string1_0");
+
+ // Test creating all column readers
+ Map<String, ColumnReader> allReaders = factory.getAllColumnReaders();
+ Assert.assertEquals(allReaders.size(), _originalSchema.getPhysicalColumnNames().size());
+ }
+ } finally {
+ segment.destroy();
+ }
+ }
+
+ @Test
+ public void testColumnReaderWithNewColumns() throws Exception {
+ // Create a segment to test with
+ File segmentDir = createRowMajorSegment();
+ ImmutableSegment segment = ImmutableSegmentLoader.load(segmentDir, ReadMode.mmap);
+
+ try {
+ // Test creating readers for new columns (should return default value readers)
+ try (PinotSegmentColumnReaderFactory factory = new PinotSegmentColumnReaderFactory(segment)) {
+ factory.init(_extendedSchema);
+
+ // Test creating reader for new column
+ ColumnReader newStringReader = factory.createColumnReader(NEW_STRING_COL,
+ _extendedSchema.getFieldSpecFor(NEW_STRING_COL));
+ Assert.assertEquals(newStringReader.getColumnName(), NEW_STRING_COL);
+
+ // Verify it returns default values using iterator pattern
+ Assert.assertTrue(newStringReader.hasNext());
+ Object defaultValue = newStringReader.next();
+ Assert.assertEquals(defaultValue, _extendedSchema.getFieldSpecFor(NEW_STRING_COL).getDefaultNullValue());
+
+ // Test that all values are the same (default)
+ int valueCount = 1; // We already read one value
+ while (newStringReader.hasNext()) {
+ Object value = newStringReader.next();
+ Assert.assertEquals(value, defaultValue);
+ valueCount++;
+ }
+ Assert.assertEquals(valueCount, _testData.size());
+ }
+ } finally {
+ segment.destroy();
+ }
+ }
+
+ @Test
+ public void testAllDataTypes() throws Exception {
+ // This test validates that all supported data types work correctly with columnar building
+ File rowMajorSegmentDir = createRowMajorSegment();
+ File columnarSegmentDir = createColumnarSegment(rowMajorSegmentDir);
+
+ // Validate that both segments have identical data for all data types
+ validateSegmentsIdentical(rowMajorSegmentDir, columnarSegmentDir);
+
+ // Additionally validate that all expected columns and data types are present
+ ImmutableSegment segment = ImmutableSegmentLoader.load(columnarSegmentDir, ReadMode.mmap);
+ try {
+ Set<String> columnNames = segment.getPhysicalColumnNames();
+
+ // Validate all data types are present
+ Assert.assertTrue(columnNames.contains(STRING_COL_1), "STRING column missing");
+ Assert.assertTrue(columnNames.contains(INT_COL_1), "INT column missing");
+ Assert.assertTrue(columnNames.contains(LONG_COL), "LONG column missing");
+ Assert.assertTrue(columnNames.contains(FLOAT_COL), "FLOAT column missing");
+ Assert.assertTrue(columnNames.contains(DOUBLE_COL), "DOUBLE column missing");
+ Assert.assertTrue(columnNames.contains(BIG_DECIMAL_COL), "BIG_DECIMAL column missing");
+ Assert.assertTrue(columnNames.contains(BYTES_COL), "BYTES column missing");
+ Assert.assertTrue(columnNames.contains(MV_INT_COL), "Multi-value INT column missing");
+ Assert.assertTrue(columnNames.contains(MV_STRING_COL), "Multi-value STRING column missing");
+ Assert.assertTrue(columnNames.contains(TIME_COL), "TIME column missing");
+ } finally {
+ segment.destroy();
+ }
+ }
+
+
+
+ private File createRowMajorSegment() throws Exception {
+ File outputDir = new File(_tempDir, "rowMajorSegment");
+ FileUtils.deleteQuietly(outputDir);
+ outputDir.mkdirs();
+
+ SegmentGeneratorConfig config = new SegmentGeneratorConfig(_tableConfig, _originalSchema);
+ config.setOutDir(outputDir.getAbsolutePath());
+ config.setSegmentName(SEGMENT_NAME + "_rowMajor");
+
+ SegmentIndexCreationDriverImpl driver = new SegmentIndexCreationDriverImpl();
+ TestRecordReader recordReader = new TestRecordReader(_testData);
+ driver.init(config, recordReader);
+ driver.build();
+
+ return new File(outputDir, SEGMENT_NAME + "_rowMajor");
+ }
+
+ private File createColumnarSegment(File sourceSegmentDir) throws Exception {
+ File outputDir = new File(_tempDir, "columnarSegment");
+ FileUtils.deleteQuietly(outputDir);
+ outputDir.mkdirs();
+
+ // Load the source segment
+ ImmutableSegment sourceSegment = ImmutableSegmentLoader.load(sourceSegmentDir, ReadMode.mmap);
+
+ try {
+ SegmentGeneratorConfig config = new SegmentGeneratorConfig(_tableConfig, _originalSchema);
+ config.setOutDir(outputDir.getAbsolutePath());
+ config.setSegmentName(SEGMENT_NAME + "_columnar");
+
+ SegmentIndexCreationDriverImpl driver = new SegmentIndexCreationDriverImpl();
+
+ // Use the new columnar building approach with config
+ try (PinotSegmentColumnReaderFactory factory = new PinotSegmentColumnReaderFactory(sourceSegment)) {
+ driver.init(config, factory);
+ driver.build();
+ }
+
+ return new File(outputDir, SEGMENT_NAME + "_columnar");
+ } finally {
+ sourceSegment.destroy();
+ }
+ }
+
+ private File createColumnarSegmentWithNewColumns(File sourceSegmentDir) throws Exception {
+ File outputDir = new File(_tempDir, "columnarSegmentWithNewColumns");
+ FileUtils.deleteQuietly(outputDir);
+ outputDir.mkdirs();
+
+ // Load the source segment
+ ImmutableSegment sourceSegment = ImmutableSegmentLoader.load(sourceSegmentDir, ReadMode.mmap);
+
+ try {
+ // Use extended schema with new columns
+ SegmentGeneratorConfig config = new SegmentGeneratorConfig(_tableConfig, _extendedSchema);
+ config.setOutDir(outputDir.getAbsolutePath());
+ config.setSegmentName(SEGMENT_NAME + "_withNewColumns");
+
+ SegmentIndexCreationDriverImpl driver = new SegmentIndexCreationDriverImpl();
+
+ // Use columnar building with extended schema
+ try (PinotSegmentColumnReaderFactory factory = new PinotSegmentColumnReaderFactory(sourceSegment)) {
+ driver.init(config, factory);
+ driver.build();
+ }
+
+ return new File(outputDir, SEGMENT_NAME + "_withNewColumns");
+ } finally {
+ sourceSegment.destroy();
+ }
+ }
+
+ private void validateSegmentsIdentical(File segment1Dir, File segment2Dir) throws Exception {
+ ImmutableSegment segment1 = ImmutableSegmentLoader.load(segment1Dir, ReadMode.mmap);
+ ImmutableSegment segment2 = ImmutableSegmentLoader.load(segment2Dir, ReadMode.mmap);
+
+ try {
+ // Validate metadata
+ Assert.assertEquals(segment1.getSegmentMetadata().getTotalDocs(), segment2.getSegmentMetadata().getTotalDocs());
+ Assert.assertEquals(segment1.getSegmentMetadata().getAllColumns(), segment2.getSegmentMetadata().getAllColumns());
+
+ // Validate data for each column
+ for (String columnName : segment1.getPhysicalColumnNames()) {
+ validateColumnData(segment1, segment2, columnName);
+ }
+ } finally {
+ segment1.destroy();
+ segment2.destroy();
+ }
+ }
+
+ private void validateSegmentWithNewColumns(File segmentDir) throws Exception {
+ ImmutableSegment segment = ImmutableSegmentLoader.load(segmentDir, ReadMode.mmap);
+
+ try {
+ // Validate that new columns exist
+ Assert.assertTrue(segment.getPhysicalColumnNames().contains(NEW_STRING_COL));
+ Assert.assertTrue(segment.getPhysicalColumnNames().contains(NEW_INT_COL));
+
+ // Validate that new columns have default values
+ GenericRow row = new GenericRow();
+ for (int docId = 0; docId < segment.getSegmentMetadata().getTotalDocs(); docId++) {
+ segment.getRecord(docId, row);
+
+ // Check default values
+ Assert.assertEquals(row.getValue(NEW_STRING_COL),
+ _extendedSchema.getFieldSpecFor(NEW_STRING_COL).getDefaultNullValue());
+ Assert.assertEquals(row.getValue(NEW_INT_COL),
+ _extendedSchema.getFieldSpecFor(NEW_INT_COL).getDefaultNullValue());
+ }
+ } finally {
+ segment.destroy();
+ }
+ }
+
+ private void validateColumnData(ImmutableSegment segment1, ImmutableSegment segment2, String columnName) {
+ int numDocs = segment1.getSegmentMetadata().getTotalDocs();
+
+ GenericRow row1 = new GenericRow();
+ GenericRow row2 = new GenericRow();
+
+ for (int docId = 0; docId < numDocs; docId++) {
+ segment1.getRecord(docId, row1);
+ segment2.getRecord(docId, row2);
+
+ Object value1 = row1.getValue(columnName);
+ Object value2 = row2.getValue(columnName);
+
+ Assert.assertEquals(value1, value2,
+ String.format("Column %s differs at docId %d: %s vs %s", columnName, docId, value1, value2));
+ }
+ }
+
+ private List<GenericRow> generateTestData(int numRows) {
+ List<GenericRow> data = new ArrayList<>();
+
+ for (int i = 0; i < numRows; i++) {
Review Comment:
lets also add some null values in various fields randomly
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/creator/impl/ColumnarSegmentBuildingTest.java:
##########
@@ -0,0 +1,500 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl;
+
+import com.google.common.collect.Lists;
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.segment.readers.PinotSegmentColumnReaderFactory;
+import org.apache.pinot.segment.spi.ImmutableSegment;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.data.readers.RecordReader;
+import org.apache.pinot.spi.data.readers.RecordReaderConfig;
+import org.apache.pinot.spi.utils.ReadMode;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+
+/**
+ * Tests for columnar segment building functionality.
+ *
+ * <p>This test class validates the new columnar segment building capability including:
+ * <ul>
+ * <li>ColumnReader interface implementations</li>
+ * <li>ColumnReaderFactory for different data sources</li>
+ * <li>Columnar segment building vs row-major building equivalence</li>
+ * <li>Support for new columns with default values</li>
+ * </ul>
+ */
+public class ColumnarSegmentBuildingTest {
+ private static final String TEMP_DIR = System.getProperty("java.io.tmpdir");
+ private static final String TABLE_NAME = "testTable";
+ private static final String SEGMENT_NAME = "testSegment";
+
+ // Test columns
+ private static final String STRING_COL_1 = "stringCol1";
+ private static final String STRING_COL_2 = "stringCol2";
+ private static final String INT_COL_1 = "intCol1";
+ private static final String INT_COL_2 = "intCol2";
+ private static final String LONG_COL = "longCol";
+ private static final String FLOAT_COL = "floatCol";
+ private static final String DOUBLE_COL = "doubleCol";
+ private static final String BIG_DECIMAL_COL = "bigDecimalCol";
+ private static final String BYTES_COL = "bytesCol";
+ private static final String TIME_COL = "timeCol";
+ private static final String MV_INT_COL = "mvIntCol";
+ private static final String MV_STRING_COL = "mvStringCol";
+
+ // New column for testing default value handling
+ private static final String NEW_STRING_COL = "newStringCol";
+ private static final String NEW_INT_COL = "newIntCol";
+
+ private File _tempDir;
+ private Schema _originalSchema;
+ private Schema _extendedSchema; // Schema with additional columns
+ private TableConfig _tableConfig;
+ private List<GenericRow> _testData;
+
+ @BeforeClass
+ public void setUp() throws IOException {
+ _tempDir = new File(TEMP_DIR, "ColumnarSegmentBuildingTest");
+ FileUtils.deleteQuietly(_tempDir);
+ _tempDir.mkdirs();
+
+ // Create original schema
+ _originalSchema = new Schema.SchemaBuilder()
+ .addSingleValueDimension(STRING_COL_1, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(STRING_COL_2, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(INT_COL_1, FieldSpec.DataType.INT)
+ .addSingleValueDimension(INT_COL_2, FieldSpec.DataType.INT)
+ .addSingleValueDimension(LONG_COL, FieldSpec.DataType.LONG)
+ .addSingleValueDimension(FLOAT_COL, FieldSpec.DataType.FLOAT)
+ .addSingleValueDimension(DOUBLE_COL, FieldSpec.DataType.DOUBLE)
+ .addSingleValueDimension(BIG_DECIMAL_COL, FieldSpec.DataType.BIG_DECIMAL)
+ .addSingleValueDimension(BYTES_COL, FieldSpec.DataType.BYTES)
+ .addMultiValueDimension(MV_INT_COL, FieldSpec.DataType.INT)
+ .addMultiValueDimension(MV_STRING_COL, FieldSpec.DataType.STRING)
+ .addDateTime(TIME_COL, FieldSpec.DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS")
+ .build();
+
+ // Create extended schema with additional columns
+ _extendedSchema = new Schema.SchemaBuilder()
+ .addSingleValueDimension(STRING_COL_1, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(STRING_COL_2, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(INT_COL_1, FieldSpec.DataType.INT)
+ .addSingleValueDimension(INT_COL_2, FieldSpec.DataType.INT)
+ .addSingleValueDimension(LONG_COL, FieldSpec.DataType.LONG)
+ .addSingleValueDimension(FLOAT_COL, FieldSpec.DataType.FLOAT)
+ .addSingleValueDimension(DOUBLE_COL, FieldSpec.DataType.DOUBLE)
+ .addSingleValueDimension(BIG_DECIMAL_COL, FieldSpec.DataType.BIG_DECIMAL)
+ .addSingleValueDimension(BYTES_COL, FieldSpec.DataType.BYTES)
+ .addMultiValueDimension(MV_INT_COL, FieldSpec.DataType.INT)
+ .addMultiValueDimension(MV_STRING_COL, FieldSpec.DataType.STRING)
+ .addDateTime(TIME_COL, FieldSpec.DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS")
+ .addSingleValueDimension(NEW_STRING_COL, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(NEW_INT_COL, FieldSpec.DataType.INT)
+ .build();
+
+ // Create table config
+ _tableConfig = new TableConfigBuilder(TableType.OFFLINE)
+ .setTableName(TABLE_NAME)
+ .setTimeColumnName(TIME_COL)
+ .setInvertedIndexColumns(Lists.newArrayList(STRING_COL_1, INT_COL_1))
Review Comment:
can we also add a few more indexes on some columns, mark some columns as no dictionary to increase comprehensiveness of test
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/creator/impl/ColumnarSegmentBuildingTest.java:
##########
@@ -0,0 +1,500 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl;
+
+import com.google.common.collect.Lists;
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.segment.readers.PinotSegmentColumnReaderFactory;
+import org.apache.pinot.segment.spi.ImmutableSegment;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.ColumnReader;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.data.readers.RecordReader;
+import org.apache.pinot.spi.data.readers.RecordReaderConfig;
+import org.apache.pinot.spi.utils.ReadMode;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+
+/**
+ * Tests for columnar segment building functionality.
+ *
+ * <p>This test class validates the new columnar segment building capability including:
+ * <ul>
+ * <li>ColumnReader interface implementations</li>
+ * <li>ColumnReaderFactory for different data sources</li>
+ * <li>Columnar segment building vs row-major building equivalence</li>
+ * <li>Support for new columns with default values</li>
+ * </ul>
+ */
+public class ColumnarSegmentBuildingTest {
+ private static final String TEMP_DIR = System.getProperty("java.io.tmpdir");
+ private static final String TABLE_NAME = "testTable";
+ private static final String SEGMENT_NAME = "testSegment";
+
+ // Test columns
+ private static final String STRING_COL_1 = "stringCol1";
+ private static final String STRING_COL_2 = "stringCol2";
+ private static final String INT_COL_1 = "intCol1";
+ private static final String INT_COL_2 = "intCol2";
+ private static final String LONG_COL = "longCol";
+ private static final String FLOAT_COL = "floatCol";
+ private static final String DOUBLE_COL = "doubleCol";
+ private static final String BIG_DECIMAL_COL = "bigDecimalCol";
+ private static final String BYTES_COL = "bytesCol";
+ private static final String TIME_COL = "timeCol";
+ private static final String MV_INT_COL = "mvIntCol";
+ private static final String MV_STRING_COL = "mvStringCol";
+
+ // New column for testing default value handling
+ private static final String NEW_STRING_COL = "newStringCol";
+ private static final String NEW_INT_COL = "newIntCol";
+
+ private File _tempDir;
+ private Schema _originalSchema;
+ private Schema _extendedSchema; // Schema with additional columns
+ private TableConfig _tableConfig;
+ private List<GenericRow> _testData;
+
+ @BeforeClass
+ public void setUp() throws IOException {
+ _tempDir = new File(TEMP_DIR, "ColumnarSegmentBuildingTest");
+ FileUtils.deleteQuietly(_tempDir);
+ _tempDir.mkdirs();
+
+ // Create original schema
+ _originalSchema = new Schema.SchemaBuilder()
+ .addSingleValueDimension(STRING_COL_1, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(STRING_COL_2, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(INT_COL_1, FieldSpec.DataType.INT)
+ .addSingleValueDimension(INT_COL_2, FieldSpec.DataType.INT)
+ .addSingleValueDimension(LONG_COL, FieldSpec.DataType.LONG)
+ .addSingleValueDimension(FLOAT_COL, FieldSpec.DataType.FLOAT)
+ .addSingleValueDimension(DOUBLE_COL, FieldSpec.DataType.DOUBLE)
+ .addSingleValueDimension(BIG_DECIMAL_COL, FieldSpec.DataType.BIG_DECIMAL)
+ .addSingleValueDimension(BYTES_COL, FieldSpec.DataType.BYTES)
+ .addMultiValueDimension(MV_INT_COL, FieldSpec.DataType.INT)
+ .addMultiValueDimension(MV_STRING_COL, FieldSpec.DataType.STRING)
+ .addDateTime(TIME_COL, FieldSpec.DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS")
+ .build();
+
+ // Create extended schema with additional columns
+ _extendedSchema = new Schema.SchemaBuilder()
+ .addSingleValueDimension(STRING_COL_1, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(STRING_COL_2, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(INT_COL_1, FieldSpec.DataType.INT)
+ .addSingleValueDimension(INT_COL_2, FieldSpec.DataType.INT)
+ .addSingleValueDimension(LONG_COL, FieldSpec.DataType.LONG)
+ .addSingleValueDimension(FLOAT_COL, FieldSpec.DataType.FLOAT)
+ .addSingleValueDimension(DOUBLE_COL, FieldSpec.DataType.DOUBLE)
+ .addSingleValueDimension(BIG_DECIMAL_COL, FieldSpec.DataType.BIG_DECIMAL)
+ .addSingleValueDimension(BYTES_COL, FieldSpec.DataType.BYTES)
+ .addMultiValueDimension(MV_INT_COL, FieldSpec.DataType.INT)
+ .addMultiValueDimension(MV_STRING_COL, FieldSpec.DataType.STRING)
+ .addDateTime(TIME_COL, FieldSpec.DataType.LONG, "1:MILLISECONDS:EPOCH", "1:MILLISECONDS")
+ .addSingleValueDimension(NEW_STRING_COL, FieldSpec.DataType.STRING)
+ .addSingleValueDimension(NEW_INT_COL, FieldSpec.DataType.INT)
+ .build();
+
+ // Create table config
+ _tableConfig = new TableConfigBuilder(TableType.OFFLINE)
+ .setTableName(TABLE_NAME)
+ .setTimeColumnName(TIME_COL)
+ .setInvertedIndexColumns(Lists.newArrayList(STRING_COL_1, INT_COL_1))
+ .setSortedColumn(INT_COL_1)
+ .build();
+
+ // Generate test data
+ _testData = generateTestData(100);
+ }
+
+ @AfterClass
+ public void tearDown() throws IOException {
+ FileUtils.deleteQuietly(_tempDir);
+ }
+
+ @Test
+ public void testBasicColumnarBuilding() throws Exception {
+ // First create a segment using traditional row-major approach
+ File rowMajorSegmentDir = createRowMajorSegment();
+
+ // Then create a segment using columnar approach from the row-major segment
+ File columnarSegmentDir = createColumnarSegment(rowMajorSegmentDir);
+
+ // Validate that both segments have identical data
+ validateSegmentsIdentical(rowMajorSegmentDir, columnarSegmentDir);
+ }
+
+ @Test
+ public void testColumnarBuildingWithNewColumns() throws Exception {
+ // Create original segment with original schema
+ File originalSegmentDir = createRowMajorSegment();
+
+ // Create new segment with extended schema (has additional columns)
+ File newSegmentDir = createColumnarSegmentWithNewColumns(originalSegmentDir);
+
+ // Validate that the new segment has the additional columns with default values
+ validateSegmentWithNewColumns(newSegmentDir);
+ }
+
+ @Test
+ public void testColumnReaderFactory() throws Exception {
+ // Create a segment to test column reader factory with
+ File segmentDir = createRowMajorSegment();
+ ImmutableSegment segment = ImmutableSegmentLoader.load(segmentDir, ReadMode.mmap);
+
+ try {
+ // Test PinotSegmentColumnReaderFactory
+ try (PinotSegmentColumnReaderFactory factory = new PinotSegmentColumnReaderFactory(segment)) {
+ factory.init(_originalSchema);
+
+ // Test that all expected columns are available
+ Set<String> availableColumns = factory.getAvailableColumns();
+ Assert.assertTrue(availableColumns.contains(STRING_COL_1));
+ Assert.assertTrue(availableColumns.contains(INT_COL_1));
+ Assert.assertTrue(availableColumns.contains(MV_INT_COL));
+
+ // Test creating individual column readers
+ ColumnReader stringReader = factory.createColumnReader(STRING_COL_1,
+ _originalSchema.getFieldSpecFor(STRING_COL_1));
+ Assert.assertEquals(stringReader.getColumnName(), STRING_COL_1);
+
+ ColumnReader mvReader = factory.createColumnReader(MV_INT_COL,
+ _originalSchema.getFieldSpecFor(MV_INT_COL));
+ Assert.assertEquals(mvReader.getColumnName(), MV_INT_COL);
+
+ // Test reading values using iterator pattern
+ Assert.assertTrue(stringReader.hasNext());
+ Object firstStringValue = stringReader.next();
+ Assert.assertEquals(firstStringValue, "string1_0");
+
+ // Test that we can continue reading
+ Assert.assertTrue(stringReader.hasNext());
+ Object secondStringValue = stringReader.next();
+ Assert.assertEquals(secondStringValue, "string1_1");
+
+ // Reset and test again
+ stringReader.rewind();
+ Assert.assertTrue(stringReader.hasNext());
+ firstStringValue = stringReader.next();
+ Assert.assertEquals(firstStringValue, "string1_0");
+
+ // Test creating all column readers
+ Map<String, ColumnReader> allReaders = factory.getAllColumnReaders();
+ Assert.assertEquals(allReaders.size(), _originalSchema.getPhysicalColumnNames().size());
+ }
+ } finally {
+ segment.destroy();
+ }
+ }
+
+ @Test
+ public void testColumnReaderWithNewColumns() throws Exception {
+ // Create a segment to test with
+ File segmentDir = createRowMajorSegment();
+ ImmutableSegment segment = ImmutableSegmentLoader.load(segmentDir, ReadMode.mmap);
+
+ try {
+ // Test creating readers for new columns (should return default value readers)
+ try (PinotSegmentColumnReaderFactory factory = new PinotSegmentColumnReaderFactory(segment)) {
+ factory.init(_extendedSchema);
+
+ // Test creating reader for new column
+ ColumnReader newStringReader = factory.createColumnReader(NEW_STRING_COL,
+ _extendedSchema.getFieldSpecFor(NEW_STRING_COL));
+ Assert.assertEquals(newStringReader.getColumnName(), NEW_STRING_COL);
+
+ // Verify it returns default values using iterator pattern
+ Assert.assertTrue(newStringReader.hasNext());
+ Object defaultValue = newStringReader.next();
+ Assert.assertEquals(defaultValue, _extendedSchema.getFieldSpecFor(NEW_STRING_COL).getDefaultNullValue());
+
+ // Test that all values are the same (default)
+ int valueCount = 1; // We already read one value
+ while (newStringReader.hasNext()) {
+ Object value = newStringReader.next();
+ Assert.assertEquals(value, defaultValue);
+ valueCount++;
+ }
+ Assert.assertEquals(valueCount, _testData.size());
+ }
+ } finally {
+ segment.destroy();
+ }
+ }
+
+ @Test
+ public void testAllDataTypes() throws Exception {
+ // This test validates that all supported data types work correctly with columnar building
+ File rowMajorSegmentDir = createRowMajorSegment();
+ File columnarSegmentDir = createColumnarSegment(rowMajorSegmentDir);
+
+ // Validate that both segments have identical data for all data types
+ validateSegmentsIdentical(rowMajorSegmentDir, columnarSegmentDir);
+
+ // Additionally validate that all expected columns and data types are present
+ ImmutableSegment segment = ImmutableSegmentLoader.load(columnarSegmentDir, ReadMode.mmap);
+ try {
+ Set<String> columnNames = segment.getPhysicalColumnNames();
+
+ // Validate all data types are present
+ Assert.assertTrue(columnNames.contains(STRING_COL_1), "STRING column missing");
+ Assert.assertTrue(columnNames.contains(INT_COL_1), "INT column missing");
+ Assert.assertTrue(columnNames.contains(LONG_COL), "LONG column missing");
+ Assert.assertTrue(columnNames.contains(FLOAT_COL), "FLOAT column missing");
+ Assert.assertTrue(columnNames.contains(DOUBLE_COL), "DOUBLE column missing");
+ Assert.assertTrue(columnNames.contains(BIG_DECIMAL_COL), "BIG_DECIMAL column missing");
+ Assert.assertTrue(columnNames.contains(BYTES_COL), "BYTES column missing");
+ Assert.assertTrue(columnNames.contains(MV_INT_COL), "Multi-value INT column missing");
+ Assert.assertTrue(columnNames.contains(MV_STRING_COL), "Multi-value STRING column missing");
+ Assert.assertTrue(columnNames.contains(TIME_COL), "TIME column missing");
+ } finally {
+ segment.destroy();
+ }
+ }
+
+
+
+ private File createRowMajorSegment() throws Exception {
+ File outputDir = new File(_tempDir, "rowMajorSegment");
+ FileUtils.deleteQuietly(outputDir);
+ outputDir.mkdirs();
+
+ SegmentGeneratorConfig config = new SegmentGeneratorConfig(_tableConfig, _originalSchema);
+ config.setOutDir(outputDir.getAbsolutePath());
+ config.setSegmentName(SEGMENT_NAME + "_rowMajor");
+
+ SegmentIndexCreationDriverImpl driver = new SegmentIndexCreationDriverImpl();
+ TestRecordReader recordReader = new TestRecordReader(_testData);
+ driver.init(config, recordReader);
+ driver.build();
+
+ return new File(outputDir, SEGMENT_NAME + "_rowMajor");
+ }
+
+ private File createColumnarSegment(File sourceSegmentDir) throws Exception {
+ File outputDir = new File(_tempDir, "columnarSegment");
+ FileUtils.deleteQuietly(outputDir);
+ outputDir.mkdirs();
+
+ // Load the source segment
+ ImmutableSegment sourceSegment = ImmutableSegmentLoader.load(sourceSegmentDir, ReadMode.mmap);
+
+ try {
+ SegmentGeneratorConfig config = new SegmentGeneratorConfig(_tableConfig, _originalSchema);
+ config.setOutDir(outputDir.getAbsolutePath());
+ config.setSegmentName(SEGMENT_NAME + "_columnar");
+
+ SegmentIndexCreationDriverImpl driver = new SegmentIndexCreationDriverImpl();
+
+ // Use the new columnar building approach with config
+ try (PinotSegmentColumnReaderFactory factory = new PinotSegmentColumnReaderFactory(sourceSegment)) {
+ driver.init(config, factory);
+ driver.build();
+ }
+
+ return new File(outputDir, SEGMENT_NAME + "_columnar");
+ } finally {
+ sourceSegment.destroy();
+ }
+ }
+
+ private File createColumnarSegmentWithNewColumns(File sourceSegmentDir) throws Exception {
+ File outputDir = new File(_tempDir, "columnarSegmentWithNewColumns");
+ FileUtils.deleteQuietly(outputDir);
+ outputDir.mkdirs();
+
+ // Load the source segment
+ ImmutableSegment sourceSegment = ImmutableSegmentLoader.load(sourceSegmentDir, ReadMode.mmap);
+
+ try {
+ // Use extended schema with new columns
+ SegmentGeneratorConfig config = new SegmentGeneratorConfig(_tableConfig, _extendedSchema);
+ config.setOutDir(outputDir.getAbsolutePath());
+ config.setSegmentName(SEGMENT_NAME + "_withNewColumns");
+
+ SegmentIndexCreationDriverImpl driver = new SegmentIndexCreationDriverImpl();
+
+ // Use columnar building with extended schema
+ try (PinotSegmentColumnReaderFactory factory = new PinotSegmentColumnReaderFactory(sourceSegment)) {
+ driver.init(config, factory);
+ driver.build();
+ }
+
+ return new File(outputDir, SEGMENT_NAME + "_withNewColumns");
+ } finally {
+ sourceSegment.destroy();
+ }
+ }
+
+ private void validateSegmentsIdentical(File segment1Dir, File segment2Dir) throws Exception {
+ ImmutableSegment segment1 = ImmutableSegmentLoader.load(segment1Dir, ReadMode.mmap);
+ ImmutableSegment segment2 = ImmutableSegmentLoader.load(segment2Dir, ReadMode.mmap);
+
+ try {
+ // Validate metadata
+ Assert.assertEquals(segment1.getSegmentMetadata().getTotalDocs(), segment2.getSegmentMetadata().getTotalDocs());
+ Assert.assertEquals(segment1.getSegmentMetadata().getAllColumns(), segment2.getSegmentMetadata().getAllColumns());
+
+ // Validate data for each column
+ for (String columnName : segment1.getPhysicalColumnNames()) {
+ validateColumnData(segment1, segment2, columnName);
+ }
+ } finally {
+ segment1.destroy();
+ segment2.destroy();
+ }
+ }
+
+ private void validateSegmentWithNewColumns(File segmentDir) throws Exception {
+ ImmutableSegment segment = ImmutableSegmentLoader.load(segmentDir, ReadMode.mmap);
+
+ try {
+ // Validate that new columns exist
+ Assert.assertTrue(segment.getPhysicalColumnNames().contains(NEW_STRING_COL));
+ Assert.assertTrue(segment.getPhysicalColumnNames().contains(NEW_INT_COL));
+
+ // Validate that new columns have default values
+ GenericRow row = new GenericRow();
+ for (int docId = 0; docId < segment.getSegmentMetadata().getTotalDocs(); docId++) {
+ segment.getRecord(docId, row);
+
+ // Check default values
+ Assert.assertEquals(row.getValue(NEW_STRING_COL),
+ _extendedSchema.getFieldSpecFor(NEW_STRING_COL).getDefaultNullValue());
+ Assert.assertEquals(row.getValue(NEW_INT_COL),
+ _extendedSchema.getFieldSpecFor(NEW_INT_COL).getDefaultNullValue());
+ }
+ } finally {
+ segment.destroy();
+ }
+ }
+
+ private void validateColumnData(ImmutableSegment segment1, ImmutableSegment segment2, String columnName) {
+ int numDocs = segment1.getSegmentMetadata().getTotalDocs();
+
+ GenericRow row1 = new GenericRow();
+ GenericRow row2 = new GenericRow();
+
+ for (int docId = 0; docId < numDocs; docId++) {
+ segment1.getRecord(docId, row1);
+ segment2.getRecord(docId, row2);
+
+ Object value1 = row1.getValue(columnName);
+ Object value2 = row2.getValue(columnName);
+
+ Assert.assertEquals(value1, value2,
+ String.format("Column %s differs at docId %d: %s vs %s", columnName, docId, value1, value2));
+ }
+ }
+
+ private List<GenericRow> generateTestData(int numRows) {
+ List<GenericRow> data = new ArrayList<>();
+
+ for (int i = 0; i < numRows; i++) {
+ GenericRow row = new GenericRow();
+ row.putValue(STRING_COL_1, "string1_" + i);
+ row.putValue(STRING_COL_2, "string2_" + (i % 10));
+ row.putValue(INT_COL_1, i);
Review Comment:
Currently, the input data is sorted on the sort column INT_COL_1.
we can add a test with behaviour with unsorted input data too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] column major segment build for columnar datasource [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on code in PR #16727:
URL: https://github.com/apache/pinot/pull/16727#discussion_r2441677879
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentIndexCreationDriverImpl.java:
##########
@@ -677,4 +706,63 @@ public int getSkippedRowsFound() {
public int getSanitizedRowsFound() {
return _sanitizedRowsFound;
}
+
+ /**
+ * Build segment using columnar approach.
+ * This method builds the segment by processing data column-wise instead of row-wise.
+ *
+ * @throws Exception if segment building fails
+ */
+ public void buildColumnar() throws Exception {
+ if (!(_dataSource instanceof ColumnarSegmentCreationDataSource)) {
+ throw new IllegalStateException("buildColumnar() can only be called after initColumnar()");
+ }
+
+ ColumnarSegmentCreationDataSource columnarDataSource = (ColumnarSegmentCreationDataSource) _dataSource;
+ Map<String, ColumnReader> columnReaders = columnarDataSource.getColumnReaders();
+
+ LOGGER.info("Starting columnar segment building for {} columns", columnReaders.size());
+
+ // Reuse existing stats collection and index creation info logic
+ LOGGER.debug("Start building StatsCollector!");
+ collectStatsAndIndexCreationInfo();
+ LOGGER.info("Finished building StatsCollector!");
+ LOGGER.info("Collected stats for {} documents", _totalDocs);
+
+ if (_totalDocs == 0) {
+ LOGGER.warn("No documents found in data source");
+ handlePostCreation();
+ return;
+ }
+
+ try {
+ // Initialize the index creation using the per-column statistics information
+ _indexCreator.init(_config, _segmentIndexCreationInfo, _indexCreationInfoMap, _dataSchema, _tempIndexDir, null);
+
+ // Build the indexes column-wise (true column-major approach)
+ LOGGER.info("Start building Index using columnar approach");
+ long indexStartTime = System.nanoTime();
+
+ TreeSet<String> columns = _dataSchema.getPhysicalColumnNames();
+ for (String columnName : columns) {
+ LOGGER.debug("Indexing column: {}", columnName);
+ ColumnReader columnReader = columnReaders.get(columnName);
+ if (columnReader == null) {
+ throw new IllegalStateException("No column reader found for column: " + columnName);
+ }
+
+ // Index each column independently using true column-major approach
+ // This is similar to how buildByColumn works but uses ColumnReader instead of IndexSegment
+ ((SegmentColumnarIndexCreator) _indexCreator).indexColumn(columnName, columnReader);
+ }
+
+ _totalIndexTimeNs = System.nanoTime() - indexStartTime;
+ } catch (Exception e) {
+ _indexCreator.close();
+ throw e;
+ }
Review Comment:
makes sense, expanding the try catch and adding the close call in finally block
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org