You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/02/10 18:36:12 UTC
[GitHub] [incubator-pinot] npawar commented on a change in pull request #6466: Handle creation of segments with 0 rows
npawar commented on a change in pull request #6466:
URL: https://github.com/apache/incubator-pinot/pull/6466#discussion_r573976548
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/indexsegment/immutable/ImmutableSegmentLoader.java
##########
@@ -105,32 +105,34 @@ public static ImmutableSegment load(File indexDir, IndexLoadingConfig indexLoadi
// Load the segment
ReadMode readMode = indexLoadingConfig.getReadMode();
SegmentDirectory segmentDirectory = SegmentDirectory.createFromLocalFS(indexDir, segmentMetadata, readMode);
- SegmentDirectory.Reader segmentReader = segmentDirectory.createReader();
Map<String, ColumnIndexContainer> indexContainerMap = new HashMap<>();
- for (Map.Entry<String, ColumnMetadata> entry : segmentMetadata.getColumnMetadataMap().entrySet()) {
- indexContainerMap.put(entry.getKey(),
- new PhysicalColumnIndexContainer(segmentReader, entry.getValue(), indexLoadingConfig, indexDir));
- }
+ StarTreeIndexContainer starTreeIndexContainer = null;
- // Instantiate virtual columns
- Schema segmentSchema = segmentMetadata.getSchema();
- VirtualColumnProviderFactory.addBuiltInVirtualColumnsToSegmentSchema(segmentSchema, segmentName);
- for (FieldSpec fieldSpec : segmentSchema.getAllFieldSpecs()) {
- if (fieldSpec.isVirtualColumn()) {
- String columnName = fieldSpec.getName();
- VirtualColumnContext context = new VirtualColumnContext(fieldSpec, segmentMetadata.getTotalDocs());
- VirtualColumnProvider provider = VirtualColumnProviderFactory.buildProvider(context);
- indexContainerMap.put(columnName, provider.buildColumnIndexContainer(context));
- segmentMetadata.getColumnMetadataMap().put(columnName, provider.buildMetadata(context));
+ if (segmentMetadata.getTotalDocs() > 0) {
Review comment:
If I don't add this special casing, I get this exceptions from forward index reader
```
java.lang.RuntimeException: java.lang.RuntimeException: Could not find index for column: studentID, type: FORWARD_INDEX, segment: /Users/npawar/quick_start_configs/pinot-quick-start/dataDir/data/transcript_REALTIME/transcript__0__1__20210210T1830Z/v3
at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.replaceLLSegment(RealtimeTableDataManager.java:481) ~[classes/:?]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.commitSegment(LLRealtimeSegmentDataManager.java:854) ~[classes/:?]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:615) [classes/:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
Caused by: java.lang.RuntimeException: Could not find index for column: studentID, type: FORWARD_INDEX, segment: /Users/npawar/quick_start_configs/pinot-quick-start/dataDir/data/transcript_REALTIME/transcript__0__1__20210210T1830Z/v3
at org.apache.pinot.core.segment.store.SingleFileIndexDirectory.checkAndGetIndexBuffer(SingleFileIndexDirectory.java:142) ~[classes/:?]
at org.apache.pinot.core.segment.store.SingleFileIndexDirectory.getBuffer(SingleFileIndexDirectory.java:101) ~[classes/:?]
at org.apache.pinot.core.segment.store.SegmentLocalFSDirectory.getIndexForColumn(SegmentLocalFSDirectory.java:229) ~[classes/:?]
at org.apache.pinot.core.segment.store.SegmentLocalFSDirectory.access$000(SegmentLocalFSDirectory.java:36) ~[classes/:?]
at org.apache.pinot.core.segment.store.SegmentLocalFSDirectory$Reader.getIndexFor(SegmentLocalFSDirectory.java:288) ~[classes/:?]
at org.apache.pinot.core.segment.index.column.PhysicalColumnIndexContainer.<init>(PhysicalColumnIndexContainer.java:134) ~[classes/:?]
at org.apache.pinot.core.indexsegment.immutable.ImmutableSegmentLoader.load(ImmutableSegmentLoader.java:114) ~[classes/:?]
at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.replaceLLSegment(RealtimeTableDataManager.java:479) ~[classes/:?]
... 3 more
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org