You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/11/01 13:30:58 UTC
[GitHub] [pinot] bdstuart opened a new issue, #9698: RealtimeToOfflineSegmentsTask fails if the OFFLINE table has a Timestamp Index
bdstuart opened a new issue, #9698:
URL: https://github.com/apache/pinot/issues/9698
-- RealtimeToOfflineSegmentTask config
"task": {
"taskTypeConfigsMap": {
"RealtimeToOfflineSegmentsTask": {
"bucketTimePeriod": "1d",
"bufferTimePeriod": "1d",
"mergeType": "concat",
"maxNumRecordsPerSegment": "5000000",
"schedule": "$REALTIME_TO_OFFLINE_SEGMENT_TASK_SCHEDULE"
}
}
}
-- Timestamp index config in both REALTIME and OFFLINE tables
"fieldConfigList": [
{
"name": "event_timestamp",
"encodingType": "DICTIONARY",
"indexType": "TIMESTAMP",
"indexTypes": [
"TIMESTAMP"
],
"timestampConfig": {
"granularities": [
"HOUR",
"DAY",
"WEEK",
"MONTH"
]
}
}
]
-- datetimeFieldSpecs
"dateTimeFieldSpecs":
[
{
"name": "event_timestamp",
"dataType": "TIMESTAMP",
"format" : "1:MILLISECONDS:TIMESTAMP",
"granularity": "1:MINUTES"
}
]
-- Stack trace
Caught exception while executing task: Task_RealtimeToOfflineSegmentsTask_653223a4-b56a-4625-a899-c956b3ed77f0_1667244900023_0
java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
at org.apache.pinot.segment.local.segment.creator.impl.stats.LongColumnPreIndexStatsCollector.getMinValue(LongColumnPreIndexStatsCollector.java:71) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936
at org.apache.pinot.segment.local.segment.creator.impl.stats.LongColumnPreIndexStatsCollector.getMinValue(LongColumnPreIndexStatsCollector.java:27) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936
at org.apache.pinot.segment.spi.creator.ColumnIndexCreationInfo.getMin(ColumnIndexCreationInfo.java:55) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator.init(SegmentColumnarIndexCreator.java:205) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:216) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.core.segment.processing.framework.SegmentProcessorFramework.process(SegmentProcessorFramework.java:154) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.plugin.minion.tasks.realtimetoofflinesegments.RealtimeToOfflineSegmentsTaskExecutor.convert(RealtimeToOfflineSegmentsTaskExecutor.java:163) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422
at org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.executeTask(BaseMultipleSegmentsConversionExecutor.java:165) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056
at org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.executeTask(BaseMultipleSegmentsConversionExecutor.java:62) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f
at org.apache.pinot.minion.taskfactory.TaskFactoryRegistry$1.runInternal(TaskFactoryRegistry.java:113) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.minion.taskfactory.TaskFactoryRegistry$1.run(TaskFactoryRegistry.java:89) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.helix.task.TaskRunner.run(TaskRunner.java:75) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Task: Task_RealtimeToOfflineSegmentsTask_653223a4-b56a-4625-a899-c956b3ed77f0_1667244900023_0 completed in: 2015ms
If I remove that config from my offline table then the realtime to offline job works fine but I no longer get my timestamp index columns on my offline table. If I remove the timestamp config from offline and run a query using the timestamp $ columns, I also get this error:
There are 13 invalid segment/s. This usually means that they were created with an older schema. Please reload the table in order to refresh these segments to the new schema.
CC @Jackie-Jiang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang closed issue #9698: RealtimeToOfflineSegmentsTask fails if the OFFLINE table has a Timestamp Index
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang closed issue #9698: RealtimeToOfflineSegmentsTask fails if the OFFLINE table has a Timestamp Index
URL: https://github.com/apache/pinot/issues/9698
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org