You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/01/25 21:25:48 UTC
[GitHub] [pinot] abhijeetkushe opened a new issue #8070: GetRecord.IteratorAgeInMilliseconds metrics not being reflected and data not being queried in Table
abhijeetkushe opened a new issue #8070:
URL: https://github.com/apache/pinot/issues/8070
Table Config
```{
"tableName": "workflowEvents",
"tableType": "REALTIME",
"segmentsConfig": {
"timeColumnName": "eventTimestamp",
"timeType": "MILLISECONDS",
"schemaName": "workflowEvents",
"replicasPerPartition": "1",
"retentionTimeUnit": "DAYS",
"retentionTimeValue": "365",
"segmentPushType": "APPEND"
},
"tenants": {
"broker":"DefaultTenant",
"server":"DefaultTenant"
},
"tableIndexConfig": {
"loadMode": "MMAP",
"streamConfigs": {
"streamType": "kinesis",
"stream.kinesis.topic.name": "prod-rel-cdp-dl-workflow-metrics-stream",
"region": "us-east-1",
"shardIteratorType": "LATEST",
"stream.kinesis.consumer.type": "lowlevel",
"stream.kinesis.fetch.timeout.millis": "30000",
"stream.kinesis.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
"stream.kinesis.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kinesis.KinesisConsumerFactory",
"realtime.segment.flush.threshold.size": "5000000",
"realtime.segment.flush.threshold.time": "1d"
}
},
"upsertConfig": {
"mode": "FULL"
},
"routing": {
"instanceSelectorType": "strictReplicaGroup"
},
"metadata": {
"customConfigs": {}
}
}
```
- I upgraded pinot to 0.9.1 version from 0.8.0 last month and after that I started seeing GetRecord.IteratorAgeInMilliseconds metrics displaying in correct values
- Yesterday I observed the data stopped being displayed in the Table from the Query Console even though the GetRecords metrics showed that records were read but where not read from the Table.There were some new data fields that were added to the record but the schema was not updated
- I decided to change Table config to AT_SEQUENCE_NUMBER and I did not see the changes immediately but this morning I was able to see those records but post that when records were added this morning they were not being reflected in the table
- I decided to switch the Table config back to LATEST.I also saw that the old segments got terminated and new segments were created but the records were still not reflected in the Table
So both the data issue and GetRecord.IteratorAgeInMilliseconds still persist
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] abhijeetkushe edited a comment on issue #8070: GetRecord.IteratorAgeInMilliseconds metrics not being reflected and data not being queried in Table
Posted by GitBox <gi...@apache.org>.
abhijeetkushe edited a comment on issue #8070:
URL: https://github.com/apache/pinot/issues/8070#issuecomment-1022462519
I have more updates.I do see the data in the table but not realtime. shardIteratorType as LATEST.I ran the below query
`select $segmentName,ToDateTime(eventTimestamp,'yyyyMMddHHmm') from workflowEvents order by eventTimestamp desc limit 100 `
```
$segmentName todatetime(eventTimestamp,'yyyyMMddHHmm')
workflowEvents__0__58__20220125T2005Z 202201260000
workflowEvents__1__60__20220125T2106Z 202201260000
workflowEvents__1__60__20220125T2106Z 202201251640
workflowEvents__1__60__20220125T2106Z 202201251640
workflowEvents__1__60__20220125T2106Z 202201251640
workflowEvents__1__60__20220125T2106Z 202201251640
workflowEvents__1__60__20220125T2106Z 202201251640
workflowEvents__1__60__20220125T2106Z 202201251640
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] abhijeetkushe commented on issue #8070: GetRecord.IteratorAgeInMilliseconds metrics not being reflected and data not being queried in Table
Posted by GitBox <gi...@apache.org>.
abhijeetkushe commented on issue #8070:
URL: https://github.com/apache/pinot/issues/8070#issuecomment-1022486392
I also the records do ge read at that time
```16:43:05.029 INFO [LLRealtimeSegmentDataManager_workflowEvents__1__60__20220125T2106Z] [workflowEvents__1__60__20220125T2106Z] Consumed 12 events from (rate:0.19965394/s), currentOffset={"shardId-000000000001":"49622318632194742639402514501316476653895236003058155538"}, numRowsConsumedSoFar=43, numRowsIndexedSoFar=43```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] abhijeetkushe edited a comment on issue #8070: GetRecord.IteratorAgeInMilliseconds metrics not being reflected and data not being queried in Table
Posted by GitBox <gi...@apache.org>.
abhijeetkushe edited a comment on issue #8070:
URL: https://github.com/apache/pinot/issues/8070#issuecomment-1022486392
Records do get read at 1643 but I do not see corresponding events at 1543
```16:43:05.029 INFO [LLRealtimeSegmentDataManager_workflowEvents__1__60__20220125T2106Z] [workflowEvents__1__60__20220125T2106Z] Consumed 12 events from (rate:0.19965394/s), currentOffset={"shardId-000000000001":"49622318632194742639402514501316476653895236003058155538"}, numRowsConsumedSoFar=43, numRowsIndexedSoFar=43```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] abhijeetkushe commented on issue #8070: GetRecord.IteratorAgeInMilliseconds metrics not being reflected and data not being queried in Table
Posted by GitBox <gi...@apache.org>.
abhijeetkushe commented on issue #8070:
URL: https://github.com/apache/pinot/issues/8070#issuecomment-1022477831
Kinesis Get Records shows that it is read
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] KKcorps commented on issue #8070: GetRecord.IteratorAgeInMilliseconds metrics not being reflected and data not being queried in Table
Posted by GitBox <gi...@apache.org>.
KKcorps commented on issue #8070:
URL: https://github.com/apache/pinot/issues/8070#issuecomment-1021625120
Looking into it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] abhijeetkushe commented on issue #8070: GetRecord.IteratorAgeInMilliseconds metrics not being reflected and data not being queried in Table
Posted by GitBox <gi...@apache.org>.
abhijeetkushe commented on issue #8070:
URL: https://github.com/apache/pinot/issues/8070#issuecomment-1022462519
I have more updates.I do see the data in the table but not realtime. shardIteratorType as LATEST.I ran the below query
`select $segmentName,ToDateTime(eventTimestamp,'yyyyMMddHHmm') from workflowEvents order by eventTimestamp desc limit 100 `
$segmentName todatetime(eventTimestamp,'yyyyMMddHHmm')
workflowEvents__0__58__20220125T2005Z 202201260000
workflowEvents__1__60__20220125T2106Z 202201260000
workflowEvents__1__60__20220125T2106Z 202201251640
workflowEvents__1__60__20220125T2106Z 202201251640
workflowEvents__1__60__20220125T2106Z 202201251640
workflowEvents__1__60__20220125T2106Z 202201251640
workflowEvents__1__60__20220125T2106Z 202201251640
workflowEvents__1__60__20220125T2106Z 202201251640
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] abhijeetkushe removed a comment on issue #8070: GetRecord.IteratorAgeInMilliseconds metrics not being reflected and data not being queried in Table
Posted by GitBox <gi...@apache.org>.
abhijeetkushe removed a comment on issue #8070:
URL: https://github.com/apache/pinot/issues/8070#issuecomment-1022486392
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] abhijeetkushe commented on issue #8070: GetRecord.IteratorAgeInMilliseconds metrics not being reflected and data not being queried in Table
Posted by GitBox <gi...@apache.org>.
abhijeetkushe commented on issue #8070:
URL: https://github.com/apache/pinot/issues/8070#issuecomment-1021624582
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] abhijeetkushe edited a comment on issue #8070: GetRecord.IteratorAgeInMilliseconds metrics not being reflected and data not being queried in Table
Posted by GitBox <gi...@apache.org>.
abhijeetkushe edited a comment on issue #8070:
URL: https://github.com/apache/pinot/issues/8070#issuecomment-1022462519
I have more updates.I do see the data in the table but not realtime. shardIteratorType as LATEST.I ran the below query
`select $segmentName,ToDateTime(eventTimestamp,'yyyyMMddHHmm') from workflowEvents order by eventTimestamp desc limit 100 `
```
$segmentName todatetime(eventTimestamp,'yyyyMMddHHmm')
workflowEvents__0__58__20220125T2005Z 202201260000
workflowEvents__1__60__20220125T2106Z 202201260000
workflowEvents__1__60__20220125T2106Z 202201251640
workflowEvents__1__60__20220125T2106Z 202201251640
workflowEvents__1__60__20220125T2106Z 202201251640
workflowEvents__1__60__20220125T2106Z 202201251640
workflowEvents__1__60__20220125T2106Z 202201251640
workflowEvents__1__60__20220125T2106Z 202201251640
```
8 Records are sent at ~1543
Kinesis Get Records shows that it is read at 1643
From the server logs records do get read at 1643 but I do not see corresponding events at 1543
16:43:05.029 INFO [LLRealtimeSegmentDataManager_workflowEvents__1__60__20220125T2106Z] [workflowEvents__1__60__20220125T2106Z] Consumed 12 events from (rate:0.19965394/s), currentOffset={"shardId-000000000001":"49622318632194742639402514501316476653895236003058155538"}, numRowsConsumedSoFar=43, numRowsIndexedSoFar=43
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org