You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "satnash (via GitHub)" <gi...@apache.org> on 2025/09/28 19:12:21 UTC
[I] Retention Manager not deleting segments [pinot]
satnash opened a new issue, #16913:
URL: https://github.com/apache/pinot/issues/16913
Question about retention manager's behavior:
I have a table whose dateTime field is defined as below:
`"dateTimeFieldSpecs": [
{ "name": "collection_time", "dataType": "LONG", "format": "1:SECONDS:EPOCH", "granularity": "1:MILLISECONDS" }
]`
The segment config is below:
`"segmentsConfig": {
"timeColumnName": "collection_time",
"timeType": "SECONDS",
"segmentPushType": "APPEND",
"replication": "2",
"retentionTimeUnit": "DAYS",
"retentionTimeValue": "7",
"segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy",
"peerSegmentDownloadScheme": "http"
},`
I see that the segments are not getting removed past 7 days. One issue i have is the incoming data sometimes has the data from source where NTP is not synced, hence they report timestamps in seconds as 147, 243, etc. All on 1970 Jan 01. So the metadata on segments have snaphots like below
`"stats__6__87__20250925T2024Z" : {
"segmentName" : "stats__6__87__20250925T2024Z",
"schemaName" : null,
"crc" : 2382557963,
"creationTimeMillis" : 1758837918668,
"creationTimeReadable" : "2025-09-25T22:05:18:668 UTC",
"timeColumn" : "collection_time",
"timeUnit" : "SECONDS",
"timeGranularitySec" : 1,
"startTimeMillis" : 79000,
"startTimeReadable" : "1970-01-01T00:01:19.000Z",
"endTimeMillis" : 1758983934000,
"endTimeReadable" : "2025-09-27T14:38:54.000Z",
"segmentVersion" : "v3",
"creatorName" : null,
"totalDocs" : 2652778,
"custom" : { },
"startOffset" : "560677999",
"endOffset" : "563330777",
"columns" : [ ],
"indexes" : { },
"star-tree-index" : null
},
`
1. Do I need to stay away from using time from sources ? or does the rentention manager only care about end time in which case I am wondering what else i need to be looking at ?
2. I read in the docs that granularity is for documentation purposes only. Technically that should also be 1:SECONDS since i wanted each row to be treated separately.
Any guidance here is much appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Segment time boundaries, retention manager segment deletion question [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #16913: Segment time boundaries, retention manager segment deletion question
URL: https://github.com/apache/pinot/issues/16913
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Segment time boundaries, retention manager segment deletion question [pinot]
Posted by "satnash (via GitHub)" <gi...@apache.org>.
satnash commented on issue #16913:
URL: https://github.com/apache/pinot/issues/16913#issuecomment-3347580312
Needed to be patient. Retention manager does delete segments once every 6 hours as expected. The issue is how segments are getting created. I see 2 day long segments that were closed in under 2 hours. One such example:
{
"segment.realtime.endOffset": "555890883",
"segment.start.time": "78000",
"segment.time.unit": "MILLISECONDS",
"segment.flush.threshold.size": "2031024",
"segment.realtime.startOffset": "553859859",
"segment.size.in.bytes": "209888137",
"segment.end.time": "1758940829000",
"segment.total.docs": "2031024",
"segment.realtime.numReplicas": "2",
"segment.creation.time": "1758821361131",
"segment.index.version": "v3",
"segment.crc": "2095163110",
"segment.realtime.status": "DONE",
"segment.download.url": "s3://pinot/kpi/kpi__0__108__20250925T1729Z"
}
Verified that i am not getting source timestamps into the future
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Segment time boundaries, retention manager segment deletion question [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #16913:
URL: https://github.com/apache/pinot/issues/16913#issuecomment-3398710752
> Do I need to stay away from using time from sources ? or does the rentention manager only care about end time in which case I am wondering what else i need to be looking at to ensure uniformity in segments size.
Retention manager only cares about end time, and removes segment only if the entire segment expires. In the segment metadata above, `segment.end.time` is larger than `segment.creation.time`, which means the source data is referencing a time in the future.
> I read in the docs that granularity is for documentation purposes only. Technically that should also be 1:SECONDS since i wanted each row to be treated separately.
You may configure that to `1:SECONDS`. What really matters is the `format`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Retention Manager not deleting segments [pinot]
Posted by "satnash (via GitHub)" <gi...@apache.org>.
satnash commented on issue #16913:
URL: https://github.com/apache/pinot/issues/16913#issuecomment-3344170668
I have refered this ticket, [before](https://github.com/apache/pinot/issues/6351) posting as I think my issue is slightly different, may yet be related.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Segment time boundaries, retention manager segment deletion question [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16913:
URL: https://github.com/apache/pinot/issues/16913#issuecomment-4317211976
This issue has been closed because it has been inactive for 180 days. If you believe this issue is still relevant, please feel free to reopen it or create a new issue. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Segment time boundaries, retention manager segment deletion question [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16913:
URL: https://github.com/apache/pinot/issues/16913#issuecomment-4113107177
This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org