You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "eaugene (via GitHub)" <gi...@apache.org> on 2023/05/30 14:48:35 UTC
[GitHub] [pinot] eaugene opened a new issue, #10816: Segment Data mismatch
eaugene opened a new issue, #10816:
URL: https://github.com/apache/pinot/issues/10816
The Creation Time Millis value received for a realtime segment from `/segments/<table_name>/metadata` API is not aligning with the value received from`/segments/<table_name>/<segmant_name>/metadata` API . I assume they denote the time when the segment started consuming .
Value received for a segment from `/segments/<table_name>/metadata` :
```
{
"segmentName": "segment_name",
"schemaName": null,
"crc": 1641778138,
**"creationTimeMillis": 1685394183377,**
"creationTimeReadable": "2023-05-29T21:03:03:377 UTC",
"timeColumn": "timestamp",
"timeUnit": "SECONDS",
"timeGranularitySec": 1,
"startTimeMillis": 1685307699000,
"startTimeReadable": "2023-05-28T21:01:39.000Z",
"endTimeMillis": 1685394146000,
"endTimeReadable": "2023-05-29T21:02:26.000Z",
"segmentVersion": "v3",
"creatorName": null,
"totalDocs": 1946111,
"custom": {},
"startOffset": "1498385432",
"endOffset": "1500331543",
"columns": [],
"indexes": {},
"star-tree-index": null
},
```
Value received the same segment from `/segments/<table_name>/metadata` :
```
{
"segment.realtime.endOffset": "1500331543",
"segment.start.time": "1685307699000",
"segment.time.unit": "MILLISECONDS",
"segment.flush.threshold.size": "2500000",
"segment.realtime.startOffset": "1498385432",
"segment.end.time": "1685394146000",
"segment.total.docs": "1946111",
"segment.realtime.numReplicas": "2",
**"segment.creation.time": "1685307748436",**
"segment.index.version": "v3",
"segment.crc": "1641778138",
"segment.realtime.status": "DONE",
"segment.download.url": "<deepstore_path>"
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] eaugene commented on issue #10816: Segment MetaData mismatch between API's
Posted by "eaugene (via GitHub)" <gi...@apache.org>.
eaugene commented on issue #10816:
URL: https://github.com/apache/pinot/issues/10816#issuecomment-1573182830
Thanks @Jackie-Jiang for the clarification. Should we enhance the first API to have a different naming convention for this property, say "segmentCommitedTime" to make it more explicit, as it kind of confuses ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10816: Segment MetaData mismatch between API's
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10816:
URL: https://github.com/apache/pinot/issues/10816#issuecomment-1574388213
The creation time within the segment metadata (stored in the segment file) is the time when the segment is created. The creation time within the ZK metadata is the same as segment metadata for pushed segment, but different for consuming segment. We can consider always making the ZK metadata the same as segment metadata, or add a `segment.realtime.creation.time` to differentiate with the physical segment creation time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] eaugene commented on issue #10816: Segment MetaData mismatch between API's
Posted by "eaugene (via GitHub)" <gi...@apache.org>.
eaugene commented on issue #10816:
URL: https://github.com/apache/pinot/issues/10816#issuecomment-1576101176
> add a segment.realtime.creation.time to differentiate with the physical segment creation time
Thanks. This is more effective. I'll work on this change. Please assign this to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10816: Segment MetaData mismatch between API's
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10816:
URL: https://github.com/apache/pinot/issues/10816#issuecomment-1572809481
The first metadata is returned from the server local metadata; the second metadata is returned from the ZK.
The creation time for the server local metadata is when the segment is created (consuming segment being committed and sealed), while the creation time for the ZK metadata is when the consuming segment being created. That is the reason why you see about 1 day difference between these 2 timestamps
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org