You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "itschrispeck (via GitHub)" <gi...@apache.org> on 2023/06/29 20:54:27 UTC
[GitHub] [pinot] itschrispeck opened a new issue, #11004: Follower server redownloads segments upon server restart due to CRC change
itschrispeck opened a new issue, #11004:
URL: https://github.com/apache/pinot/issues/11004
When restarting servers in our clusters running on 0.12+, we observe numerous Helix pending messages due to `Failed to Load LLC Segment`. This is caused by CRC mismatch between ZK and the starting server.
Initial investigation showed that the 'leader' server commits and updates ZK, while the 'follower' server catches up and builds the segment locally with a different CRC. When a server is restarted all the segments that it 'followed' fail to load and are redownloaded from our deep store. We've confirmed that startOffset/endOffset match and the difference between two segments lies in the `columns.psf` file.
Logs for a segment:
```
2023-06-29T13:45:25-07:00 [host] Adding segment: <segment_name> to table: <table_name>
2023-06-29T13:45:25-07:00 [host] Segment: <segment_name> of table: <table_name> has crc change from: 3828021013 to: 1125725625
2023-06-29T13:45:25-07:00 [host] Failed to load LLC segment: <segment_name>, downloading a new copy
```
This behavior isn't seen on our clusters running on a 0.11 base. Is it possible some non-deterministic was introduced in the segment build process?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #11004: Follower server redownloads segments upon server restart due to CRC change
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #11004:
URL: https://github.com/apache/pinot/issues/11004#issuecomment-1614221573
I think this issue also exists in 0.11. Here are 2 related fixes, which will be picked up in the next release: #10231 #10468
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] itschrispeck commented on issue #11004: Follower server redownloads segments upon server restart due to CRC change
Posted by "itschrispeck (via GitHub)" <gi...@apache.org>.
itschrispeck commented on issue #11004:
URL: https://github.com/apache/pinot/issues/11004#issuecomment-1638567703
You're right, I think it's coincidence that the clusters we saw issue with are running a newer version of Pinot. We've jumped ahead to a commit that includes both fixes and we're still seeing the behavior.
It looks like the lucene index may not be created deterministically also:
```
Binary files replica_1/json_data.lucene.index/_0.cfe and replica_2/json_data.lucene.index/_0.cfe differ
72 | Binary files replica_1/json_data.lucene.index/_0.cfs and replica_2/json_data.lucene.index/_0.cfs differ
73 | Binary files replica_1/json_data.lucene.index/_0.si and replica_2/json_data.lucene.index/_0.si differ
74 | Binary files replica_1/json_data.lucene.index/segments_1 and replica_2/json_data.lucene.index/segments_1 differ
75 | Binary files replica_1/message_dictionaryVars.lucene.index/_0.cfe and replica_2/message_dictionaryVars.lucene.index/_0.cfe differ
76 | Binary files replica_1/message_dictionaryVars.lucene.index/_0.cfs and replica_2/message_dictionaryVars.lucene.index/_0.cfs differ
77 | Binary files replica_1/message_dictionaryVars.lucene.index/_0.si and replica_2/message_dictionaryVars.lucene.index/_0.si differ
78 | Binary files replica_1/message_dictionaryVars.lucene.index/segments_1 and replica_2/message_dictionaryVars.lucene.index/segments_1 differ
79 | Binary files replica_1/message_logtype.lucene.index/_0.cfe and replica_2/message_logtype.lucene.index/_0.cfe differ
80 | Binary files replica_1/message_logtype.lucene.index/_0.cfs and replica_2/message_logtype.lucene.index/_0.cfs differ
81 | Binary files replica_1/message_logtype.lucene.index/_0.si and replica_2/message_logtype.lucene.index/_0.si differ
82 | Binary files replica_1/message_logtype.lucene.index/segments_1 and replica_2/message_logtype.lucene.index/segments_1 differ
83
```
And I think this issue would also present if we use certain transform functions (e.g. storing ingestion time via `now()`).
What necessitates computing the CRC for all segments each restart? Assuming no data corruption happened, it seems that any data difference was already being 'served' as valid data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #11004: Follower server redownloads segments upon server restart due to CRC change
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #11004:
URL: https://github.com/apache/pinot/issues/11004#issuecomment-1644658468
Good point on the indeterministic index and functions. We should consider always downloading from deep-store for non-committing servers, or fetch CRC from ZK and set it into the local segment (this is kind of hacky)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] chenboat commented on issue #11004: Follower server redownloads segments upon server restart due to CRC change
Posted by "chenboat (via GitHub)" <gi...@apache.org>.
chenboat commented on issue #11004:
URL: https://github.com/apache/pinot/issues/11004#issuecomment-1613871058
@Jackie-Jiang @mayankshriv
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org