You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "wirybeaver (via GitHub)" <gi...@apache.org> on 2024/04/02 04:05:49 UTC
[I] Speed up immutable json index build time [pinot]
wirybeaver opened a new issue, #12774:
URL: https://github.com/apache/pinot/issues/12774
previously have an internal discussion with the team on translating mutable json index onto the immutable json index to avoid repeatedly parsing and flatten json docs. But It was temporarily blocked due to the fact that flattened doc id of mutable / immutable segments are different and lack of some interfaces. https://github.com/apache/pinot/pull/12744 has adds the mapping of ids in the IndexCreationContext. Thus, I plan to implement the initial idea this month
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Speed up immutable json index build time [pinot]
Posted by "wirybeaver (via GitHub)" <gi...@apache.org>.
wirybeaver commented on issue #12774:
URL: https://github.com/apache/pinot/issues/12774#issuecomment-2041665938
since json index internally store the flatten doc id, it will be tricky during transformation but is still doable with additional information about the flatten doc id length per docID. The idea below.
Mutable Index
| docID | 0 | 1 | 2 | 3 | 4 |
|--------------------|---|---|---|---|----|
| flattenDocIDLength | 3 | 5 | 7 | 2 | 10 |
Let say the sortedDocID list in immutable index is [2, 3, 4, 1, 0].
| docID | 2 | 3 | 4 | 1 | 0 |
|--------------------|---|---|----|---|---|
| flattenDocIDLength | 7 | 2 | 10 | 5 | 3 |
let say we can compute the array sortPos to reflect the index position in the sortedDocID. Use the example above.
sortPos = [4, 3, 0, 1, 2]. Given the mutableDocID, we would know the position in the sortedDocID is sortPos[mutableDocID]. (assume the docid offset is 0 in the segment for easier explain)
The flattenDocID start offset of the originalDocID would be changed from prefixSumOfMutableDocID[originalDocID] to prefixSumOfSortedDocID[sortPos[originalDocID]].
Take the docID 3 for example, it's flatten ids falls in [3+5+7, 3+5+7+2) the mutable index. Would be shifted to [7, 7+2).
Given a flatten doc id of the mutable segment, we can use binary search over the prefixSumOfMutableDocID to induct the docID position in MutableSegment and then compute the flattenDocID in the immutable segment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Speed up immutable json index build time [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #12774: Speed up immutable json index build time
URL: https://github.com/apache/pinot/issues/12774
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Speed up immutable json index build time [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12774:
URL: https://github.com/apache/pinot/issues/12774#issuecomment-5075855342
This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Speed up immutable json index build time [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12774:
URL: https://github.com/apache/pinot/issues/12774#issuecomment-5403187403
This issue has been closed because it has been inactive for 180 days. If you believe this issue is still relevant, please feel free to reopen it or create a new issue. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org