You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/06/06 22:19:38 UTC

[GitHub] [pinot] klsince opened a new issue, #8843: Allow a server to host segments on multi storage tiers

klsince opened a new issue, #8843:
URL: https://github.com/apache/pinot/issues/8843

   As of today, [a table with multi-tiers](https://docs.pinot.apache.org/operators/operating-pinot/tiered-storage#tier-config) has to use different server pools to support multi storage tiers, and relies on SegmentRelocator to move the segments among those server pools to put them on the right storage tier. An example cluster setup may be like using a pool of servers with SSD as a tier and another pool of servers with HDD as a second tier. 
   
   But this model doesn't work when a server have both SSD and HDD, and wants to define multi tiers backed by them. 
   
   We'd like to address this issue. And one possible way is to implement a SegmentDirectoryLoader that's aware of multi data dirs; and enhance current way of tier migration to move segments across tiers even if they are not relocated to a different server, i.e. to be done on the same server locally.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] klsince commented on issue #8843: Allow a server to host segments on multi storage tiers

Posted by GitBox <gi...@apache.org>.
klsince commented on issue #8843:
URL: https://github.com/apache/pinot/issues/8843#issuecomment-1196277533

   found another related issue: https://github.com/apache/pinot/issues/6343


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] klsince commented on issue #8843: Allow a server to host segments on multi storage tiers

Posted by GitBox <gi...@apache.org>.
klsince commented on issue #8843:
URL: https://github.com/apache/pinot/issues/8843#issuecomment-1360410739

   The major functionality is completed. What's remaining is about how to cleanup orphan segments, which is a related problem but orthogonal, because orphan segments can happen in many situations other than tier migration, as detailed in the design doc above. I'll address that separately.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] klsince commented on issue #8843: Allow a server to host segments on multi storage tiers

Posted by GitBox <gi...@apache.org>.
klsince commented on issue #8843:
URL: https://github.com/apache/pinot/issues/8843#issuecomment-1211555892

   Proposed some ideas about how to tackle this issue in [design doc](https://docs.google.com/document/d/1B_8k2XGMrxfFAo_g-MEjEPGj84Xy8YW4-yUiZ2BQkuE/edit#heading=h.2ic29gs71g43). Feel free to comment.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on issue #8843: Allow a server to host segments on multi storage tiers

Posted by GitBox <gi...@apache.org>.
somandal commented on issue #8843:
URL: https://github.com/apache/pinot/issues/8843#issuecomment-1211398961

   hi, this issue had a design doc earlier, but now i see it's been removed and I don't seem to have permissions to access it anymore. Can you provide access?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] klsince commented on issue #8843: Allow a server to host segments on multi storage tiers

Posted by GitBox <gi...@apache.org>.
klsince commented on issue #8843:
URL: https://github.com/apache/pinot/issues/8843#issuecomment-1207067625

   A design doc about how to approach this issue, meanwhile adding the support for multi data dir for Pinot servers: https://docs.google.com/document/d/1B_8k2XGMrxfFAo_g-MEjEPGj84Xy8YW4-yUiZ2BQkuE/edit# 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org