You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/11/17 09:04:21 UTC

[GitHub] [pulsar] Renkai opened a new issue #8591: Configurable data source for offloaded messages

Renkai opened a new issue #8591:
URL: https://github.com/apache/pulsar/issues/8591


   Currently, if the data in pulsar was offloaded to the second storage layer, data can still exists in bookkeeper for a period of time, but the client will directly read data from the second layer. 
   
   This may lead to several problems:
   
   - Read from second layer have different performance characteristics, which may lead wrong estimate from users if they didn't know which layer they are reading.
   - The second layer may be managed by another team rather than Pulsar management team(for example, a independent HDFS management team), they may have independent quota or authority policy to users.
   - The second layer storage can be infinite in theory, if user set cursor to an error time in accident, it will cause a lot of resource waste.
   
   So it's better to make data source configurable if data exists in both layer.
   
   Maybe the below options are enough:
   
   - first layer only
   - first layer first
   - second layer only
   - second layer first
   
   We can make `second layer fist` as the default value, which will result to the same behavior with current version.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Renkai commented on issue #8591: Configurable data source for offloaded messages

Posted by GitBox <gi...@apache.org>.
Renkai commented on issue #8591:
URL: https://github.com/apache/pulsar/issues/8591#issuecomment-728883911


   I will try to implement this design.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] gaoran10 commented on issue #8591: Configurable data source for offloaded messages

Posted by GitBox <gi...@apache.org>.
gaoran10 commented on issue #8591:
URL: https://github.com/apache/pulsar/issues/8591#issuecomment-728883236


   LGTM


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Renkai commented on issue #8591: Configurable data source for offloaded messages

Posted by GitBox <gi...@apache.org>.
Renkai commented on issue #8591:
URL: https://github.com/apache/pulsar/issues/8591#issuecomment-731067631


   I posted a PIP draft at https://gist.github.com/Renkai/e5be927404fbfd8289e7703c55812b1c


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Renkai commented on issue #8591: Configurable data source for offloaded messages

Posted by GitBox <gi...@apache.org>.
Renkai commented on issue #8591:
URL: https://github.com/apache/pulsar/issues/8591#issuecomment-736969222


   PIP draft at google docs:
   https://docs.google.com/document/d/1IyTd3GyUjGacu9bP3M1R9YINMTPC8xzhRigApsCpzTo/edit#heading=h.irwvg5wh2y3i 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Renkai commented on issue #8591: Configurable data source for offloaded messages

Posted by GitBox <gi...@apache.org>.
Renkai commented on issue #8591:
URL: https://github.com/apache/pulsar/issues/8591#issuecomment-730029846


   > @Renkai Can you write a PIP for that?
   
   Of course, what should I supply to make this issue description a PIP?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Renkai closed issue #8591: Configurable data source for offloaded messages

Posted by GitBox <gi...@apache.org>.
Renkai closed issue #8591:
URL: https://github.com/apache/pulsar/issues/8591


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] sijie commented on issue #8591: Configurable data source for offloaded messages

Posted by GitBox <gi...@apache.org>.
sijie commented on issue #8591:
URL: https://github.com/apache/pulsar/issues/8591#issuecomment-729836479


   @Renkai Can you write a PIP for that?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Renkai commented on issue #8591: Configurable data source for offloaded messages

Posted by GitBox <gi...@apache.org>.
Renkai commented on issue #8591:
URL: https://github.com/apache/pulsar/issues/8591#issuecomment-738627075


   Official PIP: https://github.com/apache/pulsar/wiki/PIP-73%3A-Configurable-data-source-priority-for-message-reading


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org