You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by GitBox <gi...@apache.org> on 2022/11/13 13:59:42 UTC

[GitHub] [rocketmq] hzh0425 opened a new issue, #5518: [Discussion] Support tiered storage for RocketMQ

hzh0425 opened a new issue, #5518:
URL: https://github.com/apache/rocketmq/issues/5518

   Hi, teams!
   At present, we store all topic data on the local disk of the Broker. If the amount of business data is very large, it may not be possible to store the full amount of data. 
   BTW, We prefer to store some old historical data on cheap disks, such as s3, and store the newer data on the local disk of the Broker.
   So should we let RocketMQ support tiered storage?
   Thanks to the storage features of RocketMQ, it should not be difficult to support tiered storage.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] lizhanhui commented on issue #5518: [Discussion] Support tiered storage for RocketMQ

Posted by GitBox <gi...@apache.org>.
lizhanhui commented on issue #5518:
URL: https://github.com/apache/rocketmq/issues/5518#issuecomment-1313135630

   @ShadowySpirits Would you please share the roadmap or insights?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] nowinkeyy commented on issue #5518: [Discussion] Support tiered storage for RocketMQ

Posted by GitBox <gi...@apache.org>.
nowinkeyy commented on issue #5518:
URL: https://github.com/apache/rocketmq/issues/5518#issuecomment-1312779187

   Hi @hzh0425 
   I think there was a discussion about that earlier, #5418 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] ShadowySpirits commented on issue #5518: [Discussion] Support tiered storage for RocketMQ

Posted by GitBox <gi...@apache.org>.
ShadowySpirits commented on issue #5518:
URL: https://github.com/apache/rocketmq/issues/5518#issuecomment-1357715536

   @azhsmesos Yeah. I'll open a new issue and submit the first pr later this week.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] ShadowySpirits commented on issue #5518: [Discussion] Support tiered storage for RocketMQ

Posted by GitBox <gi...@apache.org>.
ShadowySpirits commented on issue #5518:
URL: https://github.com/apache/rocketmq/issues/5518#issuecomment-1313210519

   Hi @hzh0425 @azhsmesos. We do have plans for tiered storage on our roadmap. But the relationship between tiered storage and compact topic also needs to be discussed. These two approaches have the same goal: both are to extend the message storage time and are suitable for different situations. We should decide how to evolve the storage of RocketMQ to be more suitable for the current architecture and positioning.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] hzh0425 commented on issue #5518: [Discussion] Support tiered storage for RocketMQ

Posted by GitBox <gi...@apache.org>.
hzh0425 commented on issue #5518:
URL: https://github.com/apache/rocketmq/issues/5518#issuecomment-1312738643

   Like [Kip--Kafka Tiered storage](https://cwiki-vm6.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] ShadowySpirits commented on issue #5518: [Discussion] Support tiered storage for RocketMQ

Posted by GitBox <gi...@apache.org>.
ShadowySpirits commented on issue #5518:
URL: https://github.com/apache/rocketmq/issues/5518#issuecomment-1322108059

   @hzh0425 @azhsmesos @lizhanhui I have submitted the [RIP-57](https://github.com/apache/rocketmq/wiki/RIP-57-Tiered-storage-for-RocketMQ) to implement tiered storage for RocketMQ. We can discuss it in [mail list](https://lists.apache.org/thread/nzp123b2syv719sznnt74kjbth5m6rto).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] ShadowySpirits commented on issue #5518: [Discussion] Support tiered storage for RocketMQ

Posted by GitBox <gi...@apache.org>.
ShadowySpirits commented on issue #5518:
URL: https://github.com/apache/rocketmq/issues/5518#issuecomment-1313486213

   > Cool, is there any channel to discuss this feature? @ShadowySpirits @lizhanhui
   
   We will submit a rip later.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] azhsmesos commented on issue #5518: [Discussion] Support tiered storage for RocketMQ

Posted by GitBox <gi...@apache.org>.
azhsmesos commented on issue #5518:
URL: https://github.com/apache/rocketmq/issues/5518#issuecomment-1356777760

   > @hzh0425 @azhsmesos @lizhanhui I have submitted the [RIP-57](https://github.com/apache/rocketmq/wiki/RIP-57-Tiered-storage-for-RocketMQ) to implement tiered storage for RocketMQ. We can discuss it in [mail list](https://lists.apache.org/thread/nzp123b2syv719sznnt74kjbth5m6rto).
   
   Cool! Is it already under development?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] azhsmesos commented on issue #5518: [Discussion] Support tiered storage for RocketMQ

Posted by GitBox <gi...@apache.org>.
azhsmesos commented on issue #5518:
URL: https://github.com/apache/rocketmq/issues/5518#issuecomment-1313143073

   I think it might be considered, especially for topic-compressed messages, to store the pre-compressed data on cheap storage media to enable queries over a longer time rangeļ¼Œ like [pip-pulsar Tiered storage](https://github.com/apache/pulsar/wiki/PIP-17:-Tiered-storage-for-Pulsar-topics)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] hzh0425 commented on issue #5518: [Discussion] Support tiered storage for RocketMQ

Posted by GitBox <gi...@apache.org>.
hzh0425 commented on issue #5518:
URL: https://github.com/apache/rocketmq/issues/5518#issuecomment-1313266854

   Cool, is there any channel to discuss this feature? @ShadowySpirits @lizhanhui 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] hzh0425 closed issue #5518: [Discussion] Support tiered storage for RocketMQ

Posted by GitBox <gi...@apache.org>.
hzh0425 closed issue #5518: [Discussion] Support tiered storage for RocketMQ
URL: https://github.com/apache/rocketmq/issues/5518


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org