You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/12/15 08:03:06 UTC

[GitHub] [iceberg] deadwind4 opened a new issue, #6429: [Feature Proposal] Log Store in Iceberg

deadwind4 opened a new issue, #6429:
URL: https://github.com/apache/iceberg/issues/6429

   ### Feature Request / Improvement
   
   This proposal aims to improve Iceberg's capability in real-time via importing a log store system. Streaming read data that are cached in a log store(Kafka).
   
   Proposal document:
   https://docs.google.com/document/d/1Zy4bbvSuN9ax7Tq4TuxL2vi4xQ4DR4wbR4QK5m69qQo/edit?usp=sharing
   
   Code demo:
   https://github.com/deadwind4/iceberg/tree/log-store
   
   ### Query engine
   
   None


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] bryanck commented on issue #6429: [Feature Proposal] Log Store in Iceberg

Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on issue #6429:
URL: https://github.com/apache/iceberg/issues/6429#issuecomment-1585987456

   I don't believe so. With the Kafka Connect sink, all files written and commits performed are logged as messages on a control topic. That acts as a log for write activity (at file-level grain), so a downstream process could consume from that without requiring an addition log.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] bitsondatadev commented on issue #6429: [Feature Proposal] Log Store in Iceberg

Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on issue #6429:
URL: https://github.com/apache/iceberg/issues/6429#issuecomment-1585932053

   @bryanck is there any overlap here with the kafka connect sink?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org