You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "coderzc (via GitHub)" <gi...@apache.org> on 2023/07/25 03:39:20 UTC

[GitHub] [pulsar] coderzc opened a new pull request, #20867: PIP-286: Add `findNewestPosition` method in the `TopicCompactionService` API

coderzc opened a new pull request, #20867:
URL: https://github.com/apache/pulsar/pull/20867

   ### Documentation
   
   <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
   
   - [ ] `doc` <!-- Your PR contains doc changes. -->
   - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
   - [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
   - [ ] `doc-complete` <!-- Docs have been already added -->
   
   ### Matching PR in forked repository
   
   PR in forked repository: <!-- ENTER URL HERE -->
   
   <!--
   After opening this PR, the build in apache/pulsar will fail and instructions will
   be provided for opening a PR in the PR author's forked repository.
   
   apache/pulsar pull requests should be first tested in your own fork since the 
   apache/pulsar CI based on GitHub Actions has constrained resources and quota.
   GitHub Actions provides separate quota for pull requests that are executed in 
   a forked repository.
   
   The tests will be run in the forked repository until all PR review comments have
   been handled, the tests pass and the PR is approved by a reviewer.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] coderzc commented on pull request #20867: [improve][pip] PIP-286: Make the TopicCompactionService to support find entry based on publishTime or index

Posted by "coderzc (via GitHub)" <gi...@apache.org>.
coderzc commented on PR #20867:
URL: https://github.com/apache/pulsar/pull/20867#issuecomment-1734691028

   The vote has already closed, please see: https://lists.apache.org/thread/q27zg49mpr8otwh29s3sncdcx8ly7ws6


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] asafm commented on pull request #20867: [improve][pip] PIP-286: Add `findNewestPosition` method in the `TopicCompactionService` API

Posted by "asafm (via GitHub)" <gi...@apache.org>.
asafm commented on PR #20867:
URL: https://github.com/apache/pulsar/pull/20867#issuecomment-1651267239

   I understand, but to have holes than go back and forth.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] asafm commented on pull request #20867: [improve][pip] PIP-286: Add `findNewestPosition` method in the `TopicCompactionService` API

Posted by "asafm (via GitHub)" <gi...@apache.org>.
asafm commented on PR #20867:
URL: https://github.com/apache/pulsar/pull/20867#issuecomment-1651124948

   If your number is off. Newest one I see before you is 288


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] codelipenghui merged pull request #20867: [improve][pip] PIP-286: Make the TopicCompactionService to support find entry based on publishTime or index

Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui merged PR #20867:
URL: https://github.com/apache/pulsar/pull/20867


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] codelipenghui commented on a diff in pull request #20867: [improve][pip] PIP-286: Make the TopicCompactionService to support find entry based on publishTime or index

Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui commented on code in PR #20867:
URL: https://github.com/apache/pulsar/pull/20867#discussion_r1325206354


##########
pip/pip-286.md:
##########
@@ -0,0 +1,57 @@
+# Background knowledge
+
+In https://github.com/apache/pulsar/pull/11139 we support get position based on timestamp, but it doesn't work well with topic compaction enabled because the data may have been move to the compacted ledger.
+
+In [PIP-278](https://github.com/apache/pulsar/pull/20624) we introduced the pluggable topic compaction service to extend the compaction.
+
+# Motivation
+
+In order for `get-message-id` to work well with topic compaction enabled we need to find the position according to publish time from topic compaction service,
+but `TopicCompactionService` missing a method that find positions according to publish time or other metadata, so we should add it.
+
+In addition, this method can also be used to find the position/offset according to offset/timestamp in the KoP.
+
+# Goals
+
+# High Level Design
+
+We need to add a method to `Topic Compaction Service` that can find the matching position and other metadata information according to publishTime/index.
+
+# Detailed Design
+
+Add `findEntryByPublishTime` in the `TopicCompactionService` API.
+
+Add `findEntryByEntryIndex` in the `TopicCompactionService` API.
+
+Implement them in the `PulsarTopicCompactionService` using binary search.
+
+When get messageId by timestamp, find position from topicCompactionService if we can't find position in the manageLedger.
+
+## Public-facing Changes
+
+ ```java
+
+  public interface TopicCompactionService extends AutoCloseable { 

Review Comment:
   Please mention that this interface has `@InterfaceStability.Evolving` annotation. So that we are able to add new methods directly. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] coderzc commented on pull request #20867: [improve][pip] PIP-286: Add `findNewestPosition` method in the `TopicCompactionService` API

Posted by "coderzc (via GitHub)" <gi...@apache.org>.
coderzc commented on PR #20867:
URL: https://github.com/apache/pulsar/pull/20867#issuecomment-1651191634

   > If your number is off. Newest one I see before you is 288
   
   But I didn't find PIP-286 in the front PR


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org