You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/04/01 09:55:12 UTC
[GitHub] [pulsar] lordcheng10 opened a new pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
lordcheng10 opened a new pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985
### Motivation
We found a problem: even if we adjusted the cache eviction size and time a lot, there would still be some misscaches when the client restarted.
The current cache eviction strategy is to evict according to the read position, so there may be a miss cache when the data is pushed again.
In our scenario, there are tens of thousands of clients, and they are deployed on k8s. In order to save resources, they may dynamically expand or shrink, so that data re-pushing will happen from time to time.
### Modifications
Therefore, to improve the cache hit rate , I think a new cache eviction strategy should be added: evict data according to the markdelete position.
### Documentation
Check the box below or label this PR directly (if you have committer privilege).
Need to update docs?
- [ ] `doc-required`
(If you need help on updating docs, create a doc issue)
- [x] `no-need-doc`
(Please explain why)
- [ ] `doc`
(If this PR contains doc changes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086537636
> markDeletedPosition is better than readPosition in normal case. Maybe we can use markDeletedPosition by default.
OK, I aggree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 edited a comment on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 edited a comment on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086582710
> Make sense. IMO, `markDeletedPosition` is better than `readPosition` in normal case. Maybe we can use `markDeletedPosition` by default, and use `readPosition` iff overall cache usage is over some threshold (like 85%)?
Fixed. PTAL,thanks! @Jason918 @eolivelli @aloyszhang
1.add unit test;
2.fix evictionPos: use markDeletedPosition.next;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086783516
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086617559
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086696647
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086788393
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086528781
> Make sense, especially if you have some numbers to share.
>
> Can you please add a test case ?
>
> cc @merlimat @rdhabalia WDTY?
OK ,I will fix!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086680286
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086682818
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on a change in pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on a change in pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#discussion_r841077580
##########
File path: managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerTest.java
##########
@@ -291,6 +291,189 @@ public void acknowledge1() throws Exception {
ledger.close();
}
+ @Test
+ public void testCacheEvictionByMarkDeletedPosition() throws Throwable {
+ final CountDownLatch counter = new CountDownLatch(1);
+ ManagedLedgerConfig config = new ManagedLedgerConfig();
+ config.setCacheEvictionByMarkDeletedPosition(true);
+ factory.updateCacheEvictionTimeThreshold(TimeUnit.MILLISECONDS
+ .toNanos(30000));
+ factory.asyncOpen("my_test_ledger", config, new OpenLedgerCallback() {
+ @Override
+ public void openLedgerComplete(ManagedLedger ledger, Object ctx) {
+ ledger.asyncOpenCursor("test-cursor", new OpenCursorCallback() {
+ @Override
+ public void openCursorComplete(ManagedCursor cursor, Object ctx) {
+ ManagedLedger ledger = (ManagedLedger) ctx;
+ String message1 = "test";
+ ledger.asyncAddEntry(message1.getBytes(Encoding), new AddEntryCallback() {
+ @Override
+ public void addComplete(Position position, ByteBuf entryData, Object ctx) {
+ @SuppressWarnings("unchecked")
+ Pair<ManagedLedger, ManagedCursor> pair = (Pair<ManagedLedger, ManagedCursor>) ctx;
+ ManagedLedger ledger = pair.getLeft();
+ ManagedCursor cursor = pair.getRight();
+ assertEquals(((ManagedLedgerImpl) ledger).getCacheSize(), message1.getBytes(Encoding).length);
+
+ cursor.asyncReadEntries(1, new ReadEntriesCallback() {
+ @Override
+ public void readEntriesComplete(List<Entry> entries, Object ctx) {
+ ManagedCursor cursor = (ManagedCursor) ctx;
+ assertEquals(entries.size(), 1);
+ Entry entry = entries.get(0);
+ final Position position = entry.getPosition();
+ assertEquals(new String(entry.getDataAndRelease(), Encoding), message1);
+ assertEquals(((ManagedLedgerImpl) ledger).getCacheSize(), message1.getBytes(Encoding).length);
Review comment:
Fixed.
PTAL,thanks! @Jason918
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086682688
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086683379
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on a change in pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on a change in pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#discussion_r841155555
##########
File path: managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerTest.java
##########
@@ -291,6 +291,185 @@ public void acknowledge1() throws Exception {
ledger.close();
}
+ @Test
+ public void testCacheEvictionByMarkDeletedPosition() throws Throwable {
+ final CountDownLatch counter = new CountDownLatch(1);
+ ManagedLedgerConfig config = new ManagedLedgerConfig();
+ config.setCacheEvictionByMarkDeletedPosition(true);
+ factory.updateCacheEvictionTimeThreshold(TimeUnit.MILLISECONDS
+ .toNanos(30000));
+ factory.asyncOpen("my_test_ledger", config, new OpenLedgerCallback() {
+ @Override
+ public void openLedgerComplete(ManagedLedger ledger, Object ctx) {
+ ledger.asyncOpenCursor("test-cursor", new OpenCursorCallback() {
+ @Override
+ public void openCursorComplete(ManagedCursor cursor, Object ctx) {
+ ManagedLedger ledger = (ManagedLedger) ctx;
+ String message1 = "test";
+ ledger.asyncAddEntry(message1.getBytes(Encoding), new AddEntryCallback() {
+ @Override
+ public void addComplete(Position position, ByteBuf entryData, Object ctx) {
+ @SuppressWarnings("unchecked")
+ Pair<ManagedLedger, ManagedCursor> pair = (Pair<ManagedLedger, ManagedCursor>) ctx;
+ ManagedLedger ledger = pair.getLeft();
+ ManagedCursor cursor = pair.getRight();
+ assertEquals(((ManagedLedgerImpl) ledger).getCacheSize(), message1.getBytes(Encoding).length);
+
+ cursor.asyncReadEntries(1, new ReadEntriesCallback() {
+ @Override
+ public void readEntriesComplete(List<Entry> entries, Object ctx) {
+ ManagedCursor cursor = (ManagedCursor) ctx;
+ assertEquals(entries.size(), 1);
+ Entry entry = entries.get(0);
+ final Position position = entry.getPosition();
+ assertEquals(new String(entry.getDataAndRelease(), Encoding), message1);
+ ((ManagedLedgerImpl) ledger).doCacheEviction(
+ System.nanoTime() - TimeUnit.MILLISECONDS.toNanos(30000));
+ assertEquals(((ManagedLedgerImpl) ledger).getCacheSize(), message1.getBytes(Encoding).length);
+
+ log.debug("Mark-Deleting to position {}", position);
+ cursor.asyncMarkDelete(position, new MarkDeleteCallback() {
+ @Override
+ public void markDeleteComplete(Object ctx) {
+ log.debug("Mark delete complete");
+ ManagedCursor cursor = (ManagedCursor) ctx;
+ assertFalse(cursor.hasMoreEntries());
+ ((ManagedLedgerImpl) ledger).doCacheEviction(
+ System.nanoTime() - TimeUnit.MILLISECONDS.toNanos(30000));
+ assertEquals(((ManagedLedgerImpl) ledger).getCacheSize(), 0);
+
+ counter.countDown();
+ }
+
+ @Override
+ public void markDeleteFailed(ManagedLedgerException exception, Object ctx) {
+ fail(exception.getMessage());
+ }
+
+ }, cursor);
+ }
+
+ @Override
+ public void readEntriesFailed(ManagedLedgerException exception, Object ctx) {
+ fail(exception.getMessage());
+ }
Review comment:
I call 'fail' directly locally, which can cause the test to fail.
I have tried the following ways:
public void testCacheEvictionByMarkDeletedPosition() throws Throwable{
fail("xxx");
.....
}
public void testCacheEvictionByMarkDeletedPosition() throws Throwable{
.....
public void addComplete(Position position, ByteBuf entryData, Object ctx) {
fail("xxx");
......
}
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on a change in pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on a change in pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#discussion_r841155555
##########
File path: managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerTest.java
##########
@@ -291,6 +291,185 @@ public void acknowledge1() throws Exception {
ledger.close();
}
+ @Test
+ public void testCacheEvictionByMarkDeletedPosition() throws Throwable {
+ final CountDownLatch counter = new CountDownLatch(1);
+ ManagedLedgerConfig config = new ManagedLedgerConfig();
+ config.setCacheEvictionByMarkDeletedPosition(true);
+ factory.updateCacheEvictionTimeThreshold(TimeUnit.MILLISECONDS
+ .toNanos(30000));
+ factory.asyncOpen("my_test_ledger", config, new OpenLedgerCallback() {
+ @Override
+ public void openLedgerComplete(ManagedLedger ledger, Object ctx) {
+ ledger.asyncOpenCursor("test-cursor", new OpenCursorCallback() {
+ @Override
+ public void openCursorComplete(ManagedCursor cursor, Object ctx) {
+ ManagedLedger ledger = (ManagedLedger) ctx;
+ String message1 = "test";
+ ledger.asyncAddEntry(message1.getBytes(Encoding), new AddEntryCallback() {
+ @Override
+ public void addComplete(Position position, ByteBuf entryData, Object ctx) {
+ @SuppressWarnings("unchecked")
+ Pair<ManagedLedger, ManagedCursor> pair = (Pair<ManagedLedger, ManagedCursor>) ctx;
+ ManagedLedger ledger = pair.getLeft();
+ ManagedCursor cursor = pair.getRight();
+ assertEquals(((ManagedLedgerImpl) ledger).getCacheSize(), message1.getBytes(Encoding).length);
+
+ cursor.asyncReadEntries(1, new ReadEntriesCallback() {
+ @Override
+ public void readEntriesComplete(List<Entry> entries, Object ctx) {
+ ManagedCursor cursor = (ManagedCursor) ctx;
+ assertEquals(entries.size(), 1);
+ Entry entry = entries.get(0);
+ final Position position = entry.getPosition();
+ assertEquals(new String(entry.getDataAndRelease(), Encoding), message1);
+ ((ManagedLedgerImpl) ledger).doCacheEviction(
+ System.nanoTime() - TimeUnit.MILLISECONDS.toNanos(30000));
+ assertEquals(((ManagedLedgerImpl) ledger).getCacheSize(), message1.getBytes(Encoding).length);
+
+ log.debug("Mark-Deleting to position {}", position);
+ cursor.asyncMarkDelete(position, new MarkDeleteCallback() {
+ @Override
+ public void markDeleteComplete(Object ctx) {
+ log.debug("Mark delete complete");
+ ManagedCursor cursor = (ManagedCursor) ctx;
+ assertFalse(cursor.hasMoreEntries());
+ ((ManagedLedgerImpl) ledger).doCacheEviction(
+ System.nanoTime() - TimeUnit.MILLISECONDS.toNanos(30000));
+ assertEquals(((ManagedLedgerImpl) ledger).getCacheSize(), 0);
+
+ counter.countDown();
+ }
+
+ @Override
+ public void markDeleteFailed(ManagedLedgerException exception, Object ctx) {
+ fail(exception.getMessage());
+ }
+
+ }, cursor);
+ }
+
+ @Override
+ public void readEntriesFailed(ManagedLedgerException exception, Object ctx) {
+ fail(exception.getMessage());
+ }
Review comment:
I call 'fail' directly locally, which can cause the test to fail. @eolivelli
I have tried the following ways:
public void testCacheEvictionByMarkDeletedPosition() throws Throwable{
fail("xxx");
.....
}
public void testCacheEvictionByMarkDeletedPosition() throws Throwable{
.....
public void addComplete(Position position, ByteBuf entryData, Object ctx) {
fail("xxx");
......
}
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086684928
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086770764
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1085768118
@eolivelli @Technoboy- @codelipenghui
PTAL,thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086538024
> use readPosition iff overall cache usage is over some threshold (like 85%)?
In the scenario of message re-pushing, this way of dynamically adjusting the eviction policy is not suitable. I can reopen a discussion about dynamically adjusting the eviction policy later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086780450
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086682889
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086760713
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086887857
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086886961
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086672712
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086893232
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086836749
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086836619
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086688459
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on a change in pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
Jason918 commented on a change in pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#discussion_r841072182
##########
File path: managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerTest.java
##########
@@ -291,6 +291,189 @@ public void acknowledge1() throws Exception {
ledger.close();
}
+ @Test
+ public void testCacheEvictionByMarkDeletedPosition() throws Throwable {
+ final CountDownLatch counter = new CountDownLatch(1);
+ ManagedLedgerConfig config = new ManagedLedgerConfig();
+ config.setCacheEvictionByMarkDeletedPosition(true);
+ factory.updateCacheEvictionTimeThreshold(TimeUnit.MILLISECONDS
+ .toNanos(30000));
+ factory.asyncOpen("my_test_ledger", config, new OpenLedgerCallback() {
+ @Override
+ public void openLedgerComplete(ManagedLedger ledger, Object ctx) {
+ ledger.asyncOpenCursor("test-cursor", new OpenCursorCallback() {
+ @Override
+ public void openCursorComplete(ManagedCursor cursor, Object ctx) {
+ ManagedLedger ledger = (ManagedLedger) ctx;
+ String message1 = "test";
+ ledger.asyncAddEntry(message1.getBytes(Encoding), new AddEntryCallback() {
+ @Override
+ public void addComplete(Position position, ByteBuf entryData, Object ctx) {
+ @SuppressWarnings("unchecked")
+ Pair<ManagedLedger, ManagedCursor> pair = (Pair<ManagedLedger, ManagedCursor>) ctx;
+ ManagedLedger ledger = pair.getLeft();
+ ManagedCursor cursor = pair.getRight();
+ assertEquals(((ManagedLedgerImpl) ledger).getCacheSize(), message1.getBytes(Encoding).length);
+
+ cursor.asyncReadEntries(1, new ReadEntriesCallback() {
+ @Override
+ public void readEntriesComplete(List<Entry> entries, Object ctx) {
+ ManagedCursor cursor = (ManagedCursor) ctx;
+ assertEquals(entries.size(), 1);
+ Entry entry = entries.get(0);
+ final Position position = entry.getPosition();
+ assertEquals(new String(entry.getDataAndRelease(), Encoding), message1);
+ assertEquals(((ManagedLedgerImpl) ledger).getCacheSize(), message1.getBytes(Encoding).length);
+
+ log.debug("Mark-Deleting to position {}", position);
+ cursor.asyncMarkDelete(position, new MarkDeleteCallback() {
+ @Override
+ public void markDeleteComplete(Object ctx) {
+ log.debug("Mark delete complete");
+ ManagedCursor cursor = (ManagedCursor) ctx;
+ assertFalse(cursor.hasMoreEntries());
+ // wait eviction finish.
+ try {
Review comment:
It's better to trigger `ManagedLedgerImpl#doCacheEviction` manually instead of using sleep to avoid flaky test.
Or just use `Awaitility.await().untilAsserted`.
##########
File path: managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerTest.java
##########
@@ -291,6 +291,189 @@ public void acknowledge1() throws Exception {
ledger.close();
}
+ @Test
+ public void testCacheEvictionByMarkDeletedPosition() throws Throwable {
+ final CountDownLatch counter = new CountDownLatch(1);
+ ManagedLedgerConfig config = new ManagedLedgerConfig();
+ config.setCacheEvictionByMarkDeletedPosition(true);
+ factory.updateCacheEvictionTimeThreshold(TimeUnit.MILLISECONDS
+ .toNanos(30000));
+ factory.asyncOpen("my_test_ledger", config, new OpenLedgerCallback() {
+ @Override
+ public void openLedgerComplete(ManagedLedger ledger, Object ctx) {
+ ledger.asyncOpenCursor("test-cursor", new OpenCursorCallback() {
+ @Override
+ public void openCursorComplete(ManagedCursor cursor, Object ctx) {
+ ManagedLedger ledger = (ManagedLedger) ctx;
+ String message1 = "test";
+ ledger.asyncAddEntry(message1.getBytes(Encoding), new AddEntryCallback() {
+ @Override
+ public void addComplete(Position position, ByteBuf entryData, Object ctx) {
+ @SuppressWarnings("unchecked")
+ Pair<ManagedLedger, ManagedCursor> pair = (Pair<ManagedLedger, ManagedCursor>) ctx;
+ ManagedLedger ledger = pair.getLeft();
+ ManagedCursor cursor = pair.getRight();
+ assertEquals(((ManagedLedgerImpl) ledger).getCacheSize(), message1.getBytes(Encoding).length);
+
+ cursor.asyncReadEntries(1, new ReadEntriesCallback() {
+ @Override
+ public void readEntriesComplete(List<Entry> entries, Object ctx) {
+ ManagedCursor cursor = (ManagedCursor) ctx;
+ assertEquals(entries.size(), 1);
+ Entry entry = entries.get(0);
+ final Position position = entry.getPosition();
+ assertEquals(new String(entry.getDataAndRelease(), Encoding), message1);
+ assertEquals(((ManagedLedgerImpl) ledger).getCacheSize(), message1.getBytes(Encoding).length);
Review comment:
Need a manual trigger of `ManagedLedgerImpl#doCacheEviction` before this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 removed a comment on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 removed a comment on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1085704178
@eolivelli @Technoboy-
PTAL,thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086617848
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 edited a comment on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 edited a comment on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086537636
> markDeletedPosition is better than readPosition in normal case. Maybe we can use markDeletedPosition by default.
OK, I aggree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on a change in pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on a change in pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#discussion_r841216241
##########
File path: managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerTest.java
##########
@@ -291,6 +291,185 @@ public void acknowledge1() throws Exception {
ledger.close();
}
+ @Test
+ public void testCacheEvictionByMarkDeletedPosition() throws Throwable {
+ final CountDownLatch counter = new CountDownLatch(1);
+ ManagedLedgerConfig config = new ManagedLedgerConfig();
+ config.setCacheEvictionByMarkDeletedPosition(true);
+ factory.updateCacheEvictionTimeThreshold(TimeUnit.MILLISECONDS
+ .toNanos(30000));
+ factory.asyncOpen("my_test_ledger", config, new OpenLedgerCallback() {
+ @Override
+ public void openLedgerComplete(ManagedLedger ledger, Object ctx) {
+ ledger.asyncOpenCursor("test-cursor", new OpenCursorCallback() {
+ @Override
+ public void openCursorComplete(ManagedCursor cursor, Object ctx) {
+ ManagedLedger ledger = (ManagedLedger) ctx;
+ String message1 = "test";
+ ledger.asyncAddEntry(message1.getBytes(Encoding), new AddEntryCallback() {
+ @Override
+ public void addComplete(Position position, ByteBuf entryData, Object ctx) {
+ @SuppressWarnings("unchecked")
+ Pair<ManagedLedger, ManagedCursor> pair = (Pair<ManagedLedger, ManagedCursor>) ctx;
+ ManagedLedger ledger = pair.getLeft();
+ ManagedCursor cursor = pair.getRight();
+ assertEquals(((ManagedLedgerImpl) ledger).getCacheSize(), message1.getBytes(Encoding).length);
+
+ cursor.asyncReadEntries(1, new ReadEntriesCallback() {
+ @Override
+ public void readEntriesComplete(List<Entry> entries, Object ctx) {
+ ManagedCursor cursor = (ManagedCursor) ctx;
+ assertEquals(entries.size(), 1);
+ Entry entry = entries.get(0);
+ final Position position = entry.getPosition();
+ assertEquals(new String(entry.getDataAndRelease(), Encoding), message1);
+ ((ManagedLedgerImpl) ledger).doCacheEviction(
+ System.nanoTime() - TimeUnit.MILLISECONDS.toNanos(30000));
+ assertEquals(((ManagedLedgerImpl) ledger).getCacheSize(), message1.getBytes(Encoding).length);
+
+ log.debug("Mark-Deleting to position {}", position);
+ cursor.asyncMarkDelete(position, new MarkDeleteCallback() {
+ @Override
+ public void markDeleteComplete(Object ctx) {
+ log.debug("Mark delete complete");
+ ManagedCursor cursor = (ManagedCursor) ctx;
+ assertFalse(cursor.hasMoreEntries());
+ ((ManagedLedgerImpl) ledger).doCacheEviction(
+ System.nanoTime() - TimeUnit.MILLISECONDS.toNanos(30000));
+ assertEquals(((ManagedLedgerImpl) ledger).getCacheSize(), 0);
+
+ counter.countDown();
+ }
+
+ @Override
+ public void markDeleteFailed(ManagedLedgerException exception, Object ctx) {
+ fail(exception.getMessage());
+ }
+
+ }, cursor);
+ }
+
+ @Override
+ public void readEntriesFailed(ManagedLedgerException exception, Object ctx) {
+ fail(exception.getMessage());
+ }
Review comment:
Fixed: use CompletableFuture
PTAL,thanks! @eolivelli
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086893152
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086610628
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#discussion_r841110615
##########
File path: managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerTest.java
##########
@@ -291,6 +291,185 @@ public void acknowledge1() throws Exception {
ledger.close();
}
+ @Test
+ public void testCacheEvictionByMarkDeletedPosition() throws Throwable {
+ final CountDownLatch counter = new CountDownLatch(1);
+ ManagedLedgerConfig config = new ManagedLedgerConfig();
+ config.setCacheEvictionByMarkDeletedPosition(true);
+ factory.updateCacheEvictionTimeThreshold(TimeUnit.MILLISECONDS
+ .toNanos(30000));
+ factory.asyncOpen("my_test_ledger", config, new OpenLedgerCallback() {
+ @Override
+ public void openLedgerComplete(ManagedLedger ledger, Object ctx) {
+ ledger.asyncOpenCursor("test-cursor", new OpenCursorCallback() {
+ @Override
+ public void openCursorComplete(ManagedCursor cursor, Object ctx) {
+ ManagedLedger ledger = (ManagedLedger) ctx;
+ String message1 = "test";
+ ledger.asyncAddEntry(message1.getBytes(Encoding), new AddEntryCallback() {
+ @Override
+ public void addComplete(Position position, ByteBuf entryData, Object ctx) {
+ @SuppressWarnings("unchecked")
+ Pair<ManagedLedger, ManagedCursor> pair = (Pair<ManagedLedger, ManagedCursor>) ctx;
+ ManagedLedger ledger = pair.getLeft();
+ ManagedCursor cursor = pair.getRight();
+ assertEquals(((ManagedLedgerImpl) ledger).getCacheSize(), message1.getBytes(Encoding).length);
+
+ cursor.asyncReadEntries(1, new ReadEntriesCallback() {
+ @Override
+ public void readEntriesComplete(List<Entry> entries, Object ctx) {
+ ManagedCursor cursor = (ManagedCursor) ctx;
+ assertEquals(entries.size(), 1);
+ Entry entry = entries.get(0);
+ final Position position = entry.getPosition();
+ assertEquals(new String(entry.getDataAndRelease(), Encoding), message1);
+ ((ManagedLedgerImpl) ledger).doCacheEviction(
+ System.nanoTime() - TimeUnit.MILLISECONDS.toNanos(30000));
+ assertEquals(((ManagedLedgerImpl) ledger).getCacheSize(), message1.getBytes(Encoding).length);
+
+ log.debug("Mark-Deleting to position {}", position);
+ cursor.asyncMarkDelete(position, new MarkDeleteCallback() {
+ @Override
+ public void markDeleteComplete(Object ctx) {
+ log.debug("Mark delete complete");
+ ManagedCursor cursor = (ManagedCursor) ctx;
+ assertFalse(cursor.hasMoreEntries());
+ ((ManagedLedgerImpl) ledger).doCacheEviction(
+ System.nanoTime() - TimeUnit.MILLISECONDS.toNanos(30000));
+ assertEquals(((ManagedLedgerImpl) ledger).getCacheSize(), 0);
+
+ counter.countDown();
+ }
+
+ @Override
+ public void markDeleteFailed(ManagedLedgerException exception, Object ctx) {
+ fail(exception.getMessage());
+ }
+
+ }, cursor);
+ }
+
+ @Override
+ public void readEntriesFailed(ManagedLedgerException exception, Object ctx) {
+ fail(exception.getMessage());
+ }
Review comment:
I am not sure that calling 'fail' not in the main TestNG method/thread will cause the test to fail
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086685114
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086897316
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1085704178
@eolivelli @Technoboy-
PTAL,thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 edited a comment on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 edited a comment on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086582710
> Make sense. IMO, `markDeletedPosition` is better than `readPosition` in normal case. Maybe we can use `markDeletedPosition` by default, and use `readPosition` iff overall cache usage is over some threshold (like 85%)?
Fixed. PTAL,thanks! @Jason918 @eolivelli @aloyszhang
1.add unit test;
2.use markDeletedPosition by default;
3.fix evictionPos: use markDeletedPosition.next;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086582710
> Make sense. IMO, `markDeletedPosition` is better than `readPosition` in normal case. Maybe we can use `markDeletedPosition` by default, and use `readPosition` iff overall cache usage is over some threshold (like 85%)?
Fixed. PTAL,thanks! @Jason918
1.add unit test;
2.use markDeletedPosition by default;
3.fix evictionPos: use markDeletedPosition.next;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086791989
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086788918
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] aloyszhang commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
aloyszhang commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086537834
Make sense to me.
A test to cover this is better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 edited a comment on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 edited a comment on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086582710
> Make sense. IMO, `markDeletedPosition` is better than `readPosition` in normal case. Maybe we can use `markDeletedPosition` by default, and use `readPosition` iff overall cache usage is over some threshold (like 85%)?
Fixed. PTAL,thanks! @Jason918 @eolivelli
1.add unit test;
2.use markDeletedPosition by default;
3.fix evictionPos: use markDeletedPosition.next;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on a change in pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on a change in pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#discussion_r841077550
##########
File path: managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerTest.java
##########
@@ -291,6 +291,189 @@ public void acknowledge1() throws Exception {
ledger.close();
}
+ @Test
+ public void testCacheEvictionByMarkDeletedPosition() throws Throwable {
+ final CountDownLatch counter = new CountDownLatch(1);
+ ManagedLedgerConfig config = new ManagedLedgerConfig();
+ config.setCacheEvictionByMarkDeletedPosition(true);
+ factory.updateCacheEvictionTimeThreshold(TimeUnit.MILLISECONDS
+ .toNanos(30000));
+ factory.asyncOpen("my_test_ledger", config, new OpenLedgerCallback() {
+ @Override
+ public void openLedgerComplete(ManagedLedger ledger, Object ctx) {
+ ledger.asyncOpenCursor("test-cursor", new OpenCursorCallback() {
+ @Override
+ public void openCursorComplete(ManagedCursor cursor, Object ctx) {
+ ManagedLedger ledger = (ManagedLedger) ctx;
+ String message1 = "test";
+ ledger.asyncAddEntry(message1.getBytes(Encoding), new AddEntryCallback() {
+ @Override
+ public void addComplete(Position position, ByteBuf entryData, Object ctx) {
+ @SuppressWarnings("unchecked")
+ Pair<ManagedLedger, ManagedCursor> pair = (Pair<ManagedLedger, ManagedCursor>) ctx;
+ ManagedLedger ledger = pair.getLeft();
+ ManagedCursor cursor = pair.getRight();
+ assertEquals(((ManagedLedgerImpl) ledger).getCacheSize(), message1.getBytes(Encoding).length);
+
+ cursor.asyncReadEntries(1, new ReadEntriesCallback() {
+ @Override
+ public void readEntriesComplete(List<Entry> entries, Object ctx) {
+ ManagedCursor cursor = (ManagedCursor) ctx;
+ assertEquals(entries.size(), 1);
+ Entry entry = entries.get(0);
+ final Position position = entry.getPosition();
+ assertEquals(new String(entry.getDataAndRelease(), Encoding), message1);
+ assertEquals(((ManagedLedgerImpl) ledger).getCacheSize(), message1.getBytes(Encoding).length);
+
+ log.debug("Mark-Deleting to position {}", position);
+ cursor.asyncMarkDelete(position, new MarkDeleteCallback() {
+ @Override
+ public void markDeleteComplete(Object ctx) {
+ log.debug("Mark delete complete");
+ ManagedCursor cursor = (ManagedCursor) ctx;
+ assertFalse(cursor.hasMoreEntries());
+ // wait eviction finish.
+ try {
Review comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086659228
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086696258
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086703382
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on pull request #14985: Add a cache eviction policy:Evicting cache data by the slowest markDeletedPosition
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on pull request #14985:
URL: https://github.com/apache/pulsar/pull/14985#issuecomment-1086781449
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org