You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/06/30 09:39:33 UTC
[GitHub] [pulsar] Technoboy- opened a new pull request, #16301: [Branch-2.7][Cherry-pick] Fix skips compacted data for reader/consumer
Technoboy- opened a new pull request, #16301:
URL: https://github.com/apache/pulsar/pull/16301
Cherry-pick #12429, #12464
## Motivation
The PR is fixing the compacted data lost during the data compaction.
We see a few events deletion but the compacted events obviously dropped a lot.
After investigating more details about the issue, only the first read operation reads the data from
the compacted ledger, since the second read operation, the broker start read data from the original
topic.
```
2021-10-19T23:09:30,021+0800 [broker-topic-workers-OrderedScheduler-7-0] INFO org.apache.pulsar.compaction.CompactedTopicImpl - =====[public/default/persistent/c499d42c-75d7-48d1-9225-2e724c0e1d83] Read from compacted Ledger = cursor position: -1:-1, Horizon: 16:-1, isFirstRead: true
2021-10-19T23:09:30,049+0800 [broker-topic-workers-OrderedScheduler-7-0] INFO org.apache.pulsar.compaction.CompactedTopicImpl - =====[public/default/persistent/c499d42c-75d7-48d1-9225-2e724c0e1d83] Read from original Ledger = cursor position: 16:0, Horizon: 16:-1, isFirstRead: false
```
#12429 only fixed the compactor skips data issue, but the normal reader/consumer also skips data while enabled read compacted data and read from the earliest position.
## Modifications
The compaction task depends on the last snapshot and the incremental
entries to build the new snapshot. So for the compaction cursor, we
need to force seek the read position to ensure the compactor can read
the complete last snapshot because the compactor will read the data
before the compaction cursor mark delete position.
## Verifying this change
New test added for checking the compacted data will not lost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on a diff in pull request #16301: [Branch-2.7][Cherry-pick] Fix skips compacted data for reader/consumer
Posted by GitBox <gi...@apache.org>.
Technoboy- commented on code in PR #16301:
URL: https://github.com/apache/pulsar/pull/16301#discussion_r913477695
##########
pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactedTopicTest.java:
##########
@@ -18,6 +18,7 @@
*/
package org.apache.pulsar.compaction;
+import static org.apache.pulsar.compaction.Compactor.COMPACTION_SUBSCRIPTION;
Review Comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #16301: [Branch-2.7][Cherry-pick] Fix skips compacted data for reader/consumer
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on code in PR #16301:
URL: https://github.com/apache/pulsar/pull/16301#discussion_r913438234
##########
pulsar-broker/src/test/java/org/apache/pulsar/compaction/CompactedTopicTest.java:
##########
@@ -18,6 +18,7 @@
*/
package org.apache.pulsar.compaction;
+import static org.apache.pulsar.compaction.Compactor.COMPACTION_SUBSCRIPTION;
Review Comment:
Why import it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- closed pull request #16301: [Branch-2.7][Cherry-pick] Fix skips compacted data for reader/consumer
Posted by GitBox <gi...@apache.org>.
Technoboy- closed pull request #16301: [Branch-2.7][Cherry-pick] Fix skips compacted data for reader/consumer
URL: https://github.com/apache/pulsar/pull/16301
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- merged pull request #16301: [Branch-2.7][Cherry-pick] Fix skips compacted data for reader/consumer
Posted by GitBox <gi...@apache.org>.
Technoboy- merged PR #16301:
URL: https://github.com/apache/pulsar/pull/16301
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org