You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "michalcukierman (via GitHub)" <gi...@apache.org> on 2023/08/09 16:42:00 UTC
[GitHub] [pulsar] michalcukierman opened a new issue, #20968: [Bug] Brokers restart during read/writes causes data loss
michalcukierman opened a new issue, #20968:
URL: https://github.com/apache/pulsar/issues/20968
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
- Affected Pulsar versions:
- 3.0.1
- 3.0.0
- Tested environments
- GCP Kubernentes Cluster - 3-3-3-4 (Zk-proxy-broker-BK), Qurum 2-2-2
- MicroK8S local cluster - 1-1-3-1 (Zk-proxy-broker-BK), Qurum 1-1-1
Deplyments done using official Helm Chart.
### Minimal reproduce step
Checkout:
https://github.com/websight-io/pulsar-chaos-test/tree/main/environment-local and follow README.MD
OR:
1. Create partitioned (12 partitions), set retention to -1, -1
2. Create a producer and start writing to a topic
3. Create a consumer with a new subscription set to `Earliest` position and start reading the messages
4. Restart all Broker Pods one-by-one
5. Wait for the brokers to start (2-3 minutes)
6. Stop producer created in step 2
7. Stop consumer created in step 3
8. Create a new consumer with a new subscription set to `Earliest` position and read all the messages
### What did you expect to see?
Both consumers read the same number of messages.
Topic does not contain all the messages that were stored in it.
### What did you see instead?
Consumer created after the restart cannot read the same number of messages as the one created before the restart.
Topic should contain all the messages that were stored in it.
### Anything else?
- Retention is set before starting the set. It's preserved and can be verified:
`bin/pulsar-admin topics get-retention persistent://websight/dxp/chaos-test`
```
{
"retentionTimeInMinutes" : -1,
"retentionSizeInMB" : -1
}
```
- Pulsar Manager shows that some of the partitions are empty after restart (or the data is not equaly distributed)
<img width="1217" alt="Screenshot 2023-08-09 at 18 40 31" src="https://github.com/apache/pulsar/assets/4356553/e0326f27-0031-4dcd-95e3-98968cda8962">
- Pulsar Manager state from before the restart:
<img width="1218" alt="Screenshot 2023-08-09 at 18 41 28" src="https://github.com/apache/pulsar/assets/4356553/aaa45b43-03e7-4f48-a994-889a466d80eb">
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1673393274
<img width="2900" alt="Screenshot 2023-08-10 at 16 53 59" src="https://github.com/apache/pulsar/assets/4356553/b8932375-dc46-4aa8-8e46-a1c23918e8dc">
I've also found that some partition ledgers are removed:
`Ledger 609 contains the current last confirmed entry 609:2, and it is going to be deleted`
Topic policies are updated few lines below:
`Policies updated successfully`
`INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [websight/dxp/persistent/chaos-topic-partition-10] Removing ledger 565 - size: 1069`
I don't know if it helps, but maybe the policies are applied too late?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1678690344
Let me first try if its possible to reproduce the error with pulsar-perf. Till now i was not able to. I observed it using Java/Python clients (one example you can find in the linked repository)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1671785763
Bookies are never restarted, that's why I've tested it locally without PVC.
What's more on GCP, the situation occur with Bookies with persistence and PVs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1678576425
Also you need to use readers, not just writers, as I observed the issue during messages consumption.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1677961577
For now we are just using non-partitioned topics. I haven't observed this issues on a single-partition topic. Its a short-term solution, as it does not scale well across brokers.
Namespace level policies could be a workaround (to be checked if it works), after some refactoring on our side.
Still the question is how it's going to work with compacted, partitioned topics, which should also retain the messages.
I will also check Pulsar 3.1, but haven't seen anything related in a changelog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1682121128
I've set the dedicated namespace with retention:
```bash
NAMESPACE="public/test"
TOPIC="persistent://$NAMESPACE/test-partitioned-0"
kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin namespaces create $NAMESPACE
kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin namespaces set-retention -s -1 -t -1 $NAMESPACE
```
I run the test a couple of times in a row. It looks like the messages are preserved between restarts.
I'll run two more tests:
- restart only one broker, not all a couple of times to check if the issue occure
- try with brokerDeleteInactiveTopicsMode
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1680552842
@michalcukierman Thanks for providing the detailed information. Have you changed the `brokerDeleteInactiveTopicsMode` and `brokerDeleteInactivePartitionedTopicMetadataEnabled`?
```
kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin topics stats-internal test-partitioned-topic-partition-0`
Partitioned Topic not found: persistent://public/default/test-partitioned-topic-partition-0 has zero partitions
```
It looks like somehow the topic has been deleted. Could you please try to disable the inactive topic auto-deletion?
```
brokerDeleteInactiveTopicsEnabled=false
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1685894042
I haven't notice any warn or errors. Please checkout:
https://github.com/websight-io/pulsar-chaos-test-perf/
You should be able to recreate the issue by running single SH script.
I also think that the screenshot mentioned here may show the sequence of deleting a leadger/applying policy:
https://github.com/apache/pulsar/issues/20968#issuecomment-1673393274
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1682477362
Looks like the: `brokerDeleteInactiveTopicsEnabled=false` works as well. I'll run some need to observe it a bit more.
> Notice that policies to the topic are applied after the ledger is removed.
This could be it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman closed issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman closed issue #20968: [Bug] Brokers restart during read/writes causes data loss
URL: https://github.com/apache/pulsar/issues/20968
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1677471692
@michalcukierman I think you can try to set the retention policy to the namespace level or broker level. Just to avoid the potential problems from the topic-level policies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1678449419
I was not able to reproduce the issue with a non-partitioned topic. It does not mean that the issue is not there, maybe it's just harder to reproduce.
I've used 12 partitions. Try to increase the number of partitions.
You can checkout linked repository, the only dependency there is Pulsar.
To make sure there is nothing on my computer causing the problem, I work with it on GCP.
I will try to use Pulsar Perf and check if I can use it to reproduce. Till now, I observed it on Java and Python Clients.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1673013938
I am also getting the warnings:
<img width="1094" alt="Screenshot 2023-08-10 at 13 04 53" src="https://github.com/apache/pulsar/assets/4356553/82ca615a-e2a2-4f6c-80c9-73f0bd1947e8">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1680437807
@codelipenghui
I was able to reproduce the error with `pulsar-perf` and have some new observations:
- Data loss occurs on a single topic, it's harder to reproduce (does not happen every time)
- Messages producing is not required to reproduce the issue. A single, active consumer is needed to recreate the error.
- (Sometimes?) Partitions are lost after the test, what could be the cause of the lack of messages
I've created a project with two scripts (one for partitioned, one for non-partitioned topics) and put in a GH repository:
https://github.com/websight-io/pulsar-chaos-test-perf
The reproduction steps (simplified, see the project for the one you would like to execute):
```
TOPIC="test-partitioned-0"
# Preparation
kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin topics create-partitioned-topic $TOPIC -p 12
kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin topics set-retention -s -1 -t -1 $TOPIC
kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin topics get-retention $TOPIC
kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-perf produce -bm 1 -r 100 -m 10000 -s 102400 $TOPIC
# Execution
kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-perf consume -ioThreads 4 -c 4 -sp Earliest $TOPIC &
CONSUMER_PID=$!
sleep 15 # wait for the producer to start sending messages
kubectl delete pod pulsar-broker-0 -n pulsar
kubectl delete pod pulsar-broker-1 -n pulsar
kubectl delete pod pulsar-broker-2 -n pulsar
sleep 90 # wait for the messages to be received and brokers to stabilized
kill -15 $CONSUMER_PID
# Verification
kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-perf read $TOPIC
```
Data loss on partitioned topic takes some time, you may be read all the messages at the first attempt, they may be lost after some time (try to re-run the test). You may also try to run tests several times, or change the parameters like `sleep` time or messages number/size.
Alternative verification that can replace reader (I see consistent results in sum of `numberOfEntries` on partitions and read messages):
```
for i in {0..11}
do
kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin topics stats-internal $TOPIC-partition-$i
done
````
The output of internal stats for partitioned topic is sometimes:
```
kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin topics stats-internal test-partitioned-topic-partition-0`
Partitioned Topic not found: persistent://public/default/test-partitioned-topic-partition-0 has zero partitions
```
Example internal-stats taken after one of my run:
```
chaos-test-pulsar-perf git:(main) ✗ for i in {0..11}
do
kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin topics stats-internal test-partitioned-10-partition-$i
done
{
"entriesAddedCounter" : 0,
"numberOfEntries" : 0,
"totalSize" : 0,
"currentLedgerEntries" : 0,
"currentLedgerSize" : 0,
"lastLedgerCreatedTimestamp" : "2023-08-16T11:02:26.644Z",
"waitingCursorsCount" : 2,
"pendingAddEntriesCount" : 0,
"lastConfirmedEntry" : "2022:841",
"state" : "LedgerOpened",
"ledgers" : [ {
"ledgerId" : 2110,
"entries" : 0,
"size" : 0,
"offloaded" : false,
"underReplicated" : false
} ],
"cursors" : {
"sub" : {
"markDeletePosition" : "2022:841",
"readPosition" : "2022:842",
"waitingReadOp" : true,
"pendingReadOps" : 0,
"messagesConsumedCounter" : 0,
"cursorLedger" : -1,
"cursorLedgerLastEntry" : -1,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2023-08-16T11:02:26.861Z",
"state" : "NoLedger",
"active" : true,
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : true,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
},
"multiTopicsReader-883d8df4ce" : {
"markDeletePosition" : "2022:-1",
"readPosition" : "2110:0",
"waitingReadOp" : true,
"pendingReadOps" : 0,
"messagesConsumedCounter" : -842,
"cursorLedger" : -1,
"cursorLedgerLastEntry" : -1,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2023-08-16T11:03:29.347Z",
"state" : "Open",
"active" : true,
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : true,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
}
},
"schemaLedgers" : [ ],
"compactedLedger" : {
"ledgerId" : -1,
"entries" : -1,
"size" : -1,
"offloaded" : false,
"underReplicated" : false
}
}
{
"entriesAddedCounter" : 0,
"numberOfEntries" : 0,
"totalSize" : 0,
"currentLedgerEntries" : 0,
"currentLedgerSize" : 0,
"lastLedgerCreatedTimestamp" : "2023-08-16T11:03:03.491Z",
"waitingCursorsCount" : 2,
"pendingAddEntriesCount" : 0,
"lastConfirmedEntry" : "2042:818",
"state" : "LedgerOpened",
"ledgers" : [ {
"ledgerId" : 2174,
"entries" : 0,
"size" : 0,
"offloaded" : false,
"underReplicated" : false
} ],
"cursors" : {
"sub" : {
"markDeletePosition" : "2042:818",
"readPosition" : "2042:819",
"waitingReadOp" : true,
"pendingReadOps" : 0,
"messagesConsumedCounter" : 0,
"cursorLedger" : -1,
"cursorLedgerLastEntry" : -1,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2023-08-16T11:03:03.725Z",
"state" : "NoLedger",
"active" : true,
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : true,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
},
"multiTopicsReader-883d8df4ce" : {
"markDeletePosition" : "2042:-1",
"readPosition" : "2174:0",
"waitingReadOp" : true,
"pendingReadOps" : 0,
"messagesConsumedCounter" : -819,
"cursorLedger" : -1,
"cursorLedgerLastEntry" : -1,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2023-08-16T11:03:29.382Z",
"state" : "Open",
"active" : true,
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : true,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
}
},
"schemaLedgers" : [ ],
"compactedLedger" : {
"ledgerId" : -1,
"entries" : -1,
"size" : -1,
"offloaded" : false,
"underReplicated" : false
}
}
{
"entriesAddedCounter" : 0,
"numberOfEntries" : 0,
"totalSize" : 0,
"currentLedgerEntries" : 0,
"currentLedgerSize" : 0,
"lastLedgerCreatedTimestamp" : "2023-08-16T11:03:02.064Z",
"waitingCursorsCount" : 2,
"pendingAddEntriesCount" : 0,
"lastConfirmedEntry" : "2026:849",
"state" : "LedgerOpened",
"ledgers" : [ {
"ledgerId" : 2138,
"entries" : 0,
"size" : 0,
"offloaded" : false,
"underReplicated" : false
} ],
"cursors" : {
"sub" : {
"markDeletePosition" : "2026:849",
"readPosition" : "2026:850",
"waitingReadOp" : true,
"pendingReadOps" : 0,
"messagesConsumedCounter" : 0,
"cursorLedger" : -1,
"cursorLedgerLastEntry" : -1,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2023-08-16T11:03:02.244Z",
"state" : "NoLedger",
"active" : true,
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : true,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
},
"multiTopicsReader-883d8df4ce" : {
"markDeletePosition" : "2026:-1",
"readPosition" : "2138:0",
"waitingReadOp" : true,
"pendingReadOps" : 0,
"messagesConsumedCounter" : -850,
"cursorLedger" : -1,
"cursorLedgerLastEntry" : -1,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2023-08-16T11:03:29.415Z",
"state" : "Open",
"active" : true,
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : true,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
}
},
"schemaLedgers" : [ ],
"compactedLedger" : {
"ledgerId" : -1,
"entries" : -1,
"size" : -1,
"offloaded" : false,
"underReplicated" : false
}
}
{
"entriesAddedCounter" : 0,
"numberOfEntries" : 0,
"totalSize" : 0,
"currentLedgerEntries" : 0,
"currentLedgerSize" : 0,
"lastLedgerCreatedTimestamp" : "2023-08-16T11:03:03.179Z",
"waitingCursorsCount" : 2,
"pendingAddEntriesCount" : 0,
"lastConfirmedEntry" : "2029:817",
"state" : "LedgerOpened",
"ledgers" : [ {
"ledgerId" : 2164,
"entries" : 0,
"size" : 0,
"offloaded" : false,
"underReplicated" : false
} ],
"cursors" : {
"sub" : {
"markDeletePosition" : "2029:817",
"readPosition" : "2029:818",
"waitingReadOp" : true,
"pendingReadOps" : 0,
"messagesConsumedCounter" : 0,
"cursorLedger" : -1,
"cursorLedgerLastEntry" : -1,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2023-08-16T11:03:03.304Z",
"state" : "NoLedger",
"active" : true,
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : true,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
},
"multiTopicsReader-883d8df4ce" : {
"markDeletePosition" : "2029:-1",
"readPosition" : "2164:0",
"waitingReadOp" : true,
"pendingReadOps" : 0,
"messagesConsumedCounter" : -818,
"cursorLedger" : -1,
"cursorLedgerLastEntry" : -1,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2023-08-16T11:03:29.409Z",
"state" : "Open",
"active" : true,
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : true,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
}
},
"schemaLedgers" : [ ],
"compactedLedger" : {
"ledgerId" : -1,
"entries" : -1,
"size" : -1,
"offloaded" : false,
"underReplicated" : false
}
}
{
"entriesAddedCounter" : 0,
"numberOfEntries" : 0,
"totalSize" : 0,
"currentLedgerEntries" : 0,
"currentLedgerSize" : 0,
"lastLedgerCreatedTimestamp" : "2023-08-16T11:02:26.658Z",
"waitingCursorsCount" : 2,
"pendingAddEntriesCount" : 0,
"lastConfirmedEntry" : "2024:843",
"state" : "LedgerOpened",
"ledgers" : [ {
"ledgerId" : 2112,
"entries" : 0,
"size" : 0,
"offloaded" : false,
"underReplicated" : false
} ],
"cursors" : {
"sub" : {
"markDeletePosition" : "2024:843",
"readPosition" : "2024:844",
"waitingReadOp" : true,
"pendingReadOps" : 0,
"messagesConsumedCounter" : 0,
"cursorLedger" : -1,
"cursorLedgerLastEntry" : -1,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2023-08-16T11:02:26.868Z",
"state" : "NoLedger",
"active" : true,
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : true,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
},
"multiTopicsReader-883d8df4ce" : {
"markDeletePosition" : "2024:-1",
"readPosition" : "2112:0",
"waitingReadOp" : true,
"pendingReadOps" : 0,
"messagesConsumedCounter" : -844,
"cursorLedger" : -1,
"cursorLedgerLastEntry" : -1,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2023-08-16T11:03:29.369Z",
"state" : "Open",
"active" : true,
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : true,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
}
},
"schemaLedgers" : [ ],
"compactedLedger" : {
"ledgerId" : -1,
"entries" : -1,
"size" : -1,
"offloaded" : false,
"underReplicated" : false
}
}
{
"entriesAddedCounter" : 0,
"numberOfEntries" : 0,
"totalSize" : 0,
"currentLedgerEntries" : 0,
"currentLedgerSize" : 0,
"lastLedgerCreatedTimestamp" : "2023-08-16T11:03:03.526Z",
"waitingCursorsCount" : 2,
"pendingAddEntriesCount" : 0,
"lastConfirmedEntry" : "2041:833",
"state" : "LedgerOpened",
"ledgers" : [ {
"ledgerId" : 2178,
"entries" : 0,
"size" : 0,
"offloaded" : false,
"underReplicated" : false
} ],
"cursors" : {
"sub" : {
"markDeletePosition" : "2041:833",
"readPosition" : "2041:834",
"waitingReadOp" : true,
"pendingReadOps" : 0,
"messagesConsumedCounter" : 0,
"cursorLedger" : -1,
"cursorLedgerLastEntry" : -1,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2023-08-16T11:03:03.727Z",
"state" : "NoLedger",
"active" : true,
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : true,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
},
"multiTopicsReader-883d8df4ce" : {
"markDeletePosition" : "2041:-1",
"readPosition" : "2178:0",
"waitingReadOp" : true,
"pendingReadOps" : 0,
"messagesConsumedCounter" : -834,
"cursorLedger" : -1,
"cursorLedgerLastEntry" : -1,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2023-08-16T11:03:29.412Z",
"state" : "Open",
"active" : true,
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : true,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
}
},
"schemaLedgers" : [ ],
"compactedLedger" : {
"ledgerId" : -1,
"entries" : -1,
"size" : -1,
"offloaded" : false,
"underReplicated" : false
}
}
{
"entriesAddedCounter" : 0,
"numberOfEntries" : 0,
"totalSize" : 0,
"currentLedgerEntries" : 0,
"currentLedgerSize" : 0,
"lastLedgerCreatedTimestamp" : "2023-08-16T11:03:02.1Z",
"waitingCursorsCount" : 2,
"pendingAddEntriesCount" : 0,
"lastConfirmedEntry" : "2025:830",
"state" : "LedgerOpened",
"ledgers" : [ {
"ledgerId" : 2150,
"entries" : 0,
"size" : 0,
"offloaded" : false,
"underReplicated" : false
} ],
"cursors" : {
"sub" : {
"markDeletePosition" : "2025:830",
"readPosition" : "2025:831",
"waitingReadOp" : true,
"pendingReadOps" : 0,
"messagesConsumedCounter" : 0,
"cursorLedger" : -1,
"cursorLedgerLastEntry" : -1,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2023-08-16T11:03:02.464Z",
"state" : "NoLedger",
"active" : true,
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : true,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
},
"multiTopicsReader-883d8df4ce" : {
"markDeletePosition" : "2025:-1",
"readPosition" : "2150:0",
"waitingReadOp" : true,
"pendingReadOps" : 0,
"messagesConsumedCounter" : -831,
"cursorLedger" : -1,
"cursorLedgerLastEntry" : -1,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2023-08-16T11:03:29.403Z",
"state" : "Open",
"active" : true,
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : true,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
}
},
"schemaLedgers" : [ ],
"compactedLedger" : {
"ledgerId" : -1,
"entries" : -1,
"size" : -1,
"offloaded" : false,
"underReplicated" : false
}
}
{
"entriesAddedCounter" : 0,
"numberOfEntries" : 0,
"totalSize" : 0,
"currentLedgerEntries" : 0,
"currentLedgerSize" : 0,
"lastLedgerCreatedTimestamp" : "2023-08-16T11:03:03.109Z",
"waitingCursorsCount" : 2,
"pendingAddEntriesCount" : 0,
"lastConfirmedEntry" : "2030:831",
"state" : "LedgerOpened",
"ledgers" : [ {
"ledgerId" : 2158,
"entries" : 0,
"size" : 0,
"offloaded" : false,
"underReplicated" : false
} ],
"cursors" : {
"sub" : {
"markDeletePosition" : "2030:831",
"readPosition" : "2030:832",
"waitingReadOp" : true,
"pendingReadOps" : 0,
"messagesConsumedCounter" : 0,
"cursorLedger" : -1,
"cursorLedgerLastEntry" : -1,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2023-08-16T11:03:03.183Z",
"state" : "NoLedger",
"active" : true,
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : true,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
},
"multiTopicsReader-883d8df4ce" : {
"markDeletePosition" : "2030:-1",
"readPosition" : "2158:0",
"waitingReadOp" : true,
"pendingReadOps" : 0,
"messagesConsumedCounter" : -832,
"cursorLedger" : -1,
"cursorLedgerLastEntry" : -1,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2023-08-16T11:03:29.392Z",
"state" : "Open",
"active" : true,
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : true,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
}
},
"schemaLedgers" : [ ],
"compactedLedger" : {
"ledgerId" : -1,
"entries" : -1,
"size" : -1,
"offloaded" : false,
"underReplicated" : false
}
}
{
"entriesAddedCounter" : 0,
"numberOfEntries" : 0,
"totalSize" : 0,
"currentLedgerEntries" : 0,
"currentLedgerSize" : 0,
"lastLedgerCreatedTimestamp" : "2023-08-16T11:02:26.727Z",
"waitingCursorsCount" : 2,
"pendingAddEntriesCount" : 0,
"lastConfirmedEntry" : "2023:832",
"state" : "LedgerOpened",
"ledgers" : [ {
"ledgerId" : 2126,
"entries" : 0,
"size" : 0,
"offloaded" : false,
"underReplicated" : false
} ],
"cursors" : {
"sub" : {
"markDeletePosition" : "2023:832",
"readPosition" : "2023:833",
"waitingReadOp" : true,
"pendingReadOps" : 0,
"messagesConsumedCounter" : 0,
"cursorLedger" : -1,
"cursorLedgerLastEntry" : -1,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2023-08-16T11:02:27.185Z",
"state" : "NoLedger",
"active" : true,
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : true,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
},
"multiTopicsReader-883d8df4ce" : {
"markDeletePosition" : "2023:-1",
"readPosition" : "2126:0",
"waitingReadOp" : true,
"pendingReadOps" : 0,
"messagesConsumedCounter" : -833,
"cursorLedger" : -1,
"cursorLedgerLastEntry" : -1,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2023-08-16T11:03:29.362Z",
"state" : "Open",
"active" : true,
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : true,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
}
},
"schemaLedgers" : [ ],
"compactedLedger" : {
"ledgerId" : -1,
"entries" : -1,
"size" : -1,
"offloaded" : false,
"underReplicated" : false
}
}
{
"entriesAddedCounter" : 0,
"numberOfEntries" : 0,
"totalSize" : 0,
"currentLedgerEntries" : 0,
"currentLedgerSize" : 0,
"lastLedgerCreatedTimestamp" : "2023-08-16T11:03:03.437Z",
"waitingCursorsCount" : 2,
"pendingAddEntriesCount" : 0,
"lastConfirmedEntry" : "2037:826",
"state" : "LedgerOpened",
"ledgers" : [ {
"ledgerId" : 2169,
"entries" : 0,
"size" : 0,
"offloaded" : false,
"underReplicated" : false
} ],
"cursors" : {
"sub" : {
"markDeletePosition" : "2037:826",
"readPosition" : "2037:827",
"waitingReadOp" : true,
"pendingReadOps" : 0,
"messagesConsumedCounter" : 0,
"cursorLedger" : -1,
"cursorLedgerLastEntry" : -1,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2023-08-16T11:03:03.713Z",
"state" : "NoLedger",
"active" : true,
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : true,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
},
"multiTopicsReader-883d8df4ce" : {
"markDeletePosition" : "2037:-1",
"readPosition" : "2169:0",
"waitingReadOp" : true,
"pendingReadOps" : 0,
"messagesConsumedCounter" : -827,
"cursorLedger" : -1,
"cursorLedgerLastEntry" : -1,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2023-08-16T11:03:29.395Z",
"state" : "Open",
"active" : true,
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : true,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
}
},
"schemaLedgers" : [ ],
"compactedLedger" : {
"ledgerId" : -1,
"entries" : -1,
"size" : -1,
"offloaded" : false,
"underReplicated" : false
}
}
{
"entriesAddedCounter" : 0,
"numberOfEntries" : 0,
"totalSize" : 0,
"currentLedgerEntries" : 0,
"currentLedgerSize" : 0,
"lastLedgerCreatedTimestamp" : "2023-08-16T11:03:03.074Z",
"waitingCursorsCount" : 2,
"pendingAddEntriesCount" : 0,
"lastConfirmedEntry" : "2028:841",
"state" : "LedgerOpened",
"ledgers" : [ {
"ledgerId" : 2157,
"entries" : 0,
"size" : 0,
"offloaded" : false,
"underReplicated" : false
} ],
"cursors" : {
"sub" : {
"markDeletePosition" : "2028:841",
"readPosition" : "2028:842",
"waitingReadOp" : true,
"pendingReadOps" : 0,
"messagesConsumedCounter" : 0,
"cursorLedger" : -1,
"cursorLedgerLastEntry" : -1,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2023-08-16T11:03:03.172Z",
"state" : "NoLedger",
"active" : true,
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : true,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
},
"multiTopicsReader-883d8df4ce" : {
"markDeletePosition" : "2028:-1",
"readPosition" : "2157:0",
"waitingReadOp" : true,
"pendingReadOps" : 0,
"messagesConsumedCounter" : -842,
"cursorLedger" : -1,
"cursorLedgerLastEntry" : -1,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2023-08-16T11:03:29.384Z",
"state" : "Open",
"active" : true,
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : true,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
}
},
"schemaLedgers" : [ ],
"compactedLedger" : {
"ledgerId" : -1,
"entries" : -1,
"size" : -1,
"offloaded" : false,
"underReplicated" : false
}
}
{
"entriesAddedCounter" : 0,
"numberOfEntries" : 0,
"totalSize" : 0,
"currentLedgerEntries" : 0,
"currentLedgerSize" : 0,
"lastLedgerCreatedTimestamp" : "2023-08-16T11:03:02.068Z",
"waitingCursorsCount" : 2,
"pendingAddEntriesCount" : 0,
"lastConfirmedEntry" : "2027:827",
"state" : "LedgerOpened",
"ledgers" : [ {
"ledgerId" : 2139,
"entries" : 0,
"size" : 0,
"offloaded" : false,
"underReplicated" : false
} ],
"cursors" : {
"sub" : {
"markDeletePosition" : "2027:827",
"readPosition" : "2027:828",
"waitingReadOp" : true,
"pendingReadOps" : 0,
"messagesConsumedCounter" : 0,
"cursorLedger" : -1,
"cursorLedgerLastEntry" : -1,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2023-08-16T11:03:02.224Z",
"state" : "NoLedger",
"active" : true,
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : true,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
},
"multiTopicsReader-883d8df4ce" : {
"markDeletePosition" : "2027:-1",
"readPosition" : "2139:0",
"waitingReadOp" : true,
"pendingReadOps" : 0,
"messagesConsumedCounter" : -828,
"cursorLedger" : -1,
"cursorLedgerLastEntry" : -1,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2023-08-16T11:03:29.373Z",
"state" : "Open",
"active" : true,
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"subscriptionHavePendingRead" : true,
"subscriptionHavePendingReplayRead" : false,
"properties" : { }
}
},
"schemaLedgers" : [ ],
"compactedLedger" : {
"ledgerId" : -1,
"entries" : -1,
"size" : -1,
"offloaded" : false,
"underReplicated" : false
}
}
```
The logs for `public/default/persistent/test-partitioned-10-partition-0`
<img width="2927" alt="Screenshot 2023-08-16 at 13 11 36" src="https://github.com/apache/pulsar/assets/4356553/f19f1499-268d-46e1-abce-e7d010bc4b55">
You may see that the ledger 2022 (which was holding the data) was deleted
The logs for `persistent://public/default/test-partitioned-10-partition-0`
<img width="2919" alt="Screenshot 2023-08-16 at 13 23 29" src="https://github.com/apache/pulsar/assets/4356553/4d26181e-fd52-471f-854c-8d74fe1c6d34">
Notice that `policies` to the topic are applied after the ledger is removed. It also runing on different threads. Note that all 3 brokers were managing the topic (log entries per broker: broker-0: 23, broker-1: 23, broker-2: 1).
(please verify if it may be relevant):
<img width="2922" alt="Screenshot 2023-08-16 at 13 28 29" src="https://github.com/apache/pulsar/assets/4356553/29aa7849-1c90-46c2-9edf-1b4d5ac1340b">
The logs for ledger `#2022`
<img width="2924" alt="Screenshot 2023-08-16 at 13 15 19" src="https://github.com/apache/pulsar/assets/4356553/7d95eab7-80e5-4618-ac2a-e6a2ce2590c1">
The logs for current ledger `#2110`:
<img width="2918" alt="Screenshot 2023-08-16 at 13 16 36" src="https://github.com/apache/pulsar/assets/4356553/6bd9cbc6-d4fa-4abf-bb72-23f103e7f5a6">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1685834315
> Looks like setting the: `brokerDeleteInactiveTopicsEnabled=false` does not fix the problem. The ledgers are being deleted.
are there any error logs ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1685865694
> Thanks for the update. Most likely, it could be an issue from the topic policies if a dedicated namespace retention policy works fine.
>
> loop @mattisonchao and @Technoboy- here for the potential topic policy issue.
Now, there is a situation that if the topic is successfully created, the topic policies may use ns or broker level, because topic polices have a delay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1678356174
I don't think we have related fixes in 3.1.0. Is It only happening on the partitioned topic?
I also tried to reproduce the issue on my laptop, but not able to reproduce it.
My reproduction steps:
1. Change the `managedLedgerMinLedgerRolloverTimeMinutes=1` in broker.conf to make the ledger rollover more frequently
2. Create a partitioned topic (bin/pulsar-admin topics create-partitioned-topic test -p 3)
3. Set data retention (bin/pulsar-admin topics set-retention -s -1 -t -1 test)
4. Get data retention to make sure the data retention is applied properly
5. Start a perf producer to publish messages to the topic (bin/pulsar-perf produce -bm 1 -r 10000 test)
6. Check the internal stats to see if there are ledgers that have been removed (bin/pulsar-admin topics stats-internal test-partition-0)
```json
{
"entriesAddedCounter" : 3249591,
"numberOfEntries" : 3249591,
"totalSize" : 3461950342,
"currentLedgerEntries" : 45797,
"currentLedgerSize" : 48819602,
"lastLedgerCreatedTimestamp" : "2023-08-15T10:51:13.877+08:00",
"waitingCursorsCount" : 0,
"pendingAddEntriesCount" : 0,
"lastConfirmedEntry" : "67:45796",
"state" : "LedgerOpened",
"ledgers" : [ {
"ledgerId" : 19,
"entries" : 200046,
"size" : 213032478,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 22,
"entries" : 200126,
"size" : 213134190,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 25,
"entries" : 200173,
"size" : 213184245,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 28,
"entries" : 200887,
"size" : 213944655,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 31,
"entries" : 199916,
"size" : 212910540,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 34,
"entries" : 200138,
"size" : 213146970,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 37,
"entries" : 200400,
"size" : 213426000,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 40,
"entries" : 200165,
"size" : 213175725,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 43,
"entries" : 200232,
"size" : 213247080,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 46,
"entries" : 200352,
"size" : 213374880,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 49,
"entries" : 199942,
"size" : 213043455,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 52,
"entries" : 200235,
"size" : 213450510,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 55,
"entries" : 200280,
"size" : 213498480,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 58,
"entries" : 200405,
"size" : 213631730,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 61,
"entries" : 200196,
"size" : 213408936,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 64,
"entries" : 200301,
"size" : 213520866,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 67,
"entries" : 0,
"size" : 0,
"offloaded" : false,
"underReplicated" : false
} ],
"cursors" : { },
"schemaLedgers" : [ ],
"compactedLedger" : {
"ledgerId" : -1,
"entries" : -1,
"size" : -1,
"offloaded" : false,
"underReplicated" : false
}
}
```
No ledgers were removed.
> Still the question is how it's going to work with compacted, partitioned topics, which should also retain the messages.
Have you enabled compaction for the topic? The compacted data will have infinite data retention
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1700984001
I can confirm, that we do not experience this problem anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1688300916
@codelipenghui Thanks for providing the fix.
Are there any plans to release a hotfix or should we wait for 3.2.0? Once it's released I can verify the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1685925610
> I haven't notice any warn or errors. Please checkout: https://github.com/websight-io/pulsar-chaos-test-perf/
>
> You should be able to recreate the issue by running single SH script. I also think that the screenshot mentioned here may show the sequence of deleting a ledger/applying policy: [#20968 (comment)](https://github.com/apache/pulsar/issues/20968#issuecomment-1673393274)
>
> Please search for "Policies updated successfully"
@michalcukierman I have found one potential case that may cause the topic retention policy not work. I will push a fix later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1673359573
Just re-run the tests:
- I didn't obverse the problem on non-partitioned topics
- I didn't obverse the problem when at least one broker remains active
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1680784827
I changed the repository visibility to public:
Here are the values I am using:
https://github.com/websight-io/pulsar-chaos-test-perf/blob/main/values.yaml
You can find two scripts there I use to reproduce the error.
I will test the settings you mentioned and the namespace level policies tomorrow.
I will also perform the test on Pulsar 2.10.4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1677368406
@codelipenghui
I set only the retention policy, which is a part of topic creation. Done using admin commands before producing any messages. I can validate it by running get policy on topic before the restart and after the restart. The output is:
'''
{
"retentionTimeInMinutes" : -1,
"retentionSizeInMB" : -1
}
'''
It may be wrong, but maybe during replication(?) of a topic there is a race condition.
You may ignore my assumptions about the cause of the issue. Its just what i can see from logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui closed issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui closed issue #20968: [Bug] Brokers restart during read/writes causes data loss
URL: https://github.com/apache/pulsar/issues/20968
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1677158647
@michalcukierman Which policy have you applied? It looks like either all the consumers acknowledged the messages or no subscriptions on the topic, then the broker will remove the data from the topic. Have you enabled data retention for that topic or namespace?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1682312914
Thanks for the update. Most likely, it could be an issue from the topic policies if a dedicated namespace retention policy works fine.
loop @mattisonchao and @Technoboy- here for the potential topic policy issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1689169361
@michalcukierman We can provide an image for you for testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1701233860
Oh, sorry. We just started a release this week. However, the release process is blocked by some tests. It's good to know the issue has been fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1671828353
Need to test it better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman closed issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman closed issue #20968: [Bug] Brokers restart during read/writes causes data loss
URL: https://github.com/apache/pulsar/issues/20968
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1673010782
I've just pushed fresh execution on Pulsar 3.0.1:
https://github.com/websight-io/pulsar-chaos-test/tree/main-3.0.1
I am getting:
```
Messages available before restart:
206
Messages available after restart:
30
```
Stats from before the restart:
https://github.com/websight-io/pulsar-chaos-test/blob/main-3.0.1/out/pre-stats.out
https://github.com/websight-io/pulsar-chaos-test/blob/main-3.0.1/out/pre-stats-internal.out
Stats from after the restart:
https://github.com/websight-io/pulsar-chaos-test/blob/main-3.0.1/out/post-stats.out
https://github.com/websight-io/pulsar-chaos-test/blob/main-3.0.1/out/post-stats-internal.out
If I create a new client, I am able to read 30 messages. All new produced messages are delivered, but the historical 176 are lost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1689572458
That would be great, to verify if the fix addresses the problem. Looking forward to the image.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman commented on issue #20968: [Bug] Brokers restart during read/writes causes data loss
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1691859206
@codelipenghui When can I expect the image? I would love to confirm the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org