You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "michalcukierman (via GitHub)" <gi...@apache.org> on 2023/08/27 16:32:41 UTC
[GitHub] [pulsar] michalcukierman opened a new issue, #21074: [Bug] Compaction failure leads to messages loss and messages redelivery loop
michalcukierman opened a new issue, #21074:
URL: https://github.com/apache/pulsar/issues/21074
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
3.1.0 using official Helm chart
Latest build from master using official Helm chart
### Minimal reproduce step
Use 3 bookies, 3 brokers, 3 proxies
1. Create compacted, partitioned topic with the compaction threshold set to 1GB
2. Produce 100k of messages of 100 kb size
3. Create exclusive consumer and start reading
4. Produce 100k of messages with the same keys
-- wait for the messages to be produced and the compaction to finish
6. Create new exclusive consumer and read all the messages
### What did you expect to see?
Compaction finish successfully.
First consumer receives between 100k - 200k messages
Second consumer receives 100k messages
### What did you see instead?
- I see in the error logs that the compaction fails.
<img width="1618" alt="Screenshot 2023-08-27 at 18 02 33" src="https://github.com/apache/pulsar/assets/4356553/b26ff482-de67-4b6c-96d9-6cec98bc8e13">
- The first consumer often starts to receive messages, that were already delivered (millions of messages, It never ends)
- The second consumer cannot receive all the messages. Sometimes it is able to finish, sometimes is not (falls into the loop)
- The backlog of the subscription does not change while the consumers are reading
Reads comes from one broker, the backlog is not changing (the screenshot is from my environment/system):
<img width="1679" alt="Screenshot 2023-08-27 at 18 29 22" src="https://github.com/apache/pulsar/assets/4356553/87803ebd-3488-4092-bb64-15444123c21b">
### Anything else?
The reproduction steps depends on the used setup, speed of the persistent storage. It's possible that the re balancing or adding new broker affects the steps. I cannot provide reproducible way, as it happens randomly. Nevertheless it happens to us almost every day.
My internal stats:
[internal.stats.txt](https://github.com/apache/pulsar/files/12448404/internal.stats.txt)
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Topic unloading during Compaction leads to messages loss and messages redelivery loop [pulsar]
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui closed issue #21074: [Bug] Topic unloading during Compaction leads to messages loss and messages redelivery loop
URL: https://github.com/apache/pulsar/issues/21074
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman commented on issue #21074: [Bug] Compaction failure leads to messages loss and messages redelivery loop
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman commented on issue #21074:
URL: https://github.com/apache/pulsar/issues/21074#issuecomment-1695541238
@coderzc It's not a problem with duplicated messages.
Check the [comment](https://github.com/apache/pulsar/issues/21074#issuecomment-1694749730), I put some numbers there.
- Produced messages: 60k
- Messages in compacted ledger: 87051
- Consumed messages 127k and growing (I stopped at 250k)
So the issue is not about duplicated messages, but the redelivery loop. Sometimes messages loss.
In both cases the backlog (Grafana and topic stats) does not change during the time.
Also, If you have duplicated messages in compacted ledger, you would expect different keys, right?
I get the same key, from the same ledger over and over:
```
# 306 -> (144,0,1,-1)
# 5500 -> (144,0,1,-1)
# 8862 -> (144,0,1,-1)
# 11079 -> (144,0,1,-1)
# 12825 -> (144,0,1,-1)
# 15426 -> (144,0,1,-1)
# 17804 -> (144,0,1,-1)
# 19672 -> (144,0,1,-1)
# 23374 -> (144,0,1,-1)
```
so just with the example above I read `23374` messages during a single run probably around 100k during last hour.
The compacted ledger entries is exactly 10k:
```bash
➜ chaos-test-pulsar-perf git:(main) ✗ for i in {0..12} ; do kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin topics stats-internal test-compaction-0-partition-$i | jq -r ".compactedLedger.entries" ;done
851
835
816
833
803
824
823
837
823
842
843
870
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman commented on issue #21074: [Bug] Compaction failure leads to messages loss and messages redelivery loop
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman commented on issue #21074:
URL: https://github.com/apache/pulsar/issues/21074#issuecomment-1695271984
@coderzc, I don't think it's the same issue. We observe two situations:
- consumer is not able to read all messages, even if the backlog is not empty (messages loss)
- consumer is falling into infinite loop (starts to read the same messages in loop, even after sending acks)
In the [test above](https://github.com/apache/pulsar/issues/21074#issuecomment-1694749730) the consumer finally received more than 250k messages, even if the number of produced messages was 60k.
In most of the cases where I see the Issue on our setup I publish around 1mln messages, the consumer reads around 600k-700k and stops receiving unique messages (even if it still receives 1k msg/s, all are duplicated).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] coderzc commented on issue #21074: [Bug] Compaction failure leads to messages loss and messages redelivery loop
Posted by "coderzc (via GitHub)" <gi...@apache.org>.
coderzc commented on issue #21074:
URL: https://github.com/apache/pulsar/issues/21074#issuecomment-1695470554
It seems to be the same issue as what I said above. If the topic unloads or the broker restarts, the consumer will reconnect to the broker and start reading data from the earliest position again, even if you already acknowledged some messages.
Also, this issue may also occur during the compaction process, and it will cause compacted data to have duplicate messages.
@michalcukierman Can you see if the number of compacted entries is greater than the total number of messages published?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman commented on issue #21074: [Bug] Topic unloading during Compaction leads to messages loss and messages redelivery loop
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman commented on issue #21074:
URL: https://github.com/apache/pulsar/issues/21074#issuecomment-1695889478
Thanks, that may be in fact be the cause. I'll test it more and let you know.
In the meantime:
- Are there any workarounds for the issue?
- Are there plans to fix the behavior?
For now the only options I see are:
- Not to use compaction
- or disable Load Balancing and make sure that the brokers don't restart to often
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman commented on issue #21074: [Bug] Compaction failure leads to messages loss and messages redelivery loop
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman commented on issue #21074:
URL: https://github.com/apache/pulsar/issues/21074#issuecomment-1694749730
I was able to reproduce it with the topic with 60k messages.
1. Create the topic, run the compaction (to create subscription)
```bash
TOPIC="test-compaction"
kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin topics create-partitioned-topic $TOPIC -p 12
kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin topics compact $TOPIC
kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-perf produce -bm 1 -r 300 -m 10000 -s 102400 $TOPIC
```
2. Connect with the client:
```python
import pulsar
client = pulsar.Client('pulsar://localhost:6650')
consumer = client.subscribe('test-compaction',
subscription_name='python-consume',
initial_position=pulsar.InitialPosition.Earliest,
is_read_compacted=True)
count=0
while True:
msg = consumer.receive()
count += 1
print(count)
consumer.acknowledge(msg)
client.close()
```
by invoking:
```bash
kubectl port-forward service/pulsar-proxy 6650:6650 -n pulsar &
python3 consumer.py
```
3. Run the compaction
```bash
kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin topics compact ${TOPIC}
```
4. Unlaod the topicc
```bash
kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin topics unload ${TOPIC}
```
The result:
- python client fall in infinite loop, I've stopped it after receiving `127557` messages.
- after restarting the client, it still receives new messages
Other information:
- Pulsar Manager showing `out rate` and `precise backlog`:
<img width="1474" alt="Screenshot 2023-08-27 at 21 53 33" src="https://github.com/apache/pulsar/assets/4356553/a11da3b8-8680-4845-b7f1-0043c63e722f">
- Messages backlog is not from Gradana (last 10 minutes without changes)
<img width="839" alt="Screenshot 2023-08-27 at 21 58 23" src="https://github.com/apache/pulsar/assets/4356553/cbd18ae9-b5e8-4937-9021-464b309ea821">
- Compacted ledger entries (87051):
```bash
➜ chaos-test-pulsar-perf git:(main) ✗ for i in {0..12} ; do kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin topics stats-internal test-compaction-2-partition-$i | jq -r ".compactedLedger.entries" ;done
8718
12692
8486
5113
11155
5077
5113
9513
5147
5151
5147
5739
```
I used topic unloading just to reproduce the issue. In our case it could be cuased by:
- during broker restart on OOM on Direct Memory
- during brokers scaling (i.e. adding broker)
- topics re-balancing (bundles spliting)
I see this issue a lot during broker restart.
Sometimes we observe it without the restarts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] coderzc commented on issue #21074: [Bug] Compaction failure leads to messages loss and messages redelivery loop
Posted by "coderzc (via GitHub)" <gi...@apache.org>.
coderzc commented on issue #21074:
URL: https://github.com/apache/pulsar/issues/21074#issuecomment-1694987165
@michalcukierman Whether the consumer always receive duplicate messages even though the message has been acknowledged?
If so, then this is a known issue, as the consumer reads from the earliest position after the consumer reconnects. Refer to: https://github.com/apache/pulsar/pull/11287.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] coderzc commented on issue #21074: [Bug] Topic unloading during Compaction leads to messages loss and messages redelivery loop
Posted by "coderzc (via GitHub)" <gi...@apache.org>.
coderzc commented on issue #21074:
URL: https://github.com/apache/pulsar/issues/21074#issuecomment-1695913753
There is currently no other way, we will fix this behavior as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman commented on issue #21074: [Bug] Topic unloading during Compaction leads to messages loss and messages redelivery loop
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman commented on issue #21074:
URL: https://github.com/apache/pulsar/issues/21074#issuecomment-1695854433
Is it possible that broker bundles load balancing always triggers clients to reconnect and start reading from the beginning?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] coderzc commented on issue #21074: [Bug] Topic unloading during Compaction leads to messages loss and messages redelivery loop
Posted by "coderzc (via GitHub)" <gi...@apache.org>.
coderzc commented on issue #21074:
URL: https://github.com/apache/pulsar/issues/21074#issuecomment-1695617540
> So the issue is not about duplicated messages, but the redelivery loop. Sometimes messages loss.
Yes，my means is redelivery loop，since the readPosition will be reset to earliest position after consumer reconnect to broker when topic is unloaded or broker restart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Topic unloading during Compaction leads to messages loss and messages redelivery loop [pulsar]
Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- commented on issue #21074:
URL: https://github.com/apache/pulsar/issues/21074#issuecomment-1884724176
> There is currently no other way, we will fix this behavior as soon as possible.
we have fixed some issues related the compaction, i don't know if this issue has been fixed. @coderzc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] coderzc commented on issue #21074: [Bug] Topic unloading during Compaction leads to messages loss and messages redelivery loop
Posted by "coderzc (via GitHub)" <gi...@apache.org>.
coderzc commented on issue #21074:
URL: https://github.com/apache/pulsar/issues/21074#issuecomment-1695862668
> Is it possible that broker bundles load balancing always triggers clients to reconnect and start reading from the beginning?
Yes, when the broker bundles load balancing, the consumer will reconnect to the broker, and the consumer will start reading from the beginning again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman commented on issue #21074: [Bug] Topic unloading during Compaction leads to messages loss and messages redelivery loop
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman commented on issue #21074:
URL: https://github.com/apache/pulsar/issues/21074#issuecomment-1695801069
@coderzc
> Yes，my means is redelivery loop，since the readPosition will be reset to earliest position after consumer reconnect to broker when topic is unloaded or broker restart.
Can you explain me where is the loop? Reconnection happens once. After that the loop is infinite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michalcukierman commented on issue #21074: [Bug] Compaction failure leads to messages loss and messages redelivery loop
Posted by "michalcukierman (via GitHub)" <gi...@apache.org>.
michalcukierman commented on issue #21074:
URL: https://github.com/apache/pulsar/issues/21074#issuecomment-1695434624
Having the script:
```bash
trap printout SIGINT
printout() {
[[ ! -z "$PYTHON_PID" ]] && echo "\nExiting... Killing Python Consumer" && kill -15 $PYTHON_PID
exit
}
TOPIC="test-compaction-0"
echo "Create topic"
kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin topics create-partitioned-topic ${TOPIC} -p 12
kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin topics compact ${TOPIC}
echo "Produce messages"
kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-perf produce -bm 1 -r 300 -m 10000 -s 102400 ${TOPIC}
# Compact topic
echo "Compact topic"
kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin topics compact ${TOPIC}
# Unload topic and start reading the results
sleep 10 && echo "Unload topic"
kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin topics unload ${TOPIC}
echo "Start reading"
python3 consumer.py ${TOPIC} &
PYTHON_PID=$!
# Unload topic again (don't know if it's required)
sleep 10 && echo "Unload topic"
kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin topics unload ${TOPIC}
sleep 10 && echo "Unload topic"
kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin topics unload ${TOPIC}
sleep 10 && echo "Unload topic"
kubectl exec --namespace pulsar -t pulsar-toolset-0 -- bin/pulsar-admin topics unload ${TOPIC}
sleep 100000 # wait for CTRL+C
```
and the python script `consumer.py`:
```python
import pulsar
import sys
client = pulsar.Client('pulsar://localhost:6650')
consumer = client.subscribe(sys.argv[1],
subscription_name='python-consume',
initial_position=pulsar.InitialPosition.Earliest,
is_read_compacted=True)
count=0
while True:
msg = consumer.receive()
count += 1
print("#", count, " -> ", msg.message_id())
consumer.acknowledge(msg)
client.close()
```
I can see that the messages are delivered in a loop.
The results after greping by the message ID: `python3 consumer.py ${TOPIC} | grep "(144,0"`
are:
<img width="664" alt="Screenshot 2023-08-28 at 12 22 37" src="https://github.com/apache/pulsar/assets/4356553/1f35e24b-133d-4ce2-9fee-ad8b8caf3143">
- The same message is re-delivered in a loop.
- All messages may never be delivered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #21074: [Bug] Topic unloading during Compaction leads to messages loss and messages redelivery loop
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #21074:
URL: https://github.com/apache/pulsar/issues/21074#issuecomment-1738324933
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org