You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/11/17 06:08:09 UTC
[GitHub] [pulsar] pkumar-singh opened a new issue, #18510: PIP-210: Retry producing on next partition if possible when a partition is not available
pkumar-singh opened a new issue, #18510:
URL: https://github.com/apache/pulsar/issues/18510
### Motivation
When a topic is a partitioned topic and a partition is not available for producing messages, currently pulsar client will still try to produce messages on unavailable partitions, which it may not necessarily need to do in certain cases. Pulsar Client may simply pick up another partition and try producing in certain cases.
Partition Unavailable
There could be a plethora of reasons a partition can become unavailable. But the most prominent reason is partition is moving from one broker to another, and until every actor is in sync with which broker owns the partition, the partition will be unavailable for producing. Actors are producers, old broker, new broker.
### Goal
Produce uninterrupted as long as possible when a partition is down.
### API Changes
pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ProducerBuilder.java
/**
* This config will ensure that If possible PartitionedProducer would attempt to produce message on
* another available partitions, If currently picked partition is not available for some reason.
* Next available partition will be chosen by the same routing policy as client is configured with.
* @param maxRetryOtherPartition
* How many partitions should be tried before bailing out
* @return the producer builder instance
*/
ProducerBuilder<T> maxRetryOtherPartitions(int maxRetryOtherPartition);
### Implementation
**Client Behavior**
This is the typical produce code.
producer.sendAsync(payLoad.getBytes(StandardCharsets.UTF_8));
When send is called message is enqueued in a queue(called pending message queue) and the future is returned.
And future is only completed when the message is picked from the queue and sent to the broker asynchronously and ack is received asynchronously again. Max size of the pending message queue is controlled by producer config maxPendingMessages.
When pending message queue is full, the application will start getting publish failures. Pending message queue provide a cushion towards unavailable partitions. But again it has some limits.
**When another partitions can be picked**
When the message is not keyed. That means the message is not ordered based on a key.
When routing mode is round-robin, that means a message can be produced to any of the partitions. So If a partition is unavailable try and pick up another partition for producing, by using the same round-robin algorithm.
### Alternatives
_No response_
### Anything else?
My suggestion is to keep Router(RoundRobin) not dependent on whether a partition is available or not. Or batching is enabled or publish is happening under transaction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #18510: PIP-210: Retry producing on next partition if possible when a partition is not available
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #18510:
URL: https://github.com/apache/pulsar/issues/18510#issuecomment-1374686148
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #18510: PIP-210: Retry producing on next partition if possible when a partition is not available
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #18510:
URL: https://github.com/apache/pulsar/issues/18510#issuecomment-1491181400
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on issue #18510: PIP-210: Retry producing on next partition if possible when a partition is not available
Posted by "michaeljmarshall (via GitHub)" <gi...@apache.org>.
michaeljmarshall commented on issue #18510:
URL: https://github.com/apache/pulsar/issues/18510#issuecomment-1447036247
@pkumar-singh - in order for the PIP to be approved, it must be discussed on the pulsar dev mailing list and then voted on. Let me know if you have any questions, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] PIP-210: Retry producing on next partition if possible when a partition is not available [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat closed issue #18510: PIP-210: Retry producing on next partition if possible when a partition is not available
URL: https://github.com/apache/pulsar/issues/18510
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] StevenLuMT commented on issue #18510: PIP-210: Retry producing on next partition if possible when a partition is not available
Posted by GitBox <gi...@apache.org>.
StevenLuMT commented on issue #18510:
URL: https://github.com/apache/pulsar/issues/18510#issuecomment-1342310501
> @StevenLuMT I did send a PR there were some comments on it too. If this PIP is approved I will address the comments.
@pkumar-singh can you send me the PR's link? I want to learn it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] pkumar-singh commented on issue #18510: PIP-210: Retry producing on next partition if possible when a partition is not available
Posted by GitBox <gi...@apache.org>.
pkumar-singh commented on issue #18510:
URL: https://github.com/apache/pulsar/issues/18510#issuecomment-1342307659
@StevenLuMT I did send a PR there were some comments on it too. If this PIP is approved I will address the comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] PIP-210: Retry producing on next partition if possible when a partition is not available [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on issue #18510:
URL: https://github.com/apache/pulsar/issues/18510#issuecomment-6377562976
Thanks @pkumar-singh for the proposal. PIP-210 never went to a vote on dev@ (discussion: https://lists.apache.org/thread/dz8cm6z57ph3gwnfd8o91c7zvhq7p58b), the implementation PR #18156 was closed, and there has been no activity for over three years, so I'm closing this as not planned. PIPs are now proposed as markdown documents under `pip/` in the repository. Anyone is welcome to restart this one by following pip/README.md.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] pkumar-singh commented on issue #18510: PIP-210: Retry producing on next partition if possible when a partition is not available
Posted by GitBox <gi...@apache.org>.
pkumar-singh commented on issue #18510:
URL: https://github.com/apache/pulsar/issues/18510#issuecomment-1342325586
#18156
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] StevenLuMT commented on issue #18510: PIP-210: Retry producing on next partition if possible when a partition is not available
Posted by GitBox <gi...@apache.org>.
StevenLuMT commented on issue #18510:
URL: https://github.com/apache/pulsar/issues/18510#issuecomment-1342295027
@pkumar-singh @sijie hello
How is the progress of this PIP, whether it is picked up the PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org