You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by ak...@gmail.com on 2019/03/31 23:24:58 UTC
Need help to find references to antipatterns/pitfalls/incorrect ways to use Kafka
Hello!
I ask for your help in connection with the my recent task:
- Price lists are delivered to 20,000 points of sale with a frequency of <10
price lists per day.
- The order in which the price lists follow is important. It is also
important that the price lists are delivered to the point of sale online.
- At each point of sale, an agent application is deployed, which processes
the received price lists.
This task is not particularly difficult. Help in solving the task is not
required.
The difficulty is that Kafka in our company is a new "silver bullet", and
the project manager requires me to implement the following technical
decision:
deploy 20,000 Kafka consumer instances (one instance for each point of sale)
for one topic partitioned into 20,000 partitions - one partition per
consumer.
Technical problems obtained in experiments with this technical decision do
not convince him.
Please give me references to the books/documents/blogposts. which clearly
shows that Kafka not intended for this way to use (references to other
anti-patterns/pitfalls will be useful).
My own attempts to find such references were unsuccessful.
Thank you!
Re: [External] Re: Need help to find references to
antipatterns/pitfalls/incorrect ways to use Kafka
Posted by "Tauzell, Dave" <Da...@surescripts.com>.
If somebody insists on using Kafka as a database you might be able to do the following:
1. Create a "compacted topic". The key for the topic should be the point of sales id.
2. Create a webservice which takes a point of sale id and can read or update the topic
3. Have the point of sale apps periodically call the webservice
This won't work if the price list is truly a stream of things rather than "just get the latest list".
-Dave
On 3/31/19, 7:01 PM, "Peter Bukowinski" <pm...@gmail.com> wrote:
I don’t want to be a downer, but because kafka is relatively new, the reference material you seek probably doesn’t exist. Kafka is flexible and can be made to work in many different scenarios — not all of the ideal.
It sounds like you’ve already reached a conclusion that kafka is the wrong solution for your requirements. Please share with us the evidence that you used to reach this conclusion. It would be helpful if you described the technical problems you encountered in your experiments so that others can give their opinion on whether can can be resolved or whether they are deal-breakers.
--
Peter
> On Mar 31, 2019, at 4:24 PM, <ak...@gmail.com> <ak...@gmail.com> wrote:
>
> Hello!
>
>
>
> I ask for your help in connection with the my recent task:
>
> - Price lists are delivered to 20,000 points of sale with a frequency of <10
> price lists per day.
>
> - The order in which the price lists follow is important. It is also
> important that the price lists are delivered to the point of sale online.
>
> - At each point of sale, an agent application is deployed, which processes
> the received price lists.
>
>
>
> This task is not particularly difficult. Help in solving the task is not
> required.
>
>
>
> The difficulty is that Kafka in our company is a new "silver bullet", and
> the project manager requires me to implement the following technical
> decision:
>
> deploy 20,000 Kafka consumer instances (one instance for each point of sale)
> for one topic partitioned into 20,000 partitions - one partition per
> consumer.
>
> Technical problems obtained in experiments with this technical decision do
> not convince him.
>
>
>
> Please give me references to the books/documents/blogposts. which clearly
> shows that Kafka not intended for this way to use (references to other
> anti-patterns/pitfalls will be useful).
>
> My own attempts to find such references were unsuccessful.
>
>
>
> Thank you!
>
>
>
This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
Re: Need help to find references to antipatterns/pitfalls/incorrect ways to use Kafka
Posted by Peter Bukowinski <pm...@gmail.com>.
I don’t want to be a downer, but because kafka is relatively new, the reference material you seek probably doesn’t exist. Kafka is flexible and can be made to work in many different scenarios — not all of the ideal.
It sounds like you’ve already reached a conclusion that kafka is the wrong solution for your requirements. Please share with us the evidence that you used to reach this conclusion. It would be helpful if you described the technical problems you encountered in your experiments so that others can give their opinion on whether can can be resolved or whether they are deal-breakers.
--
Peter
> On Mar 31, 2019, at 4:24 PM, <ak...@gmail.com> <ak...@gmail.com> wrote:
>
> Hello!
>
>
>
> I ask for your help in connection with the my recent task:
>
> - Price lists are delivered to 20,000 points of sale with a frequency of <10
> price lists per day.
>
> - The order in which the price lists follow is important. It is also
> important that the price lists are delivered to the point of sale online.
>
> - At each point of sale, an agent application is deployed, which processes
> the received price lists.
>
>
>
> This task is not particularly difficult. Help in solving the task is not
> required.
>
>
>
> The difficulty is that Kafka in our company is a new "silver bullet", and
> the project manager requires me to implement the following technical
> decision:
>
> deploy 20,000 Kafka consumer instances (one instance for each point of sale)
> for one topic partitioned into 20,000 partitions - one partition per
> consumer.
>
> Technical problems obtained in experiments with this technical decision do
> not convince him.
>
>
>
> Please give me references to the books/documents/blogposts. which clearly
> shows that Kafka not intended for this way to use (references to other
> anti-patterns/pitfalls will be useful).
>
> My own attempts to find such references were unsuccessful.
>
>
>
> Thank you!
>
>
>
Re: Need help to find references to antipatterns/pitfalls/incorrect
ways to use Kafka
Posted by Alexander Kuterin <ak...@gmail.com>.
Thanks, Dmitry.
It seems your solution is the the most appropriate for me, because the
reasons why consumers (POS terminals) will be added/removed are different
from the reasons why partitions will be added/removed.
I think that topic division to partitions and POS terminals division to
logical groups (each POS terminal from one group consumes one partition)
must be independent from each other. Creating specific partition for
specific POS terminal is a pitfall.
пн, 1 апр. 2019 г., 12:26 Dimitry Lvovsky <dl...@gmail.com>:
> Going off of what Hans mentioned, I don't see any reason for 200,000
> partitions...you don't need one partition per POS. You can have all of your
> pos listening to one partition and each pos agent having a unique group
> id. The POS agent only processes the messages that are relevant to him,
> and simply ignores the rest. If you have bandwidth or processing power
> concerns could also consider subdividing topics ( not partitions ) per
> regions.
>
> Dimitry
>
> On Mon, Apr 1, 2019 at 8:16 AM Hans Jespersen <ha...@confluent.io> wrote:
>
> > Yes but you have more than 1 POS terminal per location so you still don't
> > need 20,000 partitions. Just one per location. How many locations do you
> > have?
> >
> > In doesn’t matter anyway since you can build a Kafka cluster with up to
> > 200,000 partitions if you use the latest versions of Kafka.
> >
> >
> https://blogs.apache.org/kafka/entry/apache-kafka-supports-more-partitions
> >
> > “As a rule of thumb, we recommend each broker to have up to 4,000
> > partitions and each cluster to have up to 200,000 partitions”
> >
> > -hans
> >
> > > On Apr 1, 2019, at 2:02 AM, Alexander Kuterin <ak...@gmail.com>
> > wrote:
> > >
> > > Thanks, Hans!
> > > We use location specific SKU pricing and send specific price lists to
> the
> > > specific POS terminal.
> > >
> > > пн, 1 апр. 2019 г., 3:01 Hans Jespersen <ha...@confluent.io>:
> > >
> > >> Doesn’t every one of the 20,000 POS terminals want to get the same
> price
> > >> list messages? If so then there is no need for 20,000 partitions.
> > >>
> > >> -hans
> > >>
> > >>> On Mar 31, 2019, at 7:24 PM, <ak...@gmail.com> <
> akuterin@gmail.com>
> > >> wrote:
> > >>>
> > >>> Hello!
> > >>>
> > >>>
> > >>>
> > >>> I ask for your help in connection with the my recent task:
> > >>>
> > >>> - Price lists are delivered to 20,000 points of sale with a frequency
> > of
> > >> <10
> > >>> price lists per day.
> > >>>
> > >>> - The order in which the price lists follow is important. It is also
> > >>> important that the price lists are delivered to the point of sale
> > online.
> > >>>
> > >>> - At each point of sale, an agent application is deployed, which
> > >> processes
> > >>> the received price lists.
> > >>>
> > >>>
> > >>>
> > >>> This task is not particularly difficult. Help in solving the task is
> > not
> > >>> required.
> > >>>
> > >>>
> > >>>
> > >>> The difficulty is that Kafka in our company is a new "silver bullet",
> > and
> > >>> the project manager requires me to implement the following technical
> > >>> decision:
> > >>>
> > >>> deploy 20,000 Kafka consumer instances (one instance for each point
> of
> > >> sale)
> > >>> for one topic partitioned into 20,000 partitions - one partition per
> > >>> consumer.
> > >>>
> > >>> Technical problems obtained in experiments with this technical
> decision
> > >> do
> > >>> not convince him.
> > >>>
> > >>>
> > >>>
> > >>> Please give me references to the books/documents/blogposts. which
> > clearly
> > >>> shows that Kafka not intended for this way to use (references to
> other
> > >>> anti-patterns/pitfalls will be useful).
> > >>>
> > >>> My own attempts to find such references were unsuccessful.
> > >>>
> > >>>
> > >>>
> > >>> Thank you!
> > >>>
> > >>>
> > >>>
> > >>
> >
>
Re: Need help to find references to antipatterns/pitfalls/incorrect
ways to use Kafka
Posted by Alexander Kuterin <ak...@gmail.com>.
Thanks, Vincent!
Yes, i totally agree about coupling/decoupling. The need for the producer
to send messages to a specific partition and the consumer to consume from a
specific partition creates high coupling between them. So in the future
changing the topic partitioning scheme will be impossible without
reconfiguring/restarting consumers.
пн, 1 апр. 2019 г., 13:48 Vincent Maurin <vi...@gmail.com>:
> Maybe you can find some litterature about messaging patterns ?
>
> Usually, a single kafka topic is used to do PubSub pattern, i.e decoupling
> producers and consumers.
> In your case, it seems that the situation is quite coupled, i.e you need to
> generate and send 20k price lists to 20k specific consumers (so it is
> looking like more what is describe as PAIR for 0mq
>
> https://learning-0mq-with-pyzmq.readthedocs.io/en/latest/pyzmq/patterns/pair.html
> )
>
> I could still see Kafka as an option, but I am not sure about using
> partitioning for the split. I would rather go for one topics per point of
> sales with like the POS id in the name, with a single partition each.
> You guarantee the order, you know where to produce, where to consume from
> and you don't have any complicated consumption logic to implement. You just
> probably need to take care of the topic management programmatically.
>
> Best
>
>
> On Mon, Apr 1, 2019 at 11:26 AM Dimitry Lvovsky <dl...@gmail.com>
> wrote:
>
> > Going off of what Hans mentioned, I don't see any reason for 200,000
> > partitions...you don't need one partition per POS. You can have all of
> your
> > pos listening to one partition and each pos agent having a unique group
> > id. The POS agent only processes the messages that are relevant to him,
> > and simply ignores the rest. If you have bandwidth or processing power
> > concerns could also consider subdividing topics ( not partitions ) per
> > regions.
> >
> > Dimitry
> >
> > On Mon, Apr 1, 2019 at 8:16 AM Hans Jespersen <ha...@confluent.io> wrote:
> >
> > > Yes but you have more than 1 POS terminal per location so you still
> don't
> > > need 20,000 partitions. Just one per location. How many locations do
> you
> > > have?
> > >
> > > In doesn’t matter anyway since you can build a Kafka cluster with up to
> > > 200,000 partitions if you use the latest versions of Kafka.
> > >
> > >
> >
> https://blogs.apache.org/kafka/entry/apache-kafka-supports-more-partitions
> > >
> > > “As a rule of thumb, we recommend each broker to have up to 4,000
> > > partitions and each cluster to have up to 200,000 partitions”
> > >
> > > -hans
> > >
> > > > On Apr 1, 2019, at 2:02 AM, Alexander Kuterin <ak...@gmail.com>
> > > wrote:
> > > >
> > > > Thanks, Hans!
> > > > We use location specific SKU pricing and send specific price lists to
> > the
> > > > specific POS terminal.
> > > >
> > > > пн, 1 апр. 2019 г., 3:01 Hans Jespersen <ha...@confluent.io>:
> > > >
> > > >> Doesn’t every one of the 20,000 POS terminals want to get the same
> > price
> > > >> list messages? If so then there is no need for 20,000 partitions.
> > > >>
> > > >> -hans
> > > >>
> > > >>> On Mar 31, 2019, at 7:24 PM, <ak...@gmail.com> <
> > akuterin@gmail.com>
> > > >> wrote:
> > > >>>
> > > >>> Hello!
> > > >>>
> > > >>>
> > > >>>
> > > >>> I ask for your help in connection with the my recent task:
> > > >>>
> > > >>> - Price lists are delivered to 20,000 points of sale with a
> frequency
> > > of
> > > >> <10
> > > >>> price lists per day.
> > > >>>
> > > >>> - The order in which the price lists follow is important. It is
> also
> > > >>> important that the price lists are delivered to the point of sale
> > > online.
> > > >>>
> > > >>> - At each point of sale, an agent application is deployed, which
> > > >> processes
> > > >>> the received price lists.
> > > >>>
> > > >>>
> > > >>>
> > > >>> This task is not particularly difficult. Help in solving the task
> is
> > > not
> > > >>> required.
> > > >>>
> > > >>>
> > > >>>
> > > >>> The difficulty is that Kafka in our company is a new "silver
> bullet",
> > > and
> > > >>> the project manager requires me to implement the following
> technical
> > > >>> decision:
> > > >>>
> > > >>> deploy 20,000 Kafka consumer instances (one instance for each point
> > of
> > > >> sale)
> > > >>> for one topic partitioned into 20,000 partitions - one partition
> per
> > > >>> consumer.
> > > >>>
> > > >>> Technical problems obtained in experiments with this technical
> > decision
> > > >> do
> > > >>> not convince him.
> > > >>>
> > > >>>
> > > >>>
> > > >>> Please give me references to the books/documents/blogposts. which
> > > clearly
> > > >>> shows that Kafka not intended for this way to use (references to
> > other
> > > >>> anti-patterns/pitfalls will be useful).
> > > >>>
> > > >>> My own attempts to find such references were unsuccessful.
> > > >>>
> > > >>>
> > > >>>
> > > >>> Thank you!
> > > >>>
> > > >>>
> > > >>>
> > > >>
> > >
> >
>
Re: Need help to find references to antipatterns/pitfalls/incorrect
ways to use Kafka
Posted by Vincent Maurin <vi...@gmail.com>.
Maybe you can find some litterature about messaging patterns ?
Usually, a single kafka topic is used to do PubSub pattern, i.e decoupling
producers and consumers.
In your case, it seems that the situation is quite coupled, i.e you need to
generate and send 20k price lists to 20k specific consumers (so it is
looking like more what is describe as PAIR for 0mq
https://learning-0mq-with-pyzmq.readthedocs.io/en/latest/pyzmq/patterns/pair.html
)
I could still see Kafka as an option, but I am not sure about using
partitioning for the split. I would rather go for one topics per point of
sales with like the POS id in the name, with a single partition each.
You guarantee the order, you know where to produce, where to consume from
and you don't have any complicated consumption logic to implement. You just
probably need to take care of the topic management programmatically.
Best
On Mon, Apr 1, 2019 at 11:26 AM Dimitry Lvovsky <dl...@gmail.com> wrote:
> Going off of what Hans mentioned, I don't see any reason for 200,000
> partitions...you don't need one partition per POS. You can have all of your
> pos listening to one partition and each pos agent having a unique group
> id. The POS agent only processes the messages that are relevant to him,
> and simply ignores the rest. If you have bandwidth or processing power
> concerns could also consider subdividing topics ( not partitions ) per
> regions.
>
> Dimitry
>
> On Mon, Apr 1, 2019 at 8:16 AM Hans Jespersen <ha...@confluent.io> wrote:
>
> > Yes but you have more than 1 POS terminal per location so you still don't
> > need 20,000 partitions. Just one per location. How many locations do you
> > have?
> >
> > In doesn’t matter anyway since you can build a Kafka cluster with up to
> > 200,000 partitions if you use the latest versions of Kafka.
> >
> >
> https://blogs.apache.org/kafka/entry/apache-kafka-supports-more-partitions
> >
> > “As a rule of thumb, we recommend each broker to have up to 4,000
> > partitions and each cluster to have up to 200,000 partitions”
> >
> > -hans
> >
> > > On Apr 1, 2019, at 2:02 AM, Alexander Kuterin <ak...@gmail.com>
> > wrote:
> > >
> > > Thanks, Hans!
> > > We use location specific SKU pricing and send specific price lists to
> the
> > > specific POS terminal.
> > >
> > > пн, 1 апр. 2019 г., 3:01 Hans Jespersen <ha...@confluent.io>:
> > >
> > >> Doesn’t every one of the 20,000 POS terminals want to get the same
> price
> > >> list messages? If so then there is no need for 20,000 partitions.
> > >>
> > >> -hans
> > >>
> > >>> On Mar 31, 2019, at 7:24 PM, <ak...@gmail.com> <
> akuterin@gmail.com>
> > >> wrote:
> > >>>
> > >>> Hello!
> > >>>
> > >>>
> > >>>
> > >>> I ask for your help in connection with the my recent task:
> > >>>
> > >>> - Price lists are delivered to 20,000 points of sale with a frequency
> > of
> > >> <10
> > >>> price lists per day.
> > >>>
> > >>> - The order in which the price lists follow is important. It is also
> > >>> important that the price lists are delivered to the point of sale
> > online.
> > >>>
> > >>> - At each point of sale, an agent application is deployed, which
> > >> processes
> > >>> the received price lists.
> > >>>
> > >>>
> > >>>
> > >>> This task is not particularly difficult. Help in solving the task is
> > not
> > >>> required.
> > >>>
> > >>>
> > >>>
> > >>> The difficulty is that Kafka in our company is a new "silver bullet",
> > and
> > >>> the project manager requires me to implement the following technical
> > >>> decision:
> > >>>
> > >>> deploy 20,000 Kafka consumer instances (one instance for each point
> of
> > >> sale)
> > >>> for one topic partitioned into 20,000 partitions - one partition per
> > >>> consumer.
> > >>>
> > >>> Technical problems obtained in experiments with this technical
> decision
> > >> do
> > >>> not convince him.
> > >>>
> > >>>
> > >>>
> > >>> Please give me references to the books/documents/blogposts. which
> > clearly
> > >>> shows that Kafka not intended for this way to use (references to
> other
> > >>> anti-patterns/pitfalls will be useful).
> > >>>
> > >>> My own attempts to find such references were unsuccessful.
> > >>>
> > >>>
> > >>>
> > >>> Thank you!
> > >>>
> > >>>
> > >>>
> > >>
> >
>
Re: Need help to find references to antipatterns/pitfalls/incorrect
ways to use Kafka
Posted by Dimitry Lvovsky <dl...@gmail.com>.
Going off of what Hans mentioned, I don't see any reason for 200,000
partitions...you don't need one partition per POS. You can have all of your
pos listening to one partition and each pos agent having a unique group
id. The POS agent only processes the messages that are relevant to him,
and simply ignores the rest. If you have bandwidth or processing power
concerns could also consider subdividing topics ( not partitions ) per
regions.
Dimitry
On Mon, Apr 1, 2019 at 8:16 AM Hans Jespersen <ha...@confluent.io> wrote:
> Yes but you have more than 1 POS terminal per location so you still don't
> need 20,000 partitions. Just one per location. How many locations do you
> have?
>
> In doesn’t matter anyway since you can build a Kafka cluster with up to
> 200,000 partitions if you use the latest versions of Kafka.
>
> https://blogs.apache.org/kafka/entry/apache-kafka-supports-more-partitions
>
> “As a rule of thumb, we recommend each broker to have up to 4,000
> partitions and each cluster to have up to 200,000 partitions”
>
> -hans
>
> > On Apr 1, 2019, at 2:02 AM, Alexander Kuterin <ak...@gmail.com>
> wrote:
> >
> > Thanks, Hans!
> > We use location specific SKU pricing and send specific price lists to the
> > specific POS terminal.
> >
> > пн, 1 апр. 2019 г., 3:01 Hans Jespersen <ha...@confluent.io>:
> >
> >> Doesn’t every one of the 20,000 POS terminals want to get the same price
> >> list messages? If so then there is no need for 20,000 partitions.
> >>
> >> -hans
> >>
> >>> On Mar 31, 2019, at 7:24 PM, <ak...@gmail.com> <ak...@gmail.com>
> >> wrote:
> >>>
> >>> Hello!
> >>>
> >>>
> >>>
> >>> I ask for your help in connection with the my recent task:
> >>>
> >>> - Price lists are delivered to 20,000 points of sale with a frequency
> of
> >> <10
> >>> price lists per day.
> >>>
> >>> - The order in which the price lists follow is important. It is also
> >>> important that the price lists are delivered to the point of sale
> online.
> >>>
> >>> - At each point of sale, an agent application is deployed, which
> >> processes
> >>> the received price lists.
> >>>
> >>>
> >>>
> >>> This task is not particularly difficult. Help in solving the task is
> not
> >>> required.
> >>>
> >>>
> >>>
> >>> The difficulty is that Kafka in our company is a new "silver bullet",
> and
> >>> the project manager requires me to implement the following technical
> >>> decision:
> >>>
> >>> deploy 20,000 Kafka consumer instances (one instance for each point of
> >> sale)
> >>> for one topic partitioned into 20,000 partitions - one partition per
> >>> consumer.
> >>>
> >>> Technical problems obtained in experiments with this technical decision
> >> do
> >>> not convince him.
> >>>
> >>>
> >>>
> >>> Please give me references to the books/documents/blogposts. which
> clearly
> >>> shows that Kafka not intended for this way to use (references to other
> >>> anti-patterns/pitfalls will be useful).
> >>>
> >>> My own attempts to find such references were unsuccessful.
> >>>
> >>>
> >>>
> >>> Thank you!
> >>>
> >>>
> >>>
> >>
>
Re: Need help to find references to antipatterns/pitfalls/incorrect ways to use Kafka
Posted by Hans Jespersen <ha...@confluent.io>.
Yes but you have more than 1 POS terminal per location so you still don't need 20,000 partitions. Just one per location. How many locations do you have?
In doesn’t matter anyway since you can build a Kafka cluster with up to 200,000 partitions if you use the latest versions of Kafka.
https://blogs.apache.org/kafka/entry/apache-kafka-supports-more-partitions
“As a rule of thumb, we recommend each broker to have up to 4,000 partitions and each cluster to have up to 200,000 partitions”
-hans
> On Apr 1, 2019, at 2:02 AM, Alexander Kuterin <ak...@gmail.com> wrote:
>
> Thanks, Hans!
> We use location specific SKU pricing and send specific price lists to the
> specific POS terminal.
>
> пн, 1 апр. 2019 г., 3:01 Hans Jespersen <ha...@confluent.io>:
>
>> Doesn’t every one of the 20,000 POS terminals want to get the same price
>> list messages? If so then there is no need for 20,000 partitions.
>>
>> -hans
>>
>>> On Mar 31, 2019, at 7:24 PM, <ak...@gmail.com> <ak...@gmail.com>
>> wrote:
>>>
>>> Hello!
>>>
>>>
>>>
>>> I ask for your help in connection with the my recent task:
>>>
>>> - Price lists are delivered to 20,000 points of sale with a frequency of
>> <10
>>> price lists per day.
>>>
>>> - The order in which the price lists follow is important. It is also
>>> important that the price lists are delivered to the point of sale online.
>>>
>>> - At each point of sale, an agent application is deployed, which
>> processes
>>> the received price lists.
>>>
>>>
>>>
>>> This task is not particularly difficult. Help in solving the task is not
>>> required.
>>>
>>>
>>>
>>> The difficulty is that Kafka in our company is a new "silver bullet", and
>>> the project manager requires me to implement the following technical
>>> decision:
>>>
>>> deploy 20,000 Kafka consumer instances (one instance for each point of
>> sale)
>>> for one topic partitioned into 20,000 partitions - one partition per
>>> consumer.
>>>
>>> Technical problems obtained in experiments with this technical decision
>> do
>>> not convince him.
>>>
>>>
>>>
>>> Please give me references to the books/documents/blogposts. which clearly
>>> shows that Kafka not intended for this way to use (references to other
>>> anti-patterns/pitfalls will be useful).
>>>
>>> My own attempts to find such references were unsuccessful.
>>>
>>>
>>>
>>> Thank you!
>>>
>>>
>>>
>>
Re: Need help to find references to antipatterns/pitfalls/incorrect
ways to use Kafka
Posted by Alexander Kuterin <ak...@gmail.com>.
Thanks, Hans!
We use location specific SKU pricing and send specific price lists to the
specific POS terminal.
пн, 1 апр. 2019 г., 3:01 Hans Jespersen <ha...@confluent.io>:
> Doesn’t every one of the 20,000 POS terminals want to get the same price
> list messages? If so then there is no need for 20,000 partitions.
>
> -hans
>
> > On Mar 31, 2019, at 7:24 PM, <ak...@gmail.com> <ak...@gmail.com>
> wrote:
> >
> > Hello!
> >
> >
> >
> > I ask for your help in connection with the my recent task:
> >
> > - Price lists are delivered to 20,000 points of sale with a frequency of
> <10
> > price lists per day.
> >
> > - The order in which the price lists follow is important. It is also
> > important that the price lists are delivered to the point of sale online.
> >
> > - At each point of sale, an agent application is deployed, which
> processes
> > the received price lists.
> >
> >
> >
> > This task is not particularly difficult. Help in solving the task is not
> > required.
> >
> >
> >
> > The difficulty is that Kafka in our company is a new "silver bullet", and
> > the project manager requires me to implement the following technical
> > decision:
> >
> > deploy 20,000 Kafka consumer instances (one instance for each point of
> sale)
> > for one topic partitioned into 20,000 partitions - one partition per
> > consumer.
> >
> > Technical problems obtained in experiments with this technical decision
> do
> > not convince him.
> >
> >
> >
> > Please give me references to the books/documents/blogposts. which clearly
> > shows that Kafka not intended for this way to use (references to other
> > anti-patterns/pitfalls will be useful).
> >
> > My own attempts to find such references were unsuccessful.
> >
> >
> >
> > Thank you!
> >
> >
> >
>
Re: Need help to find references to antipatterns/pitfalls/incorrect ways to use Kafka
Posted by Hans Jespersen <ha...@confluent.io>.
Doesn’t every one of the 20,000 POS terminals want to get the same price list messages? If so then there is no need for 20,000 partitions.
-hans
> On Mar 31, 2019, at 7:24 PM, <ak...@gmail.com> <ak...@gmail.com> wrote:
>
> Hello!
>
>
>
> I ask for your help in connection with the my recent task:
>
> - Price lists are delivered to 20,000 points of sale with a frequency of <10
> price lists per day.
>
> - The order in which the price lists follow is important. It is also
> important that the price lists are delivered to the point of sale online.
>
> - At each point of sale, an agent application is deployed, which processes
> the received price lists.
>
>
>
> This task is not particularly difficult. Help in solving the task is not
> required.
>
>
>
> The difficulty is that Kafka in our company is a new "silver bullet", and
> the project manager requires me to implement the following technical
> decision:
>
> deploy 20,000 Kafka consumer instances (one instance for each point of sale)
> for one topic partitioned into 20,000 partitions - one partition per
> consumer.
>
> Technical problems obtained in experiments with this technical decision do
> not convince him.
>
>
>
> Please give me references to the books/documents/blogposts. which clearly
> shows that Kafka not intended for this way to use (references to other
> anti-patterns/pitfalls will be useful).
>
> My own attempts to find such references were unsuccessful.
>
>
>
> Thank you!
>
>
>
Re: Need help to find references to antipatterns/pitfalls/incorrect
ways to use Kafka
Posted by Alexander Kuterin <ak...@gmail.com>.
Yes, we have location specific SKU pricing.
пн, 1 апр. 2019 г., 4:08 Steve Howard <st...@confluent.io>:
> Do you have location specific SKU pricing?
>
> On Sun, Mar 31, 2019, 7:25 PM <akuterin@gmail.com wrote:
>
> > Hello!
> >
> >
> >
> > I ask for your help in connection with the my recent task:
> >
> > - Price lists are delivered to 20,000 points of sale with a frequency of
> > <10
> > price lists per day.
> >
> > - The order in which the price lists follow is important. It is also
> > important that the price lists are delivered to the point of sale online.
> >
> > - At each point of sale, an agent application is deployed, which
> processes
> > the received price lists.
> >
> >
> >
> > This task is not particularly difficult. Help in solving the task is not
> > required.
> >
> >
> >
> > The difficulty is that Kafka in our company is a new "silver bullet", and
> > the project manager requires me to implement the following technical
> > decision:
> >
> > deploy 20,000 Kafka consumer instances (one instance for each point of
> > sale)
> > for one topic partitioned into 20,000 partitions - one partition per
> > consumer.
> >
> > Technical problems obtained in experiments with this technical decision
> do
> > not convince him.
> >
> >
> >
> > Please give me references to the books/documents/blogposts. which clearly
> > shows that Kafka not intended for this way to use (references to other
> > anti-patterns/pitfalls will be useful).
> >
> > My own attempts to find such references were unsuccessful.
> >
> >
> >
> > Thank you!
> >
> >
> >
> >
>
Re: Need help to find references to antipatterns/pitfalls/incorrect
ways to use Kafka
Posted by Steve Howard <st...@confluent.io>.
Do you have location specific SKU pricing?
On Sun, Mar 31, 2019, 7:25 PM <akuterin@gmail.com wrote:
> Hello!
>
>
>
> I ask for your help in connection with the my recent task:
>
> - Price lists are delivered to 20,000 points of sale with a frequency of
> <10
> price lists per day.
>
> - The order in which the price lists follow is important. It is also
> important that the price lists are delivered to the point of sale online.
>
> - At each point of sale, an agent application is deployed, which processes
> the received price lists.
>
>
>
> This task is not particularly difficult. Help in solving the task is not
> required.
>
>
>
> The difficulty is that Kafka in our company is a new "silver bullet", and
> the project manager requires me to implement the following technical
> decision:
>
> deploy 20,000 Kafka consumer instances (one instance for each point of
> sale)
> for one topic partitioned into 20,000 partitions - one partition per
> consumer.
>
> Technical problems obtained in experiments with this technical decision do
> not convince him.
>
>
>
> Please give me references to the books/documents/blogposts. which clearly
> shows that Kafka not intended for this way to use (references to other
> anti-patterns/pitfalls will be useful).
>
> My own attempts to find such references were unsuccessful.
>
>
>
> Thank you!
>
>
>
>