You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Avi Flax <av...@parkassist.com> on 2016/12/12 15:02:34 UTC
[Streams] Threading Frustration
Hi all,
I’m running Kafka 0.10.0.1 on Java 8 on Linux — same for brokers and streams nodes.
I’m attempting to scale up a Streams app that does a lot of I/O — in particular, I’m hoping to isolate each partition into its own thread — and I’m confused:
* I recently created 11 new topics and partitions and modified this app to consume from the 11 new topics
* The app is now consuming from 130 partitions across those 11 topics
* I’ve set the config value num.stream.threads to 130
* In the logs from the startup routine, I can see that Streams is creating 130 threads
* When I use kafka-consumer-groups to see the status of the app, in particular lag, I can see that it’s only using 20 of those threads to consume from its topics
* This means, of course, that when processing for one partition blocks, it blocks processing for any other partition assigned to the same thread.
So:
* Why is this happening?
* How can I achieve my goal?
Thank you,
Avi
————
Software Architect @ Park Assist
We’re hiring! http://tech.parkassist.com/jobs/
Re: [Streams] Threading Frustration
Posted by Avi Flax <av...@parkassist.com>.
> On Dec 12, 2016, at 11:42, Damian Guy <da...@gmail.com> wrote:
>
> If you want to split these out so that they can run in parallel, then you
> will need to create a new stream for each topic.
Super, super helpful — thanks so much!
Avi
Re: [Streams] Threading Frustration
Posted by Avi Flax <av...@parkassist.com>.
> On Dec 12, 2016, at 12:29, Damian Guy <da...@gmail.com> wrote:
>
> Yep - that looks correct
Fantastic! Thanks again!
————
Software Architect @ Park Assist
We’re hiring! http://tech.parkassist.com/jobs/
Re: [Streams] Threading Frustration
Posted by Damian Guy <da...@gmail.com>.
It is the same. The ruby code in this example is just calling into the Java
API.
On Mon, 12 Dec 2016 at 17:40 Ali Akhtar <al...@gmail.com> wrote:
> @Damian,
>
> In the Java equivalent of this, does each KStream / KStreamBuilder.stream()
> invocation create its own topic group, i.e its own thread?
>
> On Mon, Dec 12, 2016 at 10:29 PM, Damian Guy <da...@gmail.com> wrote:
>
> > Yep - that looks correct
> >
> > On Mon, 12 Dec 2016 at 17:18 Avi Flax <av...@parkassist.com> wrote:
> >
> > >
> > > > On Dec 12, 2016, at 11:42, Damian Guy <da...@gmail.com> wrote:
> > > >
> > > > If you want to split these out so that they can run in parallel, then
> > you
> > > > will need to create a new stream for each topic.
> > >
> > >
> > > Just to make sure I’m understanding this, does this code change look
> > right?
> > >
> > > From this:
> > >
> > > ```ruby
> > > topic_names = SDP::Config.topic_names(config).to_java(:string)
> > > bay_events = builder.stream key_serde, val_serde, topic_names
> > > # etc
> > > ```
> > >
> > >
> > > To this:
> > >
> > > ```ruby
> > > SDP::Config.topic_names(config).each do |topic_name|
> > > bay_events = builder.stream key_serde, val_serde, topic_name
> > > # etc
> > > end
> > > ```
> > >
> > >
> > > I hope so!
> > >
> > > Thanks!
> > > Avi
> >
>
Re: [Streams] Threading Frustration
Posted by Ali Akhtar <al...@gmail.com>.
@Damian,
In the Java equivalent of this, does each KStream / KStreamBuilder.stream()
invocation create its own topic group, i.e its own thread?
On Mon, Dec 12, 2016 at 10:29 PM, Damian Guy <da...@gmail.com> wrote:
> Yep - that looks correct
>
> On Mon, 12 Dec 2016 at 17:18 Avi Flax <av...@parkassist.com> wrote:
>
> >
> > > On Dec 12, 2016, at 11:42, Damian Guy <da...@gmail.com> wrote:
> > >
> > > If you want to split these out so that they can run in parallel, then
> you
> > > will need to create a new stream for each topic.
> >
> >
> > Just to make sure I’m understanding this, does this code change look
> right?
> >
> > From this:
> >
> > ```ruby
> > topic_names = SDP::Config.topic_names(config).to_java(:string)
> > bay_events = builder.stream key_serde, val_serde, topic_names
> > # etc
> > ```
> >
> >
> > To this:
> >
> > ```ruby
> > SDP::Config.topic_names(config).each do |topic_name|
> > bay_events = builder.stream key_serde, val_serde, topic_name
> > # etc
> > end
> > ```
> >
> >
> > I hope so!
> >
> > Thanks!
> > Avi
>
Re: [Streams] Threading Frustration
Posted by Damian Guy <da...@gmail.com>.
Yep - that looks correct
On Mon, 12 Dec 2016 at 17:18 Avi Flax <av...@parkassist.com> wrote:
>
> > On Dec 12, 2016, at 11:42, Damian Guy <da...@gmail.com> wrote:
> >
> > If you want to split these out so that they can run in parallel, then you
> > will need to create a new stream for each topic.
>
>
> Just to make sure I’m understanding this, does this code change look right?
>
> From this:
>
> ```ruby
> topic_names = SDP::Config.topic_names(config).to_java(:string)
> bay_events = builder.stream key_serde, val_serde, topic_names
> # etc
> ```
>
>
> To this:
>
> ```ruby
> SDP::Config.topic_names(config).each do |topic_name|
> bay_events = builder.stream key_serde, val_serde, topic_name
> # etc
> end
> ```
>
>
> I hope so!
>
> Thanks!
> Avi
Re: [Streams] Threading Frustration
Posted by Avi Flax <av...@parkassist.com>.
> On Dec 12, 2016, at 11:42, Damian Guy <da...@gmail.com> wrote:
>
> If you want to split these out so that they can run in parallel, then you
> will need to create a new stream for each topic.
Just to make sure I’m understanding this, does this code change look right?
From this:
```ruby
topic_names = SDP::Config.topic_names(config).to_java(:string)
bay_events = builder.stream key_serde, val_serde, topic_names
# etc
```
To this:
```ruby
SDP::Config.topic_names(config).each do |topic_name|
bay_events = builder.stream key_serde, val_serde, topic_name
# etc
end
```
I hope so!
Thanks!
Avi
Re: [Streams] Threading Frustration
Posted by Damian Guy <da...@gmail.com>.
Hi Avi,
Thanks for sharing your code. I believe the reason you are only seeing 20
threads used is due to a couple of things:
KafkaStreams uses a concept of topic groups to group related topics into
StreamTasks. A StreamTask is a task an independent part of the topology
that can be executed on its own - they define the degree of parallelism you
can achieve in your application. As you can see here:
https://gist.github.com/aviflax/f7332af810f8281549652bc4f9a5e007#file-log_fragment-log-L366
there
are only 20 active tasks that are created. Now when you create your
KafkaStreams app I can see that you are using the overladed method that
takes an array of topic names to build the streams. This will mean that
there is only a single topic group for your application. In this case the
maximum number of StreamTasks, and hence the maximum degree of parallelism
you can achieve, is defined by the maximum partition count of the input
topics. So, i'm guessing the max partitions you have for an input topic is
20.
If you want to split these out so that they can run in parallel, then you
will need to create a new stream for each topic.
HTH,
Damian
On Mon, 12 Dec 2016 at 16:03 Avi Flax <av...@parkassist.com> wrote:
>
> > On Dec 12, 2016, at 10:24, Damian Guy <da...@gmail.com> wrote:
> >
> > The code for your Streams Application. Doesn't have to be the actual
> code,
> > but an example of how you are using Kafka Streams.
>
> OK, I’ve prepared a gist with (I hope) the relevant code, and also some
> log records just in case they might help:
>
> https://gist.github.com/aviflax/f7332af810f8281549652bc4f9a5e007
>
> Thanks for the help!
Re: [Streams] Threading Frustration
Posted by Avi Flax <av...@parkassist.com>.
> On Dec 12, 2016, at 10:24, Damian Guy <da...@gmail.com> wrote:
>
> The code for your Streams Application. Doesn't have to be the actual code,
> but an example of how you are using Kafka Streams.
OK, I’ve prepared a gist with (I hope) the relevant code, and also some log records just in case they might help:
https://gist.github.com/aviflax/f7332af810f8281549652bc4f9a5e007
Thanks for the help!
Re: [Streams] Threading Frustration
Posted by Damian Guy <da...@gmail.com>.
Hi Avi,
The code for your Streams Application. Doesn't have to be the actual code,
but an example of how you are using Kafka Streams.
On Mon, 12 Dec 2016 at 15:13 Avi Flax <av...@parkassist.com> wrote:
>
> > On Dec 12, 2016, at 10:08, Damian Guy <da...@gmail.com> wrote:
> >
> > Can you provide an example of your topology?
>
> I’d be happy to, but I’m not sure what you mean? Are you interested in the
> implementation at the code level? You want a better understanding of the
> work being done? The infrastructure?
Re: [Streams] Threading Frustration
Posted by Avi Flax <av...@parkassist.com>.
> On Dec 12, 2016, at 10:08, Damian Guy <da...@gmail.com> wrote:
>
> Can you provide an example of your topology?
I’d be happy to, but I’m not sure what you mean? Are you interested in the implementation at the code level? You want a better understanding of the work being done? The infrastructure?
Re: [Streams] Threading Frustration
Posted by Damian Guy <da...@gmail.com>.
Hi Avi,
Can you provide an example of your topology?
Thanks,
Damian
On Mon, 12 Dec 2016 at 15:02 Avi Flax <av...@parkassist.com> wrote:
Hi all,
I’m running Kafka 0.10.0.1 on Java 8 on Linux — same for brokers and
streams nodes.
I’m attempting to scale up a Streams app that does a lot of I/O — in
particular, I’m hoping to isolate each partition into its own thread — and
I’m confused:
* I recently created 11 new topics and partitions and modified this app to
consume from the 11 new topics
* The app is now consuming from 130 partitions across those 11 topics
* I’ve set the config value num.stream.threads to 130
* In the logs from the startup routine, I can see that Streams is creating
130 threads
* When I use kafka-consumer-groups to see the status of the app, in
particular lag, I can see that it’s only using 20 of those threads to
consume from its topics
* This means, of course, that when processing for one partition blocks, it
blocks processing for any other partition assigned to the same thread.
So:
* Why is this happening?
* How can I achieve my goal?
Thank you,
Avi
————
Software Architect @ Park Assist
We’re hiring! http://tech.parkassist.com/jobs/