You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Brian Krahmer <bk...@krahmer.com> on 2017/01/05 10:58:54 UTC
Problem with processor API partition assignments
Hey guys,
I'm fighting an issue where I can currently only run one instance of
my streams application because when other instances come up, the
partition reassignment (looks to me) to be incorrect.
I'm testing with docker-compose at the moment. When I scale my
application to 3 instances and the 2nd and 3rd connect to kafka, causing
a rebalance, I get the following assignment on one of my instances:
FleetData-0
FleetData-1
VehicleJourneyMapData-0
VehicleJourneyMapData-1
JourneyStarted-0
VehicleStateChanged-1
VehicleIgnitionData-0
VinVehicleMapData-1
As you can see, the assignments are clearly not symmetric, which causes
problems, as I'm essentially doing join operations. All topics have 3
partition in this testing scenario. I'm using version 0.10.1.0. Any ideas?
thanks,
brian
Re: Re: Problem with processor API partition assignments
Posted by Damian Guy <da...@gmail.com>.
Hi Brian,
That is correct.
Thanks,
Damian
On Mon, 9 Jan 2017 at 14:19 Brian Krahmer <bk...@krahmer.com> wrote:
> Hi Damian,
> I started to pull together enough code to show my topology, and have
> a feeling I see what the problem is. Is it correct that when
> configuring a processor that it uses the sources applied to that
> processor to ensure the partitions are aligned for that task?
>
> thanks,
> brian
>
> On 05.01.2017 13:37, Damian Guy wrote:
> > Hi Brian,
> >
> > It might be helpful if you provide some code showing your Topology.
> >
> > Thanks,
> > Damian
> >
> > On Thu, 5 Jan 2017 at 10:59 Brian Krahmer <bk...@krahmer.com> wrote:
> >
> >> Hey guys,
> >>
> >> I'm fighting an issue where I can currently only run one instance of
> >> my streams application because when other instances come up, the
> >> partition reassignment (looks to me) to be incorrect.
> >>
> >> I'm testing with docker-compose at the moment. When I scale my
> >> application to 3 instances and the 2nd and 3rd connect to kafka, causing
> >> a rebalance, I get the following assignment on one of my instances:
> >>
> >> FleetData-0
> >> FleetData-1
> >> VehicleJourneyMapData-0
> >> VehicleJourneyMapData-1
> >> JourneyStarted-0
> >> VehicleStateChanged-1
> >> VehicleIgnitionData-0
> >> VinVehicleMapData-1
> >>
> >> As you can see, the assignments are clearly not symmetric, which causes
> >> problems, as I'm essentially doing join operations. All topics have 3
> >> partition in this testing scenario. I'm using version 0.10.1.0. Any
> >> ideas?
> >>
> >> thanks,
> >> brian
> >>
> >>
>
>
Re: Re: Problem with processor API partition assignments
Posted by Brian Krahmer <bk...@krahmer.com>.
Hi Damian,
I started to pull together enough code to show my topology, and have
a feeling I see what the problem is. Is it correct that when
configuring a processor that it uses the sources applied to that
processor to ensure the partitions are aligned for that task?
thanks,
brian
On 05.01.2017 13:37, Damian Guy wrote:
> Hi Brian,
>
> It might be helpful if you provide some code showing your Topology.
>
> Thanks,
> Damian
>
> On Thu, 5 Jan 2017 at 10:59 Brian Krahmer <bk...@krahmer.com> wrote:
>
>> Hey guys,
>>
>> I'm fighting an issue where I can currently only run one instance of
>> my streams application because when other instances come up, the
>> partition reassignment (looks to me) to be incorrect.
>>
>> I'm testing with docker-compose at the moment. When I scale my
>> application to 3 instances and the 2nd and 3rd connect to kafka, causing
>> a rebalance, I get the following assignment on one of my instances:
>>
>> FleetData-0
>> FleetData-1
>> VehicleJourneyMapData-0
>> VehicleJourneyMapData-1
>> JourneyStarted-0
>> VehicleStateChanged-1
>> VehicleIgnitionData-0
>> VinVehicleMapData-1
>>
>> As you can see, the assignments are clearly not symmetric, which causes
>> problems, as I'm essentially doing join operations. All topics have 3
>> partition in this testing scenario. I'm using version 0.10.1.0. Any
>> ideas?
>>
>> thanks,
>> brian
>>
>>
Re: Problem with processor API partition assignments
Posted by "Matthias J. Sax" <ma...@confluent.io>.
It would also be helpful to know the number of partitions for each topic.
-Matthias
On 1/5/17 4:37 AM, Damian Guy wrote:
> Hi Brian,
>
> It might be helpful if you provide some code showing your Topology.
>
> Thanks,
> Damian
>
> On Thu, 5 Jan 2017 at 10:59 Brian Krahmer <bk...@krahmer.com> wrote:
>
>> Hey guys,
>>
>> I'm fighting an issue where I can currently only run one instance of
>> my streams application because when other instances come up, the
>> partition reassignment (looks to me) to be incorrect.
>>
>> I'm testing with docker-compose at the moment. When I scale my
>> application to 3 instances and the 2nd and 3rd connect to kafka, causing
>> a rebalance, I get the following assignment on one of my instances:
>>
>> FleetData-0
>> FleetData-1
>> VehicleJourneyMapData-0
>> VehicleJourneyMapData-1
>> JourneyStarted-0
>> VehicleStateChanged-1
>> VehicleIgnitionData-0
>> VinVehicleMapData-1
>>
>> As you can see, the assignments are clearly not symmetric, which causes
>> problems, as I'm essentially doing join operations. All topics have 3
>> partition in this testing scenario. I'm using version 0.10.1.0. Any
>> ideas?
>>
>> thanks,
>> brian
>>
>>
>
Re: Problem with processor API partition assignments
Posted by Damian Guy <da...@gmail.com>.
Hi Brian,
It might be helpful if you provide some code showing your Topology.
Thanks,
Damian
On Thu, 5 Jan 2017 at 10:59 Brian Krahmer <bk...@krahmer.com> wrote:
> Hey guys,
>
> I'm fighting an issue where I can currently only run one instance of
> my streams application because when other instances come up, the
> partition reassignment (looks to me) to be incorrect.
>
> I'm testing with docker-compose at the moment. When I scale my
> application to 3 instances and the 2nd and 3rd connect to kafka, causing
> a rebalance, I get the following assignment on one of my instances:
>
> FleetData-0
> FleetData-1
> VehicleJourneyMapData-0
> VehicleJourneyMapData-1
> JourneyStarted-0
> VehicleStateChanged-1
> VehicleIgnitionData-0
> VinVehicleMapData-1
>
> As you can see, the assignments are clearly not symmetric, which causes
> problems, as I'm essentially doing join operations. All topics have 3
> partition in this testing scenario. I'm using version 0.10.1.0. Any
> ideas?
>
> thanks,
> brian
>
>