You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Chaitanya Chebolu <ch...@datatorrent.com> on 2016/07/18 08:54:16 UTC
Dynamic partition is not working in Kafka Input Operator
Hi All,
I am facing dynamic partition issues in 0.8 version of Kafka Input
Operator. My application has the following DAG:
KafkaSinglePortStringInputOperator(Input) ->
ConsoleOutputOperator(Output)
I launched the application with below configuration:
Kafka topic created with single partition and replication factor as 1.
Partition Strategy: ONE_TO_ONE
Launched the application successfully. After some time, I increased the
topic partitions to 2. After re-partition, the window of down stream
operator is not moving. By looking into the app Physical DAG, it looks like
there is an issue in construction of Physical DAG after re-partition.
Please let me know if any one observed the same behavior. Do we have JIRA
for tracking this issue.
I am attaching some of the screenshots of this application.
Regards,
Chaitanya
Re: Dynamic partition is not working in Kafka Input Operator
Posted by Chaitanya Chebolu <ch...@datatorrent.com>.
Hi Sandesh,
No.
Kafka Input Operator(0.8 version) supports dynamic partition based on
Kafka partitions.
Created a JIRA (APEXCORE-494
<https://issues.apache.org/jira/browse/APEXCORE-494>) for tracking this
issue.
Regards,
Chaitanya
On Tue, Jul 19, 2016 at 9:47 AM, Sandesh Hegde <sa...@datatorrent.com>
wrote:
> Was this resolved?
>
> My understanding is that, Kafka Input operator doesn't support the changes
> in Kafka partitions after the initial launch.
>
> On Mon, Jul 18, 2016 at 1:54 AM Chaitanya Chebolu <
> chaitanya@datatorrent.com>
> wrote:
>
> > Hi All,
> >
> > I am facing dynamic partition issues in 0.8 version of Kafka Input
> > Operator. My application has the following DAG:
> >
> > KafkaSinglePortStringInputOperator(Input) ->
> > ConsoleOutputOperator(Output)
> >
> > I launched the application with below configuration:
> > Kafka topic created with single partition and replication factor as 1.
> > Partition Strategy: ONE_TO_ONE
> >
> > Launched the application successfully. After some time, I increased
> the
> > topic partitions to 2. After re-partition, the window of down stream
> > operator is not moving. By looking into the app Physical DAG, it looks
> like
> > there is an issue in construction of Physical DAG after re-partition.
> >
> > Please let me know if any one observed the same behavior. Do we have JIRA
> > for tracking this issue.
> > I am attaching some of the screenshots of this application.
> >
> > Regards,
> > Chaitanya
> >
> >
>
Re: Dynamic partition is not working in Kafka Input Operator
Posted by Sandesh Hegde <sa...@datatorrent.com>.
Was this resolved?
My understanding is that, Kafka Input operator doesn't support the changes
in Kafka partitions after the initial launch.
On Mon, Jul 18, 2016 at 1:54 AM Chaitanya Chebolu <ch...@datatorrent.com>
wrote:
> Hi All,
>
> I am facing dynamic partition issues in 0.8 version of Kafka Input
> Operator. My application has the following DAG:
>
> KafkaSinglePortStringInputOperator(Input) ->
> ConsoleOutputOperator(Output)
>
> I launched the application with below configuration:
> Kafka topic created with single partition and replication factor as 1.
> Partition Strategy: ONE_TO_ONE
>
> Launched the application successfully. After some time, I increased the
> topic partitions to 2. After re-partition, the window of down stream
> operator is not moving. By looking into the app Physical DAG, it looks like
> there is an issue in construction of Physical DAG after re-partition.
>
> Please let me know if any one observed the same behavior. Do we have JIRA
> for tracking this issue.
> I am attaching some of the screenshots of this application.
>
> Regards,
> Chaitanya
>
>