how to decide no of operator partitions

[rishi <ri...@target.com>]

Hi,

I am running one of the apex application which contains multiple operators.
For some operators I assigned >1 partitions as it has to do some heavy
lifting compared to other operators. Like kafka operator I have given 2
partitions.

Now thing is I restarted the running application which got killed after 2-3
days and after that I see there is huge latency in some of the
operators.Even after 2-3 days latency no is huge and application is not able
to process today's record.

So please suggest me how to handle this kind of scenario. when we restart
application and if we see huge latency no.

Thanks
Rishi






--
View this message in context: http://apache-apex-users-list.78494.x6.nabble.com/how-to-decide-no-of-operator-partitions-tp1580.html
Sent from the Apache Apex Users list mailing list archive at Nabble.com.

Re: how to decide no of operator partitions

[rishi <ri...@target.com>]

Hi Amol,

Thanks for the reply.

I restarted my apex application after 3 days of failure of old app.Till now
I have not performed benchmarking of my operators/app. Still problem is
persisting after 5days from restart. 

I manually killed application and wanted to give new configuration to old
app by increasing the partitions for all the operators, but new
configuration it is not taking.

Looks like I have to manually kill the application and start a new
application which will start writing data from today. Remaining difference
of data I will have to manually load.

Please suggest if there is any option in apex to get rid of latency so that
I can have updated data writing on HDFS ,instead of manually loading the
data.

Thanks
Rishi



--
View this message in context: http://apache-apex-users-list.78494.x6.nabble.com/how-to-decide-no-of-operator-partitions-tp1580p1583.html
Sent from the Apache Apex Users list mailing list archive at Nabble.com.

Re: how to decide no of operator partitions

[Amol Kekre <am...@datatorrent.com>]

Rishi,
How much time elapses between old app and new app? It may an issue of
resource needs to catch up. Have  you benchmarked your operators/app?

Thks
Amol



E:amol@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*

www.datatorrent.com


On Sun, May 7, 2017 at 11:02 AM, rishi <ri...@target.com> wrote:

> Hi,
>
> I am running one of the apex application which contains multiple operators.
> For some operators I assigned >1 partitions as it has to do some heavy
> lifting compared to other operators. Like kafka operator I have given 2
> partitions.
>
> Now thing is I restarted the running application which got killed after 2-3
> days and after that I see there is huge latency in some of the
> operators.Even after 2-3 days latency no is huge and application is not
> able
> to process today's record.
>
> So please suggest me how to handle this kind of scenario. when we restart
> application and if we see huge latency no.
>
> Thanks
> Rishi
>
>
>
>
>
>
> --
> View this message in context: http://apache-apex-users-list.
> 78494.x6.nabble.com/how-to-decide-no-of-operator-partitions-tp1580.html
> Sent from the Apache Apex Users list mailing list archive at Nabble.com.
>