You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/11/08 00:13:00 UTC
[jira] [Commented] (SAMZA-1482) Restart or fail Samza jobs in YARN
when input topic partition changes
[ https://issues.apache.org/jira/browse/SAMZA-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16243128#comment-16243128 ]
ASF GitHub Bot commented on SAMZA-1482:
---------------------------------------
GitHub user nickpan47 opened a pull request:
https://github.com/apache/samza/pull/351
SAMZA-1482: Restart or fail Samza jobs in YARN when detecting changes…
… in input topic partitions
Some high-lights of the changes:
# always instantiating StreamPartitionCountMonitor on all input system streams now
## it is debatable whether we want to include systems that do not implement the optimized ExtendedSystemAdmin interface. We may need to configure a long partition monitor interval for this case and the case where there are tons of input topics. (Pending perf test)
# moved the instantiation of StreamPartitionCountMonitor out of JobModelManager and allow ClusterBasedJobCoordinator associate a callback method directly to the monitor
# allow callbacks to set different application status code before throwing exception to shutdown the job
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickpan47/samza restart-on-partition-change
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/samza/pull/351.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #351
----
commit 30aaa66d8ac9f640dcb6fb117a4e4c2d2e775ecc
Author: Yi Pan (Data Infrastructure) <ni...@gmail.com>
Date: 2017-09-25T17:13:46Z
SAMZA-1482: Restart or fail Samza jobs in YARN when detecting changes in input topic partitions
----
> Restart or fail Samza jobs in YARN when input topic partition changes
> ---------------------------------------------------------------------
>
> Key: SAMZA-1482
> URL: https://issues.apache.org/jira/browse/SAMZA-1482
> Project: Samza
> Issue Type: Bug
> Reporter: Yi Pan (Data Infrastructure)
> Assignee: Yi Pan (Data Infrastructure)
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Currently, after a Samza job is started, it works only on a set of fixed input topic partitions at the start-up time. When input topic partitions are expanded, we often lose the messages sent in the new partitions, until we restart the job.
> SAMZA-882 added a input stream partition count monitor inside the JobCoordinator. This ticket is targeted to use this monitor metrics and trigger the following actions in YARN:
> # for stateless jobs, shutdown the JobCoordinator w/ UNDEFINED status code s.t. YARN will restart the whole job
> # for stateful jobs, shutdown the JobCoordinator w/ FAILED status code s.t. YARN will stop the whole job
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)