You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "Navina Ramesh (JIRA)" <ji...@apache.org> on 2015/12/18 21:28:46 UTC
[jira] [Created] (SAMZA-843) Slow start of Samza jobs with large
number of containers
Navina Ramesh created SAMZA-843:
-----------------------------------
Summary: Slow start of Samza jobs with large number of containers
Key: SAMZA-843
URL: https://issues.apache.org/jira/browse/SAMZA-843
Project: Samza
Issue Type: Improvement
Affects Versions: 0.10.0
Reporter: Navina Ramesh
Fix For: 0.10.1
We have noticed that when a job has large number of containers and is deployed in Yarn, all the containers query the coordinator URL at the same time, causing an almost herd-like effect. It takes a long time for the job to reach a steady state, where all containers start processing messages and none of them are seeing Socket Timeout exception from the Job Coordinator. This effect is amplified further, if the AM machine is already heavily loaded.
We could fix this in many ways.
1. We could have containers wait for random time period before querying the Job Coordinator.
2. We could add a cache in the JobServlet so that the JobModel is not refreshed with each request.
3. We could make the JobModel as an atomic reference that gets updated only when the AM requires to restart a failed container. It is ok for the containers to get slightly stale JobModel as long as the partition assignment doesn't change.
While the above options are good ways to solve this problem, it does bring up the question about why the containers should query the coordinator for the JobModel (which creates a SPOF for the retrieving JobModel) when it can be inferred by consuming from the Coordinator Stream directly.
We should consider an architecture where each container has an embedded job coordinator module that only reads partition assignment messages. The embedded job coordinator can act like a "slave" JC to the job coordinator running in the AM. This will be a major architecture change that requires more thought. Just wanted to put down the idea here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)