You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by sgg <sg...@gmail.com> on 2013/12/11 17:47:26 UTC
changing the number of containers in a Samza Job
A part of the documentation (http://samza.incubator.apache.org/learn/documentation/0.7.0/container/task-runner.html) suggests it ?might? be possible to change the number of containers allocated to a job:
"Your Samza job might be unable to keep up with only one container, so you ask for a second YARN container to put some of the StreamTask partitions"
How does one ask for a second YARN container? Is it a matter of stoping the job, changing the configuration and starting a new job? or is there some way to do this without starting a job?
sgg
Re: changing the number of containers in a Samza Job
Posted by sgg <sg...@gmail.com>.
ok thanks for the clarification Chris. I had hoped it could be done on the fly, but I wasn't counting on it :-)
On Dec 11, 2013, at 12:32 PM, Chris Riccomini <cr...@linkedin.com> wrote:
> Hey sgg,
>
> Yes, you can change the container count to any number where 1 <= container
> count <= max input partitions. The configuration to make this change is:
>
> yarn.container.count=1
>
> All config in Samza is resolved at job start time, and remains immutable
> for the remainder of the job execution, so you can't dynamically change
> the container count. You need to restart the Samza job to update the
> configuration.
>
> Cheers,
> Chris
>
> On 12/11/13 8:47 AM, "sgg" <sg...@gmail.com> wrote:
>
>> A part of the documentation
>> (http://samza.incubator.apache.org/learn/documentation/0.7.0/container/tas
>> k-runner.html) suggests it ?might? be possible to change the number of
>> containers allocated to a job:
>> "Your Samza job might be unable to keep up with only one container, so
>> you ask for a second YARN container to put some of the StreamTask
>> partitions"
>>
>> How does one ask for a second YARN container? Is it a matter of stoping
>> the job, changing the configuration and starting a new job? or is there
>> some way to do this without starting a job?
>>
>> sgg
>
Re: changing the number of containers in a Samza Job
Posted by Chris Riccomini <cr...@linkedin.com>.
Hey sgg,
Yes, you can change the container count to any number where 1 <= container
count <= max input partitions. The configuration to make this change is:
yarn.container.count=1
All config in Samza is resolved at job start time, and remains immutable
for the remainder of the job execution, so you can't dynamically change
the container count. You need to restart the Samza job to update the
configuration.
Cheers,
Chris
On 12/11/13 8:47 AM, "sgg" <sg...@gmail.com> wrote:
>A part of the documentation
>(http://samza.incubator.apache.org/learn/documentation/0.7.0/container/tas
>k-runner.html) suggests it ?might? be possible to change the number of
>containers allocated to a job:
> "Your Samza job might be unable to keep up with only one container, so
>you ask for a second YARN container to put some of the StreamTask
>partitions"
>
>How does one ask for a second YARN container? Is it a matter of stoping
>the job, changing the configuration and starting a new job? or is there
>some way to do this without starting a job?
>
>sgg