You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Zach Cox <zc...@gmail.com> on 2014/10/06 14:27:28 UTC
Problems running new jobs in hello-samza
Hi - I'm just getting started with Samza. I got the hello-samza example
working properly in the vagrant box. Then I wrote 2 new tasks, rebuilt
everything and submitted them to yarn using run-job.sh. These 2 new jobs
show up in the yarn web ui, however only one of them has State=RUNNING, the
other just sits forever at State=ACCEPTED.
The Cluster Metrics section shows some interesting things:
- Apps Pending = 1
- Apps Running = 4
- Containers Running = 8
- Memory Used = 8 GB
- Memory Total = 8 GB
- Memory Reserved = 0 B
Again I'm really new to samza & yarn, but does this mean that the node on
this vagrant box has 8 GB memory available but all 8 GB is being used, so
it can't run the 5th samza job?
Are there 8 containers running because each Samza job has an
ApplicationMaster and a SamzaContainer? Are each of those containers using
1 GB memory, and that's why all the available memory is used up? Do these
containers really need 1 GB memory each? Can this be adjusted somehow?
Just trying to better understand what's going on here, and see if there's a
simple way to get both of my new tasks running in hello-samza.
Thanks,
Zach
Re: Problems running new jobs in hello-samza
Posted by Zach Cox <zc...@gmail.com>.
Thanks Lukas and Mark - setting yarn.nodemanager.resource.cpu-vcores to 10
allowed the 5th job to run, and it's working beautifully now!
On Mon, Oct 6, 2014 at 1:53 PM, Mark Mindenhall <
mark.mindenhall@machineshop.io> wrote:
> Yes, looks like you need to increase the number of vCores to at least 10
> in order to run 5 jobs (yarn-site.xml):
>
> <property>
> <name>yarn.nodemanager.resource.cpu-vcores</name>
> <value>10</value>
> <description>Number of CPU cores that can be allocated for
> containers.</description>
> </property>
>
>
> On Oct 6, 2014, at 12:27 PM, Zach Cox <zc...@gmail.com> wrote:
>
> > Thanks for the replies everyone! I did the 3 things that Mark mentioned,
> > re-built & deployed the .tar.gz, then did `bin/grid stop all` and
> `bin/grid
> > start all`. But when I re-submitted the 3 hello-samza jobs along with my
> 2
> > new jobs, yarn still won't run my 5th job. The yarn web ui now shows
> Memory
> > Used = 4 GB and Memory Total = 8 GB, but my job still sits at State =
> > ACCEPTED.
> >
> > When I tail deploy/yarn/logs/yarn-vagrant-resourcemanager-precise64.log I
> > see this repeated continuously:
> >
> https://gist.githubusercontent.com/zcox/0f2b260d29e18d40d038/raw/0d805bcd7d8fec5332756efc9c990679480df117/gistfile1.txt
> >
> > I notice it says "available=<memory:4096, vCores:0>" - is my job not
> being
> > run now because vCores=0?
> >
> > I also updated Vagrantfile to use:
> >
> > samza.vm.provider :virtualbox do |vb| vb.memory = 4096 vb.cpus = 8 end
> >
> > Thanks,
> > Zach
> >
> >
> > On Mon, Oct 6, 2014 at 12:40 PM, Lukas Steiblys <lu...@doubledutch.me>
> > wrote:
> >
> >> I'll add that if you check the YARN node application master container
> log
> >> and see that the job is constantly restarting, you might need to
> increase
> >> the container memory limit to 1024MB at least. Also, a good parameter to
> >> play with in YARN is yarn.nodemanager.vmem-pmem-ratio.
> >>
> >> Lukas
> >>
> >> -----Original Message----- From: Mark Mindenhall
> >> Sent: Monday, October 6, 2014 8:44 AM
> >> To: dev@samza.incubator.apache.org
> >> Subject: Re: Problems running new jobs in hello-samza
> >>
> >>
> >> Hi Zach,
> >>
> >> I’m also a relative newbie, but I did run into this same issue. You are
> >> correct, in that your 5th job isn’t starting due to not enough resources
> >> available in the cluster, so you need to reduce the resources required.
> >>
> >> First, in yarn-site.xml I switched over to the FairScheduler<
> >> http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/
> >> FairScheduler.html>:
> >>
> >> <property>
> >> <name>yarn.resourcemanager.scheduler.class</name>
> >> <value>org.apache.hadoop.yarn.server.resourcemanager.
> >> scheduler.fair.FairScheduler</value>
> >> </property>
> >>
> >> I also added these two properties (yarn-site.xml) to control the amount
> of
> >> memory allocated to each job:
> >>
> >> <property>
> >> <name>yarn.scheduler.minimum-allocation-mb</name>
> >> <value>256</value>
> >> <description>Minimum limit of memory to allocate to each container
> >> request at the Resource Manager.</description>
> >> </property>
> >> <property>
> >> <name>yarn.scheduler.maximum-allocation-mb</name>
> >> <value>512</value>
> >> <description>Maximum limit of memory to allocate to each container
> >> request at the Resource Manager.</description>
> >> </property>
> >>
> >> Then, in each of my Samza properties files describing my jobs, I added
> the
> >> following two settings:
> >>
> >> yarn.container.memory.mb=512
> >> yarn.am.container.memory.mb=256
> >>
> >> Hope that helps!
> >>
> >> Best,
> >> Mark
> >>
> >>
> >> On Oct 6, 2014, at 6:27 AM, Zach Cox <zcox522@gmail.com<mailto:zcox
> >> 522@gmail.com>> wrote:
> >>
> >> Hi - I'm just getting started with Samza. I got the hello-samza example
> >> working properly in the vagrant box. Then I wrote 2 new tasks, rebuilt
> >> everything and submitted them to yarn using run-job.sh. These 2 new jobs
> >> show up in the yarn web ui, however only one of them has State=RUNNING,
> the
> >> other just sits forever at State=ACCEPTED.
> >>
> >> The Cluster Metrics section shows some interesting things:
> >> - Apps Pending = 1
> >> - Apps Running = 4
> >> - Containers Running = 8
> >> - Memory Used = 8 GB
> >> - Memory Total = 8 GB
> >> - Memory Reserved = 0 B
> >>
> >> Again I'm really new to samza & yarn, but does this mean that the node
> on
> >> this vagrant box has 8 GB memory available but all 8 GB is being used,
> so
> >> it can't run the 5th samza job?
> >>
> >> Are there 8 containers running because each Samza job has an
> >> ApplicationMaster and a SamzaContainer? Are each of those containers
> using
> >> 1 GB memory, and that's why all the available memory is used up? Do
> these
> >> containers really need 1 GB memory each? Can this be adjusted somehow?
> >>
> >> Just trying to better understand what's going on here, and see if
> there's a
> >> simple way to get both of my new tasks running in hello-samza.
> >>
> >> Thanks,
> >> Zach
> >>
> >>
>
>
Re: Problems running new jobs in hello-samza
Posted by Mark Mindenhall <ma...@machineshop.io>.
Yes, looks like you need to increase the number of vCores to at least 10 in order to run 5 jobs (yarn-site.xml):
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>10</value>
<description>Number of CPU cores that can be allocated for containers.</description>
</property>
On Oct 6, 2014, at 12:27 PM, Zach Cox <zc...@gmail.com> wrote:
> Thanks for the replies everyone! I did the 3 things that Mark mentioned,
> re-built & deployed the .tar.gz, then did `bin/grid stop all` and `bin/grid
> start all`. But when I re-submitted the 3 hello-samza jobs along with my 2
> new jobs, yarn still won't run my 5th job. The yarn web ui now shows Memory
> Used = 4 GB and Memory Total = 8 GB, but my job still sits at State =
> ACCEPTED.
>
> When I tail deploy/yarn/logs/yarn-vagrant-resourcemanager-precise64.log I
> see this repeated continuously:
> https://gist.githubusercontent.com/zcox/0f2b260d29e18d40d038/raw/0d805bcd7d8fec5332756efc9c990679480df117/gistfile1.txt
>
> I notice it says "available=<memory:4096, vCores:0>" - is my job not being
> run now because vCores=0?
>
> I also updated Vagrantfile to use:
>
> samza.vm.provider :virtualbox do |vb| vb.memory = 4096 vb.cpus = 8 end
>
> Thanks,
> Zach
>
>
> On Mon, Oct 6, 2014 at 12:40 PM, Lukas Steiblys <lu...@doubledutch.me>
> wrote:
>
>> I'll add that if you check the YARN node application master container log
>> and see that the job is constantly restarting, you might need to increase
>> the container memory limit to 1024MB at least. Also, a good parameter to
>> play with in YARN is yarn.nodemanager.vmem-pmem-ratio.
>>
>> Lukas
>>
>> -----Original Message----- From: Mark Mindenhall
>> Sent: Monday, October 6, 2014 8:44 AM
>> To: dev@samza.incubator.apache.org
>> Subject: Re: Problems running new jobs in hello-samza
>>
>>
>> Hi Zach,
>>
>> I’m also a relative newbie, but I did run into this same issue. You are
>> correct, in that your 5th job isn’t starting due to not enough resources
>> available in the cluster, so you need to reduce the resources required.
>>
>> First, in yarn-site.xml I switched over to the FairScheduler<
>> http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/
>> FairScheduler.html>:
>>
>> <property>
>> <name>yarn.resourcemanager.scheduler.class</name>
>> <value>org.apache.hadoop.yarn.server.resourcemanager.
>> scheduler.fair.FairScheduler</value>
>> </property>
>>
>> I also added these two properties (yarn-site.xml) to control the amount of
>> memory allocated to each job:
>>
>> <property>
>> <name>yarn.scheduler.minimum-allocation-mb</name>
>> <value>256</value>
>> <description>Minimum limit of memory to allocate to each container
>> request at the Resource Manager.</description>
>> </property>
>> <property>
>> <name>yarn.scheduler.maximum-allocation-mb</name>
>> <value>512</value>
>> <description>Maximum limit of memory to allocate to each container
>> request at the Resource Manager.</description>
>> </property>
>>
>> Then, in each of my Samza properties files describing my jobs, I added the
>> following two settings:
>>
>> yarn.container.memory.mb=512
>> yarn.am.container.memory.mb=256
>>
>> Hope that helps!
>>
>> Best,
>> Mark
>>
>>
>> On Oct 6, 2014, at 6:27 AM, Zach Cox <zcox522@gmail.com<mailto:zcox
>> 522@gmail.com>> wrote:
>>
>> Hi - I'm just getting started with Samza. I got the hello-samza example
>> working properly in the vagrant box. Then I wrote 2 new tasks, rebuilt
>> everything and submitted them to yarn using run-job.sh. These 2 new jobs
>> show up in the yarn web ui, however only one of them has State=RUNNING, the
>> other just sits forever at State=ACCEPTED.
>>
>> The Cluster Metrics section shows some interesting things:
>> - Apps Pending = 1
>> - Apps Running = 4
>> - Containers Running = 8
>> - Memory Used = 8 GB
>> - Memory Total = 8 GB
>> - Memory Reserved = 0 B
>>
>> Again I'm really new to samza & yarn, but does this mean that the node on
>> this vagrant box has 8 GB memory available but all 8 GB is being used, so
>> it can't run the 5th samza job?
>>
>> Are there 8 containers running because each Samza job has an
>> ApplicationMaster and a SamzaContainer? Are each of those containers using
>> 1 GB memory, and that's why all the available memory is used up? Do these
>> containers really need 1 GB memory each? Can this be adjusted somehow?
>>
>> Just trying to better understand what's going on here, and see if there's a
>> simple way to get both of my new tasks running in hello-samza.
>>
>> Thanks,
>> Zach
>>
>>
Re: Problems running new jobs in hello-samza
Posted by Lukas Steiblys <lu...@doubledutch.me>.
Yes, you need to increase the core count in yarn
(yarn.nodemanager.resource.cpu-vcores). In your case it seems like you need
at least 10 cores for 5 jobs.
Lukas
-----Original Message-----
From: Zach Cox
Sent: Monday, October 6, 2014 11:27 AM
To: dev@samza.incubator.apache.org
Subject: Re: Problems running new jobs in hello-samza
Thanks for the replies everyone! I did the 3 things that Mark mentioned,
re-built & deployed the .tar.gz, then did `bin/grid stop all` and `bin/grid
start all`. But when I re-submitted the 3 hello-samza jobs along with my 2
new jobs, yarn still won't run my 5th job. The yarn web ui now shows Memory
Used = 4 GB and Memory Total = 8 GB, but my job still sits at State =
ACCEPTED.
When I tail deploy/yarn/logs/yarn-vagrant-resourcemanager-precise64.log I
see this repeated continuously:
https://gist.githubusercontent.com/zcox/0f2b260d29e18d40d038/raw/0d805bcd7d8fec5332756efc9c990679480df117/gistfile1.txt
I notice it says "available=<memory:4096, vCores:0>" - is my job not being
run now because vCores=0?
I also updated Vagrantfile to use:
samza.vm.provider :virtualbox do |vb| vb.memory = 4096 vb.cpus = 8 end
Thanks,
Zach
On Mon, Oct 6, 2014 at 12:40 PM, Lukas Steiblys <lu...@doubledutch.me>
wrote:
> I'll add that if you check the YARN node application master container log
> and see that the job is constantly restarting, you might need to increase
> the container memory limit to 1024MB at least. Also, a good parameter to
> play with in YARN is yarn.nodemanager.vmem-pmem-ratio.
>
> Lukas
>
> -----Original Message----- From: Mark Mindenhall
> Sent: Monday, October 6, 2014 8:44 AM
> To: dev@samza.incubator.apache.org
> Subject: Re: Problems running new jobs in hello-samza
>
>
> Hi Zach,
>
> I’m also a relative newbie, but I did run into this same issue. You are
> correct, in that your 5th job isn’t starting due to not enough resources
> available in the cluster, so you need to reduce the resources required.
>
> First, in yarn-site.xml I switched over to the FairScheduler<
> http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/
> FairScheduler.html>:
>
> <property>
> <name>yarn.resourcemanager.scheduler.class</name>
> <value>org.apache.hadoop.yarn.server.resourcemanager.
> scheduler.fair.FairScheduler</value>
> </property>
>
> I also added these two properties (yarn-site.xml) to control the amount of
> memory allocated to each job:
>
> <property>
> <name>yarn.scheduler.minimum-allocation-mb</name>
> <value>256</value>
> <description>Minimum limit of memory to allocate to each container
> request at the Resource Manager.</description>
> </property>
> <property>
> <name>yarn.scheduler.maximum-allocation-mb</name>
> <value>512</value>
> <description>Maximum limit of memory to allocate to each container
> request at the Resource Manager.</description>
> </property>
>
> Then, in each of my Samza properties files describing my jobs, I added the
> following two settings:
>
> yarn.container.memory.mb=512
> yarn.am.container.memory.mb=256
>
> Hope that helps!
>
> Best,
> Mark
>
>
> On Oct 6, 2014, at 6:27 AM, Zach Cox <zcox522@gmail.com<mailto:zcox
> 522@gmail.com>> wrote:
>
> Hi - I'm just getting started with Samza. I got the hello-samza example
> working properly in the vagrant box. Then I wrote 2 new tasks, rebuilt
> everything and submitted them to yarn using run-job.sh. These 2 new jobs
> show up in the yarn web ui, however only one of them has State=RUNNING,
> the
> other just sits forever at State=ACCEPTED.
>
> The Cluster Metrics section shows some interesting things:
> - Apps Pending = 1
> - Apps Running = 4
> - Containers Running = 8
> - Memory Used = 8 GB
> - Memory Total = 8 GB
> - Memory Reserved = 0 B
>
> Again I'm really new to samza & yarn, but does this mean that the node on
> this vagrant box has 8 GB memory available but all 8 GB is being used, so
> it can't run the 5th samza job?
>
> Are there 8 containers running because each Samza job has an
> ApplicationMaster and a SamzaContainer? Are each of those containers using
> 1 GB memory, and that's why all the available memory is used up? Do these
> containers really need 1 GB memory each? Can this be adjusted somehow?
>
> Just trying to better understand what's going on here, and see if there's
> a
> simple way to get both of my new tasks running in hello-samza.
>
> Thanks,
> Zach
>
>
Re: Problems running new jobs in hello-samza
Posted by Zach Cox <zc...@gmail.com>.
Thanks for the replies everyone! I did the 3 things that Mark mentioned,
re-built & deployed the .tar.gz, then did `bin/grid stop all` and `bin/grid
start all`. But when I re-submitted the 3 hello-samza jobs along with my 2
new jobs, yarn still won't run my 5th job. The yarn web ui now shows Memory
Used = 4 GB and Memory Total = 8 GB, but my job still sits at State =
ACCEPTED.
When I tail deploy/yarn/logs/yarn-vagrant-resourcemanager-precise64.log I
see this repeated continuously:
https://gist.githubusercontent.com/zcox/0f2b260d29e18d40d038/raw/0d805bcd7d8fec5332756efc9c990679480df117/gistfile1.txt
I notice it says "available=<memory:4096, vCores:0>" - is my job not being
run now because vCores=0?
I also updated Vagrantfile to use:
samza.vm.provider :virtualbox do |vb| vb.memory = 4096 vb.cpus = 8 end
Thanks,
Zach
On Mon, Oct 6, 2014 at 12:40 PM, Lukas Steiblys <lu...@doubledutch.me>
wrote:
> I'll add that if you check the YARN node application master container log
> and see that the job is constantly restarting, you might need to increase
> the container memory limit to 1024MB at least. Also, a good parameter to
> play with in YARN is yarn.nodemanager.vmem-pmem-ratio.
>
> Lukas
>
> -----Original Message----- From: Mark Mindenhall
> Sent: Monday, October 6, 2014 8:44 AM
> To: dev@samza.incubator.apache.org
> Subject: Re: Problems running new jobs in hello-samza
>
>
> Hi Zach,
>
> I’m also a relative newbie, but I did run into this same issue. You are
> correct, in that your 5th job isn’t starting due to not enough resources
> available in the cluster, so you need to reduce the resources required.
>
> First, in yarn-site.xml I switched over to the FairScheduler<
> http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/
> FairScheduler.html>:
>
> <property>
> <name>yarn.resourcemanager.scheduler.class</name>
> <value>org.apache.hadoop.yarn.server.resourcemanager.
> scheduler.fair.FairScheduler</value>
> </property>
>
> I also added these two properties (yarn-site.xml) to control the amount of
> memory allocated to each job:
>
> <property>
> <name>yarn.scheduler.minimum-allocation-mb</name>
> <value>256</value>
> <description>Minimum limit of memory to allocate to each container
> request at the Resource Manager.</description>
> </property>
> <property>
> <name>yarn.scheduler.maximum-allocation-mb</name>
> <value>512</value>
> <description>Maximum limit of memory to allocate to each container
> request at the Resource Manager.</description>
> </property>
>
> Then, in each of my Samza properties files describing my jobs, I added the
> following two settings:
>
> yarn.container.memory.mb=512
> yarn.am.container.memory.mb=256
>
> Hope that helps!
>
> Best,
> Mark
>
>
> On Oct 6, 2014, at 6:27 AM, Zach Cox <zcox522@gmail.com<mailto:zcox
> 522@gmail.com>> wrote:
>
> Hi - I'm just getting started with Samza. I got the hello-samza example
> working properly in the vagrant box. Then I wrote 2 new tasks, rebuilt
> everything and submitted them to yarn using run-job.sh. These 2 new jobs
> show up in the yarn web ui, however only one of them has State=RUNNING, the
> other just sits forever at State=ACCEPTED.
>
> The Cluster Metrics section shows some interesting things:
> - Apps Pending = 1
> - Apps Running = 4
> - Containers Running = 8
> - Memory Used = 8 GB
> - Memory Total = 8 GB
> - Memory Reserved = 0 B
>
> Again I'm really new to samza & yarn, but does this mean that the node on
> this vagrant box has 8 GB memory available but all 8 GB is being used, so
> it can't run the 5th samza job?
>
> Are there 8 containers running because each Samza job has an
> ApplicationMaster and a SamzaContainer? Are each of those containers using
> 1 GB memory, and that's why all the available memory is used up? Do these
> containers really need 1 GB memory each? Can this be adjusted somehow?
>
> Just trying to better understand what's going on here, and see if there's a
> simple way to get both of my new tasks running in hello-samza.
>
> Thanks,
> Zach
>
>
Re: Problems running new jobs in hello-samza
Posted by Lukas Steiblys <lu...@doubledutch.me>.
I'll add that if you check the YARN node application master container log
and see that the job is constantly restarting, you might need to increase
the container memory limit to 1024MB at least. Also, a good parameter to
play with in YARN is yarn.nodemanager.vmem-pmem-ratio.
Lukas
-----Original Message-----
From: Mark Mindenhall
Sent: Monday, October 6, 2014 8:44 AM
To: dev@samza.incubator.apache.org
Subject: Re: Problems running new jobs in hello-samza
Hi Zach,
I’m also a relative newbie, but I did run into this same issue. You are
correct, in that your 5th job isn’t starting due to not enough resources
available in the cluster, so you need to reduce the resources required.
First, in yarn-site.xml I switched over to the
FairScheduler<http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/FairScheduler.html>:
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>
I also added these two properties (yarn-site.xml) to control the amount of
memory allocated to each job:
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>256</value>
<description>Minimum limit of memory to allocate to each container
request at the Resource Manager.</description>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>512</value>
<description>Maximum limit of memory to allocate to each container
request at the Resource Manager.</description>
</property>
Then, in each of my Samza properties files describing my jobs, I added the
following two settings:
yarn.container.memory.mb=512
yarn.am.container.memory.mb=256
Hope that helps!
Best,
Mark
On Oct 6, 2014, at 6:27 AM, Zach Cox
<zc...@gmail.com>> wrote:
Hi - I'm just getting started with Samza. I got the hello-samza example
working properly in the vagrant box. Then I wrote 2 new tasks, rebuilt
everything and submitted them to yarn using run-job.sh. These 2 new jobs
show up in the yarn web ui, however only one of them has State=RUNNING, the
other just sits forever at State=ACCEPTED.
The Cluster Metrics section shows some interesting things:
- Apps Pending = 1
- Apps Running = 4
- Containers Running = 8
- Memory Used = 8 GB
- Memory Total = 8 GB
- Memory Reserved = 0 B
Again I'm really new to samza & yarn, but does this mean that the node on
this vagrant box has 8 GB memory available but all 8 GB is being used, so
it can't run the 5th samza job?
Are there 8 containers running because each Samza job has an
ApplicationMaster and a SamzaContainer? Are each of those containers using
1 GB memory, and that's why all the available memory is used up? Do these
containers really need 1 GB memory each? Can this be adjusted somehow?
Just trying to better understand what's going on here, and see if there's a
simple way to get both of my new tasks running in hello-samza.
Thanks,
Zach
Re: Problems running new jobs in hello-samza
Posted by Mark Mindenhall <ma...@machineshop.io>.
Hi Zach,
I’m also a relative newbie, but I did run into this same issue. You are correct, in that your 5th job isn’t starting due to not enough resources available in the cluster, so you need to reduce the resources required.
First, in yarn-site.xml I switched over to the FairScheduler<http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/FairScheduler.html>:
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>
I also added these two properties (yarn-site.xml) to control the amount of memory allocated to each job:
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>256</value>
<description>Minimum limit of memory to allocate to each container request at the Resource Manager.</description>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>512</value>
<description>Maximum limit of memory to allocate to each container request at the Resource Manager.</description>
</property>
Then, in each of my Samza properties files describing my jobs, I added the following two settings:
yarn.container.memory.mb=512
yarn.am.container.memory.mb=256
Hope that helps!
Best,
Mark
On Oct 6, 2014, at 6:27 AM, Zach Cox <zc...@gmail.com>> wrote:
Hi - I'm just getting started with Samza. I got the hello-samza example
working properly in the vagrant box. Then I wrote 2 new tasks, rebuilt
everything and submitted them to yarn using run-job.sh. These 2 new jobs
show up in the yarn web ui, however only one of them has State=RUNNING, the
other just sits forever at State=ACCEPTED.
The Cluster Metrics section shows some interesting things:
- Apps Pending = 1
- Apps Running = 4
- Containers Running = 8
- Memory Used = 8 GB
- Memory Total = 8 GB
- Memory Reserved = 0 B
Again I'm really new to samza & yarn, but does this mean that the node on
this vagrant box has 8 GB memory available but all 8 GB is being used, so
it can't run the 5th samza job?
Are there 8 containers running because each Samza job has an
ApplicationMaster and a SamzaContainer? Are each of those containers using
1 GB memory, and that's why all the available memory is used up? Do these
containers really need 1 GB memory each? Can this be adjusted somehow?
Just trying to better understand what's going on here, and see if there's a
simple way to get both of my new tasks running in hello-samza.
Thanks,
Zach
Re: Problems running new jobs in hello-samza
Posted by Chris Riccomini <cr...@linkedin.com.INVALID>.
Hey Zach,
The Vagrant box is configured to have 2048MB of memory:
https://github.com/apache/incubator-samza-hello-samza/blob/master/Vagrantfi
le
The YARN NM by default is configured to have 8GB of memory allotted to it.
This is just an oversight on our part. I'll open a JIRA for that.
Now, your NM has 8GB allotted to it, and all 8GB are being used. Once this
happens, any new containers that need to be started aren't going to be
able to start because there's no space to start them. If the container
that needs to be started is a Samza AM (ApplicationMaster), then the job
will sit in the ACCEPTED state.
You'll need to do one of the following:
1. Run fewer jobs
2. Lower the yarn.container.memory.mb (and probably heap usage if you
customized task.opts).
3. Increase the NM's allotted GB space (yarn-site.xml) and bump up the
Vagrant box's memory footprint as well.
Cheers,
Chris
On 10/6/14 5:27 AM, "Zach Cox" <zc...@gmail.com> wrote:
>Hi - I'm just getting started with Samza. I got the hello-samza example
>working properly in the vagrant box. Then I wrote 2 new tasks, rebuilt
>everything and submitted them to yarn using run-job.sh. These 2 new jobs
>show up in the yarn web ui, however only one of them has State=RUNNING,
>the
>other just sits forever at State=ACCEPTED.
>
>The Cluster Metrics section shows some interesting things:
> - Apps Pending = 1
> - Apps Running = 4
> - Containers Running = 8
> - Memory Used = 8 GB
> - Memory Total = 8 GB
> - Memory Reserved = 0 B
>
>Again I'm really new to samza & yarn, but does this mean that the node on
>this vagrant box has 8 GB memory available but all 8 GB is being used, so
>it can't run the 5th samza job?
>
>Are there 8 containers running because each Samza job has an
>ApplicationMaster and a SamzaContainer? Are each of those containers using
>1 GB memory, and that's why all the available memory is used up? Do these
>containers really need 1 GB memory each? Can this be adjusted somehow?
>
>Just trying to better understand what's going on here, and see if there's
>a
>simple way to get both of my new tasks running in hello-samza.
>
>Thanks,
>Zach