You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Lior Schachter <li...@infolinks.com> on 2011/03/17 17:38:01 UTC
jobtracker heap
Hi,
I'm trying to increase the job-tracker heap by adding Xmx definition to
hadoop-env.sh (adding it to HADOOP_TASKTRACKER_OPTS).
I keep getting this error while starting hadoop:
Invalid maximum heap size: -Xmx=2048m
Could not create the Java virtual machine.
Any idea what am I doing wrong ?
Re: jobtracker heap
Posted by Harsh J <qw...@gmail.com>.
Nope, AFAIK 0.20.x does not have properties that can apply JVM options
separately to map and reduce tasks. "mapred.child.java.opts", which
applies to both, is the only go [mapred-site.xml with a final, or set
it specifically in jobs].
On Thu, Mar 17, 2011 at 10:59 PM, Lior Schachter <li...@infolinks.com> wrote:
> we are using hbase hadoop version 0.20.3
>
> maybe I can do it using the jobconf (separate jvm opts for map/reduce) ?
>
> On Thu, Mar 17, 2011 at 7:23 PM, Harsh J <qw...@gmail.com> wrote:
>>
>> What version of Hadoop are you using?
>>
>> 1 is a new property in 0.21, and is the right one if you're using that.
>> 2 is the property to use in 0.20, and applies to both Maps and
>> Reduces. In 0.20 there is no distinction available for setting JVM
>> opts to types of tasks.
>>
>> On Thu, Mar 17, 2011 at 10:37 PM, Lior Schachter <li...@infolinks.com>
>> wrote:
>> > sure. my mistake. thanks.
>> >
>> > I'm also trying to increase the reducer heap size (keeping the map with
>> > the
>> > default 200m). I found 2 related properties:
>> > 1. mapreduce.reduce.java.opts
>> > 2. mapred.child.java.opts
>> >
>> > Which should be configured and whats the purpose of the second property.
>> >
>> > thanks,
>> > Lior
>> >
>> > On Thu, Mar 17, 2011 at 6:41 PM, Harsh J <qw...@gmail.com> wrote:
>> >>
>> >> Hello,
>> >>
>> >> Remove the "=" from "-Xmx=2048m".
>> >>
>> >> P.s. If you're looking to change the JobTracker opts, it is
>> >> HADOOP_JOBTRACKER_OPTS, not TASKTRACKER :)
>> >>
>> >> On Thu, Mar 17, 2011 at 10:08 PM, Lior Schachter <li...@infolinks.com>
>> >> wrote:
>> >> > Hi,
>> >> > I'm trying to increase the job-tracker heap by adding Xmx definition
>> >> > to
>> >> > hadoop-env.sh (adding it to HADOOP_TASKTRACKER_OPTS).
>> >> > I keep getting this error while starting hadoop:
>> >> > Invalid maximum heap size: -Xmx=2048m
>> >> > Could not create the Java virtual machine.
>> >> >
>> >> > Any idea what am I doing wrong ?
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Harsh J
>> >> http://harshj.com
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>> http://harshj.com
>
>
--
Harsh J
http://harshj.com
Re: jobtracker heap
Posted by Lior Schachter <li...@infolinks.com>.
we are using hbase hadoop version 0.20.3
maybe I can do it using the jobconf (separate jvm opts for map/reduce) ?
On Thu, Mar 17, 2011 at 7:23 PM, Harsh J <qw...@gmail.com> wrote:
> What version of Hadoop are you using?
>
> 1 is a new property in 0.21, and is the right one if you're using that.
> 2 is the property to use in 0.20, and applies to both Maps and
> Reduces. In 0.20 there is no distinction available for setting JVM
> opts to types of tasks.
>
> On Thu, Mar 17, 2011 at 10:37 PM, Lior Schachter <li...@infolinks.com>
> wrote:
> > sure. my mistake. thanks.
> >
> > I'm also trying to increase the reducer heap size (keeping the map with
> the
> > default 200m). I found 2 related properties:
> > 1. mapreduce.reduce.java.opts
> > 2. mapred.child.java.opts
> >
> > Which should be configured and whats the purpose of the second property.
> >
> > thanks,
> > Lior
> >
> > On Thu, Mar 17, 2011 at 6:41 PM, Harsh J <qw...@gmail.com> wrote:
> >>
> >> Hello,
> >>
> >> Remove the "=" from "-Xmx=2048m".
> >>
> >> P.s. If you're looking to change the JobTracker opts, it is
> >> HADOOP_JOBTRACKER_OPTS, not TASKTRACKER :)
> >>
> >> On Thu, Mar 17, 2011 at 10:08 PM, Lior Schachter <li...@infolinks.com>
> >> wrote:
> >> > Hi,
> >> > I'm trying to increase the job-tracker heap by adding Xmx definition
> to
> >> > hadoop-env.sh (adding it to HADOOP_TASKTRACKER_OPTS).
> >> > I keep getting this error while starting hadoop:
> >> > Invalid maximum heap size: -Xmx=2048m
> >> > Could not create the Java virtual machine.
> >> >
> >> > Any idea what am I doing wrong ?
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
> >> http://harshj.com
> >
> >
>
>
>
> --
> Harsh J
> http://harshj.com
>
Re: jobtracker heap
Posted by Harsh J <qw...@gmail.com>.
What version of Hadoop are you using?
1 is a new property in 0.21, and is the right one if you're using that.
2 is the property to use in 0.20, and applies to both Maps and
Reduces. In 0.20 there is no distinction available for setting JVM
opts to types of tasks.
On Thu, Mar 17, 2011 at 10:37 PM, Lior Schachter <li...@infolinks.com> wrote:
> sure. my mistake. thanks.
>
> I'm also trying to increase the reducer heap size (keeping the map with the
> default 200m). I found 2 related properties:
> 1. mapreduce.reduce.java.opts
> 2. mapred.child.java.opts
>
> Which should be configured and whats the purpose of the second property.
>
> thanks,
> Lior
>
> On Thu, Mar 17, 2011 at 6:41 PM, Harsh J <qw...@gmail.com> wrote:
>>
>> Hello,
>>
>> Remove the "=" from "-Xmx=2048m".
>>
>> P.s. If you're looking to change the JobTracker opts, it is
>> HADOOP_JOBTRACKER_OPTS, not TASKTRACKER :)
>>
>> On Thu, Mar 17, 2011 at 10:08 PM, Lior Schachter <li...@infolinks.com>
>> wrote:
>> > Hi,
>> > I'm trying to increase the job-tracker heap by adding Xmx definition to
>> > hadoop-env.sh (adding it to HADOOP_TASKTRACKER_OPTS).
>> > I keep getting this error while starting hadoop:
>> > Invalid maximum heap size: -Xmx=2048m
>> > Could not create the Java virtual machine.
>> >
>> > Any idea what am I doing wrong ?
>> >
>>
>>
>>
>> --
>> Harsh J
>> http://harshj.com
>
>
--
Harsh J
http://harshj.com
Re: jobtracker heap
Posted by Lior Schachter <li...@infolinks.com>.
sure. my mistake. thanks.
I'm also trying to increase the reducer heap size (keeping the map with the
default 200m). I found 2 related properties:
1. mapreduce.reduce.java.opts
2. mapred.child.java.opts
Which should be configured and whats the purpose of the second property.
thanks,
Lior
On Thu, Mar 17, 2011 at 6:41 PM, Harsh J <qw...@gmail.com> wrote:
> Hello,
>
> Remove the "=" from "-Xmx=2048m".
>
> P.s. If you're looking to change the JobTracker opts, it is
> HADOOP_JOBTRACKER_OPTS, not TASKTRACKER :)
>
> On Thu, Mar 17, 2011 at 10:08 PM, Lior Schachter <li...@infolinks.com>
> wrote:
> > Hi,
> > I'm trying to increase the job-tracker heap by adding Xmx definition to
> > hadoop-env.sh (adding it to HADOOP_TASKTRACKER_OPTS).
> > I keep getting this error while starting hadoop:
> > Invalid maximum heap size: -Xmx=2048m
> > Could not create the Java virtual machine.
> >
> > Any idea what am I doing wrong ?
> >
>
>
>
> --
> Harsh J
> http://harshj.com
>
Re: jobtracker heap
Posted by Harsh J <qw...@gmail.com>.
Hello,
Remove the "=" from "-Xmx=2048m".
P.s. If you're looking to change the JobTracker opts, it is
HADOOP_JOBTRACKER_OPTS, not TASKTRACKER :)
On Thu, Mar 17, 2011 at 10:08 PM, Lior Schachter <li...@infolinks.com> wrote:
> Hi,
> I'm trying to increase the job-tracker heap by adding Xmx definition to
> hadoop-env.sh (adding it to HADOOP_TASKTRACKER_OPTS).
> I keep getting this error while starting hadoop:
> Invalid maximum heap size: -Xmx=2048m
> Could not create the Java virtual machine.
>
> Any idea what am I doing wrong ?
>
--
Harsh J
http://harshj.com