You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Peter Marron <Pe...@trilliumsoftware.com> on 2012/11/23 09:47:17 UTC
Creating Indexes again
Hi,
I'm trying to create indexes in Hive, and I've switched
to using CDH-4. The creation of the index is failing and
it's pretty obvious that the reducers are running out of
heap space. When I use the web interface for the
"Hadoop reduce task list" I can find this entry:
Error: Java heap space
Error: GC overhead limit exceeded
org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException: EEXIST: File exists
at org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:178)
at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:303)
at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:376)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: EEXIST: File exists
at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)
at org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:172)
... 7 more
Error: GC overhead limit exceeded
Al
If Also when the
If Also when the
If Also when the
If this e-mail shouldn't be here and should only be on
a cloudera mailing list, please re-direct me.
Thanks in advance.
Peter Marron
Trillium Software UK Limited
Tel : +44 (0) 118 940 7609
Fax : +44 (0) 118 940 7699
E: Peter.Marron@TrilliumSoftware.com<ma...@TrilliumSoftware.com>
Re: Creating Indexes again
Posted by Nitin Pawar <ni...@gmail.com>.
CDH forum might be better place to answer this. I never used CDH
On Fri, Nov 23, 2012 at 2:28 PM, Peter Marron <
Peter.Marron@trilliumsoftware.com> wrote:
> Hi Nitin,****
>
> ** **
>
> Can I set these parameters through the CDH management interface?****
>
> If not then what file do they need to be set in to make sure that CDH****
>
> picks them up?****
>
> ** **
>
> Peter Marron****
>
> Trillium Software UK Limited****
>
> ** **
>
> Tel : +44 (0) 118 940 7609****
>
> Fax : +44 (0) 118 940 7699****
>
> E: Peter.Marron@TrilliumSoftware.com <Ro...@TrilliumSoftware.com>**
> **
>
> ** **
>
> *From:* Nitin Pawar [mailto:nitinpawar432@gmail.com]
> *Sent:* 23 November 2012 08:55
> *To:* user@hive.apache.org
> *Subject:* Re: Creating Indexes again****
>
> ** **
>
> try increasing ulimit on your hadoop cluster as well increase the memory
> for map and reducer both by setting them up on hive ****
>
> set mapred.job.map.memory.mb=6000;****
>
> set mapred.job.reduce.memory.mb=4000;****
>
> ** **
>
> you can change the values based on the hadoop cluster you have setup ****
>
> ** **
>
> ** **
>
> On Fri, Nov 23, 2012 at 2:17 PM, Peter Marron <
> Peter.Marron@trilliumsoftware.com> wrote:****
>
> Hi,****
>
> ****
>
> I’m trying to create indexes in Hive, and I’ve switched****
>
> to using CDH-4. The creation of the index is failing and****
>
> it’s pretty obvious that the reducers are running out of****
>
> heap space. When I use the web interface for the****
>
> “Hadoop reduce task list” I can find this entry:****
>
> ****
>
> Error: Java heap space****
>
> Error: GC overhead limit exceeded****
>
> org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException: EEXIST: File
> exists****
>
> at
> org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:178)*
> ***
>
> at
> org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:303)****
>
> at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:376)****
>
> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)****
>
> at java.security.AccessController.doPrivileged(Native Method)****
>
> at javax.security.auth.Subject.doAs(Subject.java:396)****
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
> ****
>
> at org.apache.hadoop.mapred.Child.main(Child.java:262)****
>
> Caused by: EEXIST: File exists****
>
> at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)****
>
> at
> org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:172)*
> ***
>
> ... 7 more****
>
> ****
>
> Error: GC overhead limit exceeded****
>
> ****
>
> *Al*****
>
> ****
>
> If Also when the****
>
> If Also when the****
>
> ****
>
> ****
>
> If Also when the****
>
> ****
>
> If this e-mail shouldn’t be here and should only be on****
>
> a cloudera mailing list, please re-direct me.****
>
> ****
>
> Thanks in advance.****
>
> ****
>
> Peter Marron****
>
> Trillium Software UK Limited****
>
> ****
>
> Tel : +44 (0) 118 940 7609****
>
> Fax : +44 (0) 118 940 7699****
>
> E: Peter.Marron@TrilliumSoftware.com <Ro...@TrilliumSoftware.com>**
> **
>
> ****
>
>
>
> ****
>
> ** **
>
> --
> Nitin Pawar****
>
--
Nitin Pawar
RE: Creating Indexes again
Posted by Peter Marron <Pe...@trilliumsoftware.com>.
Hi Nitin,
Can I set these parameters through the CDH management interface?
If not then what file do they need to be set in to make sure that CDH
picks them up?
Peter Marron
Trillium Software UK Limited
Tel : +44 (0) 118 940 7609
Fax : +44 (0) 118 940 7699
E: Peter.Marron@TrilliumSoftware.com<ma...@TrilliumSoftware.com>
From: Nitin Pawar [mailto:nitinpawar432@gmail.com]
Sent: 23 November 2012 08:55
To: user@hive.apache.org
Subject: Re: Creating Indexes again
try increasing ulimit on your hadoop cluster as well increase the memory for map and reducer both by setting them up on hive
set mapred.job.map.memory.mb=6000;
set mapred.job.reduce.memory.mb=4000;
you can change the values based on the hadoop cluster you have setup
On Fri, Nov 23, 2012 at 2:17 PM, Peter Marron <Pe...@trilliumsoftware.com>> wrote:
Hi,
I'm trying to create indexes in Hive, and I've switched
to using CDH-4. The creation of the index is failing and
it's pretty obvious that the reducers are running out of
heap space. When I use the web interface for the
"Hadoop reduce task list" I can find this entry:
Error: Java heap space
Error: GC overhead limit exceeded
org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException: EEXIST: File exists
at org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:178)
at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:303)
at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:376)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: EEXIST: File exists
at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)
at org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:172)
... 7 more
Error: GC overhead limit exceeded
Al
If Also when the
If Also when the
If Also when the
If this e-mail shouldn't be here and should only be on
a cloudera mailing list, please re-direct me.
Thanks in advance.
Peter Marron
Trillium Software UK Limited
Tel : +44 (0) 118 940 7609<tel:%2B44%20%280%29%20118%20940%207609>
Fax : +44 (0) 118 940 7699<tel:%2B44%20%280%29%20118%20940%207699>
E: Peter.Marron@TrilliumSoftware.com<ma...@TrilliumSoftware.com>
--
Nitin Pawar
Re: Creating Indexes again
Posted by Nitin Pawar <ni...@gmail.com>.
try increasing ulimit on your hadoop cluster as well increase the memory
for map and reducer both by setting them up on hive
set mapred.job.map.memory.mb=6000;
set mapred.job.reduce.memory.mb=4000;
you can change the values based on the hadoop cluster you have setup
On Fri, Nov 23, 2012 at 2:17 PM, Peter Marron <
Peter.Marron@trilliumsoftware.com> wrote:
> Hi,****
>
> ** **
>
> I’m trying to create indexes in Hive, and I’ve switched****
>
> to using CDH-4. The creation of the index is failing and****
>
> it’s pretty obvious that the reducers are running out of****
>
> heap space. When I use the web interface for the****
>
> “Hadoop reduce task list” I can find this entry:****
>
> ** **
>
> Error: Java heap space****
>
> Error: GC overhead limit exceeded****
>
> org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException: EEXIST: File
> exists****
>
> at
> org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:178)*
> ***
>
> at
> org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:303)****
>
> at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:376)****
>
> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)****
>
> at java.security.AccessController.doPrivileged(Native Method)****
>
> at javax.security.auth.Subject.doAs(Subject.java:396)****
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
> ****
>
> at org.apache.hadoop.mapred.Child.main(Child.java:262)****
>
> Caused by: EEXIST: File exists****
>
> at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)****
>
> at
> org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:172)*
> ***
>
> ... 7 more****
>
> ** **
>
> Error: GC overhead limit exceeded****
>
> ** **
>
> *Al*
>
> ** **
>
> If Also when the****
>
> If Also when the****
>
> ** **
>
> ** **
>
> If Also when the****
>
> ** **
>
> If this e-mail shouldn’t be here and should only be on****
>
> a cloudera mailing list, please re-direct me.****
>
> ** **
>
> Thanks in advance.****
>
> ** **
>
> Peter Marron****
>
> Trillium Software UK Limited****
>
> ** **
>
> Tel : +44 (0) 118 940 7609****
>
> Fax : +44 (0) 118 940 7699****
>
> E: Peter.Marron@TrilliumSoftware.com <Ro...@TrilliumSoftware.com>**
> **
>
> ** **
>
--
Nitin Pawar