You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Sarfraz Ramay <sa...@gmail.com> on 2014/07/22 09:19:31 UTC
Hive shell code exception, urgent help needed
Hi,
I am using Hive 0.13.1 and Hadoop 2-2.0 on amazon EC2 t2.micro instances. I
have 4 instances, master has the namenode and yarn, secondarynode is a
separate instance and two slaves are on separate instances each.
It was working fine till now but it started to break when i tried to run
the following query on tpch generated 3GB data. same worked ok on 1GB
SELECT
l_orderkey
, sum(l_extendedprice*(1-l_discount)) as revenue
, o_orderdate
, o_shippriority
FROM
customer c JOIN orders o
ON (c.c_custkey = o.o_custkey)
JOIN lineitem l
on (l.l_orderkey = o.o_orderkey)
WHERE
o_orderdate < '1995-03-15' and l_shipdate > '1995-03-15'
AND c.c_mktsegment = 'AUTOMOBILE'
GROUP BY
l_orderkey, o_orderdate, o_shippriority
HAVING
sum(l_extendedprice*(1-l_discount)) > 38500 --average revenue
--LIMIT 10;
i have tried many things but nothing seems to work. I am also attaching my
mapred-site.xml and yarn-site.xml files for reference plus the error log. I
have also tried to limit the memory settings in mapred-site.xml and
yarn-site but nothing seems to be working. For full log details please find
attached hive.log file. Please help!
Hadoop job information for Stage-7: number of mappers: 9; number of
reducers: 0
2014-07-22 06:39:31,643 Stage-7 map = 0%, reduce = 0%
2014-07-22 06:39:43,940 Stage-7 map = 6%, reduce = 0%, Cumulative CPU 5.34
sec
2014-07-22 06:39:45,002 Stage-7 map = 11%, reduce = 0%, Cumulative CPU
6.94 sec
2014-07-22 06:40:08,373 Stage-7 map = 17%, reduce = 0%, Cumulative CPU
12.6 sec
2014-07-22 06:40:10,417 Stage-7 map = 22%, reduce = 0%, Cumulative CPU
14.06 sec
2014-07-22 06:40:22,732 Stage-7 map = 28%, reduce = 0%, Cumulative CPU
24.46 sec
2014-07-22 06:40:25,843 Stage-7 map = 33%, reduce = 0%, Cumulative CPU
25.74 sec
2014-07-22 06:40:33,039 Stage-7 map = 44%, reduce = 0%, Cumulative CPU
33.32 sec
2014-07-22 06:40:38,709 Stage-7 map = 56%, reduce = 0%, Cumulative CPU
37.19 sec
2014-07-22 06:41:07,648 Stage-7 map = 61%, reduce = 0%, Cumulative CPU
42.83 sec
2014-07-22 06:41:15,900 Stage-7 map = 56%, reduce = 0%, Cumulative CPU
39.49 sec
2014-07-22 06:41:27,299 Stage-7 map = 67%, reduce = 0%, Cumulative CPU
46.07 sec
2014-07-22 06:41:28,342 Stage-7 map = 56%, reduce = 0%, Cumulative CPU
40.9 sec
2014-07-22 06:41:43,753 Stage-7 map = 61%, reduce = 0%, Cumulative CPU
42.84 sec
2014-07-22 06:41:45,801 Stage-7 map = 100%, reduce = 0%, Cumulative CPU
37.19 sec
MapReduce Total cumulative CPU time: 37 seconds 190 msec
Ended Job = job_1406011031680_0002 with errors
Error during job, obtaining debugging information...
Job Tracking URL:
http://ec2-54-77-76-145.eu-west-1.compute.amazonaws.com:8088/proxy/application_1406011031680_0002/
Examining task ID: task_1406011031680_0002_m_000001 (and more) from job
job_1406011031680_0002
Examining task ID: task_1406011031680_0002_m_000005 (and more) from job
job_1406011031680_0002
Task with the most failures(4):
-----
Task ID:
task_1406011031680_0002_m_000008
URL:
http://ec2-54-77-76-145.eu-west-1.compute.amazonaws.com:8088/taskdetails.jsp?jobid=job_1406011031680_0002&tipid=task_1406011031680_0002_m_000008
-----
Diagnostic Messages for this Task:
Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 3 Reduce: 1 Cumulative CPU: 24.58 sec HDFS Read: 593821601
HDFS Write: 14518009 SUCCESS
Job 1: Map: 9 Cumulative CPU: 37.19 sec HDFS Read: 1342219615 HDFS
Write: 821879 FAIL
Total MapReduce CPU Time Spent: 1 minutes 1 seconds 770 msec
hive (default)> exit;
Regards,
Sarfraz Rasheed Ramay (DIT)
Dublin, Ireland.
Re: Hive shell code exception, urgent help needed
Posted by Juan Martin Pampliega <jp...@gmail.com>.
Depending on the size of the data you are processing 4gb might not be
enough. Do you have any idea what's the size of your data? Have you tried
limiting the queries with LIMIT keyword to a few hundred rows and seeing if
they run ok?
On Jul 25, 2014 6:55 AM, "Sarfraz Ramay" <sa...@gmail.com> wrote:
> Hi,
>
> I have 4 instances on ec2
> 1 master with namenode and YARN running on it
> 1 for secondary namenode and 2 slaves
>
> I used t2.medium instance for the master node only and left the rest as
> they were and still i got the same exception. t2.medium is a decent
> instance with 4GB RAM and 2 CPUs so i don't think this exception is related
> to memory. Any other suggestions please ?
>
> Regards,
> Sarfraz Rasheed Ramay (DIT)
> Dublin, Ireland.
>
>
> On Fri, Jul 25, 2014 at 1:51 AM, Juan Martin Pampliega <
> jpampliega@gmail.com> wrote:
>
>> Hi,
>> The actual useful part of the error is:
>>
>> Execution Error, return code 2 from
>> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
>>
>> If you do a search for this plus "EC2" in Google you will find a couple
>> of results that point to memory exhaustion issues. You should try
>> increasing the configurated memory size.
>>
>> Since you are using a t2.micro you should really try using a bigger
>> Amazon instance size. This might probably be a lot more useful than trying
>> different configurations.
>> On Jul 24, 2014 7:08 AM, "Sarfraz Ramay" <sa...@gmail.com>
>> wrote:
>>
>>> Can anyone please help with this ?
>>>
>>> [image: Inline image 1]
>>>
>>>
>>> i followed the advice here
>>> http://stackoverflow.com/questions/20390217/mapreduce-job-in-headless-environment-fails-n-times-due-to-am-container-exceptio
>>>
>>> and added to mapred-site.xml following properties but still getting the
>>> same error.
>>>
>>>
>>> <property>
>>> <name>mapred.child.java.opts</name>
>>> <value>-Djava.awt.headless=true</value></property><!-- add headless to default -Xmx1024m --><property>
>>> <name>yarn.app.mapreduce.am.command-opts</name>
>>> <value>-Djava.awt.headless=true -Xmx1024m</value></property><property>
>>> <name>yarn.app.mapreduce.am.admin-command-opts</name>
>>> <value>-Djava.awt.headless=true</value></property
>>>
>>>
>>>
>>> Regards,
>>> Sarfraz Rasheed Ramay (DIT)
>>> Dublin, Ireland.
>>>
>>>
>>> On Tue, Jul 22, 2014 at 8:19 AM, Sarfraz Ramay <sa...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am using Hive 0.13.1 and Hadoop 2-2.0 on amazon EC2 t2.micro
>>>> instances. I have 4 instances, master has the namenode and yarn,
>>>> secondarynode is a separate instance and two slaves are on separate
>>>> instances each.
>>>>
>>>> It was working fine till now but it started to break when i tried to
>>>> run the following query on tpch generated 3GB data. same worked ok on 1GB
>>>>
>>>> SELECT
>>>> l_orderkey
>>>> , sum(l_extendedprice*(1-l_discount)) as revenue
>>>> , o_orderdate
>>>> , o_shippriority
>>>> FROM
>>>> customer c JOIN orders o
>>>> ON (c.c_custkey = o.o_custkey)
>>>> JOIN lineitem l
>>>> on (l.l_orderkey = o.o_orderkey)
>>>> WHERE
>>>> o_orderdate < '1995-03-15' and l_shipdate > '1995-03-15'
>>>> AND c.c_mktsegment = 'AUTOMOBILE'
>>>> GROUP BY
>>>> l_orderkey, o_orderdate, o_shippriority
>>>> HAVING
>>>> sum(l_extendedprice*(1-l_discount)) > 38500 --average revenue
>>>> --LIMIT 10;
>>>>
>>>> i have tried many things but nothing seems to work. I am also attaching
>>>> my mapred-site.xml and yarn-site.xml files for reference plus the error
>>>> log. I have also tried to limit the memory settings in mapred-site.xml and
>>>> yarn-site but nothing seems to be working. For full log details please find
>>>> attached hive.log file. Please help!
>>>>
>>>> Hadoop job information for Stage-7: number of mappers: 9; number of
>>>> reducers: 0
>>>> 2014-07-22 06:39:31,643 Stage-7 map = 0%, reduce = 0%
>>>> 2014-07-22 06:39:43,940 Stage-7 map = 6%, reduce = 0%, Cumulative CPU
>>>> 5.34 sec
>>>> 2014-07-22 06:39:45,002 Stage-7 map = 11%, reduce = 0%, Cumulative CPU
>>>> 6.94 sec
>>>> 2014-07-22 06:40:08,373 Stage-7 map = 17%, reduce = 0%, Cumulative CPU
>>>> 12.6 sec
>>>> 2014-07-22 06:40:10,417 Stage-7 map = 22%, reduce = 0%, Cumulative CPU
>>>> 14.06 sec
>>>> 2014-07-22 06:40:22,732 Stage-7 map = 28%, reduce = 0%, Cumulative CPU
>>>> 24.46 sec
>>>> 2014-07-22 06:40:25,843 Stage-7 map = 33%, reduce = 0%, Cumulative CPU
>>>> 25.74 sec
>>>> 2014-07-22 06:40:33,039 Stage-7 map = 44%, reduce = 0%, Cumulative CPU
>>>> 33.32 sec
>>>> 2014-07-22 06:40:38,709 Stage-7 map = 56%, reduce = 0%, Cumulative CPU
>>>> 37.19 sec
>>>> 2014-07-22 06:41:07,648 Stage-7 map = 61%, reduce = 0%, Cumulative CPU
>>>> 42.83 sec
>>>> 2014-07-22 06:41:15,900 Stage-7 map = 56%, reduce = 0%, Cumulative CPU
>>>> 39.49 sec
>>>> 2014-07-22 06:41:27,299 Stage-7 map = 67%, reduce = 0%, Cumulative CPU
>>>> 46.07 sec
>>>> 2014-07-22 06:41:28,342 Stage-7 map = 56%, reduce = 0%, Cumulative CPU
>>>> 40.9 sec
>>>> 2014-07-22 06:41:43,753 Stage-7 map = 61%, reduce = 0%, Cumulative CPU
>>>> 42.84 sec
>>>> 2014-07-22 06:41:45,801 Stage-7 map = 100%, reduce = 0%, Cumulative
>>>> CPU 37.19 sec
>>>> MapReduce Total cumulative CPU time: 37 seconds 190 msec
>>>> Ended Job = job_1406011031680_0002 with errors
>>>> Error during job, obtaining debugging information...
>>>> Job Tracking URL:
>>>> http://ec2-54-77-76-145.eu-west-1.compute.amazonaws.com:8088/proxy/application_1406011031680_0002/
>>>> Examining task ID: task_1406011031680_0002_m_000001 (and more) from job
>>>> job_1406011031680_0002
>>>> Examining task ID: task_1406011031680_0002_m_000005 (and more) from job
>>>> job_1406011031680_0002
>>>>
>>>> Task with the most failures(4):
>>>> -----
>>>> Task ID:
>>>> task_1406011031680_0002_m_000008
>>>>
>>>> URL:
>>>>
>>>> http://ec2-54-77-76-145.eu-west-1.compute.amazonaws.com:8088/taskdetails.jsp?jobid=job_1406011031680_0002&tipid=task_1406011031680_0002_m_000008
>>>> -----
>>>> Diagnostic Messages for this Task:
>>>> Exception from container-launch:
>>>> org.apache.hadoop.util.Shell$ExitCodeException:
>>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
>>>> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>>>> at
>>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:744)
>>>>
>>>>
>>>> FAILED: Execution Error, return code 2 from
>>>> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
>>>> MapReduce Jobs Launched:
>>>> Job 0: Map: 3 Reduce: 1 Cumulative CPU: 24.58 sec HDFS Read:
>>>> 593821601 HDFS Write: 14518009 SUCCESS
>>>> Job 1: Map: 9 Cumulative CPU: 37.19 sec HDFS Read: 1342219615 HDFS
>>>> Write: 821879 FAIL
>>>> Total MapReduce CPU Time Spent: 1 minutes 1 seconds 770 msec
>>>> hive (default)> exit;
>>>>
>>>>
>>>>
>>>> Regards,
>>>> Sarfraz Rasheed Ramay (DIT)
>>>> Dublin, Ireland.
>>>>
>>>
>>>
>
Re: Hive shell code exception, urgent help needed
Posted by Sarfraz Ramay <sa...@gmail.com>.
Hi,
I have 4 instances on ec2
1 master with namenode and YARN running on it
1 for secondary namenode and 2 slaves
I used t2.medium instance for the master node only and left the rest as
they were and still i got the same exception. t2.medium is a decent
instance with 4GB RAM and 2 CPUs so i don't think this exception is related
to memory. Any other suggestions please ?
Regards,
Sarfraz Rasheed Ramay (DIT)
Dublin, Ireland.
On Fri, Jul 25, 2014 at 1:51 AM, Juan Martin Pampliega <jpampliega@gmail.com
> wrote:
> Hi,
> The actual useful part of the error is:
>
> Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
>
> If you do a search for this plus "EC2" in Google you will find a couple of
> results that point to memory exhaustion issues. You should try increasing
> the configurated memory size.
>
> Since you are using a t2.micro you should really try using a bigger Amazon
> instance size. This might probably be a lot more useful than trying
> different configurations.
> On Jul 24, 2014 7:08 AM, "Sarfraz Ramay" <sa...@gmail.com> wrote:
>
>> Can anyone please help with this ?
>>
>> [image: Inline image 1]
>>
>>
>> i followed the advice here
>> http://stackoverflow.com/questions/20390217/mapreduce-job-in-headless-environment-fails-n-times-due-to-am-container-exceptio
>>
>> and added to mapred-site.xml following properties but still getting the
>> same error.
>>
>> <property>
>> <name>mapred.child.java.opts</name>
>> <value>-Djava.awt.headless=true</value></property><!-- add headless to default -Xmx1024m --><property>
>> <name>yarn.app.mapreduce.am.command-opts</name>
>> <value>-Djava.awt.headless=true -Xmx1024m</value></property><property>
>> <name>yarn.app.mapreduce.am.admin-command-opts</name>
>> <value>-Djava.awt.headless=true</value></property
>>
>>
>>
>> Regards,
>> Sarfraz Rasheed Ramay (DIT)
>> Dublin, Ireland.
>>
>>
>> On Tue, Jul 22, 2014 at 8:19 AM, Sarfraz Ramay <sa...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I am using Hive 0.13.1 and Hadoop 2-2.0 on amazon EC2 t2.micro
>>> instances. I have 4 instances, master has the namenode and yarn,
>>> secondarynode is a separate instance and two slaves are on separate
>>> instances each.
>>>
>>> It was working fine till now but it started to break when i tried to run
>>> the following query on tpch generated 3GB data. same worked ok on 1GB
>>>
>>> SELECT
>>> l_orderkey
>>> , sum(l_extendedprice*(1-l_discount)) as revenue
>>> , o_orderdate
>>> , o_shippriority
>>> FROM
>>> customer c JOIN orders o
>>> ON (c.c_custkey = o.o_custkey)
>>> JOIN lineitem l
>>> on (l.l_orderkey = o.o_orderkey)
>>> WHERE
>>> o_orderdate < '1995-03-15' and l_shipdate > '1995-03-15'
>>> AND c.c_mktsegment = 'AUTOMOBILE'
>>> GROUP BY
>>> l_orderkey, o_orderdate, o_shippriority
>>> HAVING
>>> sum(l_extendedprice*(1-l_discount)) > 38500 --average revenue
>>> --LIMIT 10;
>>>
>>> i have tried many things but nothing seems to work. I am also attaching
>>> my mapred-site.xml and yarn-site.xml files for reference plus the error
>>> log. I have also tried to limit the memory settings in mapred-site.xml and
>>> yarn-site but nothing seems to be working. For full log details please find
>>> attached hive.log file. Please help!
>>>
>>> Hadoop job information for Stage-7: number of mappers: 9; number of
>>> reducers: 0
>>> 2014-07-22 06:39:31,643 Stage-7 map = 0%, reduce = 0%
>>> 2014-07-22 06:39:43,940 Stage-7 map = 6%, reduce = 0%, Cumulative CPU
>>> 5.34 sec
>>> 2014-07-22 06:39:45,002 Stage-7 map = 11%, reduce = 0%, Cumulative CPU
>>> 6.94 sec
>>> 2014-07-22 06:40:08,373 Stage-7 map = 17%, reduce = 0%, Cumulative CPU
>>> 12.6 sec
>>> 2014-07-22 06:40:10,417 Stage-7 map = 22%, reduce = 0%, Cumulative CPU
>>> 14.06 sec
>>> 2014-07-22 06:40:22,732 Stage-7 map = 28%, reduce = 0%, Cumulative CPU
>>> 24.46 sec
>>> 2014-07-22 06:40:25,843 Stage-7 map = 33%, reduce = 0%, Cumulative CPU
>>> 25.74 sec
>>> 2014-07-22 06:40:33,039 Stage-7 map = 44%, reduce = 0%, Cumulative CPU
>>> 33.32 sec
>>> 2014-07-22 06:40:38,709 Stage-7 map = 56%, reduce = 0%, Cumulative CPU
>>> 37.19 sec
>>> 2014-07-22 06:41:07,648 Stage-7 map = 61%, reduce = 0%, Cumulative CPU
>>> 42.83 sec
>>> 2014-07-22 06:41:15,900 Stage-7 map = 56%, reduce = 0%, Cumulative CPU
>>> 39.49 sec
>>> 2014-07-22 06:41:27,299 Stage-7 map = 67%, reduce = 0%, Cumulative CPU
>>> 46.07 sec
>>> 2014-07-22 06:41:28,342 Stage-7 map = 56%, reduce = 0%, Cumulative CPU
>>> 40.9 sec
>>> 2014-07-22 06:41:43,753 Stage-7 map = 61%, reduce = 0%, Cumulative CPU
>>> 42.84 sec
>>> 2014-07-22 06:41:45,801 Stage-7 map = 100%, reduce = 0%, Cumulative CPU
>>> 37.19 sec
>>> MapReduce Total cumulative CPU time: 37 seconds 190 msec
>>> Ended Job = job_1406011031680_0002 with errors
>>> Error during job, obtaining debugging information...
>>> Job Tracking URL:
>>> http://ec2-54-77-76-145.eu-west-1.compute.amazonaws.com:8088/proxy/application_1406011031680_0002/
>>> Examining task ID: task_1406011031680_0002_m_000001 (and more) from job
>>> job_1406011031680_0002
>>> Examining task ID: task_1406011031680_0002_m_000005 (and more) from job
>>> job_1406011031680_0002
>>>
>>> Task with the most failures(4):
>>> -----
>>> Task ID:
>>> task_1406011031680_0002_m_000008
>>>
>>> URL:
>>>
>>> http://ec2-54-77-76-145.eu-west-1.compute.amazonaws.com:8088/taskdetails.jsp?jobid=job_1406011031680_0002&tipid=task_1406011031680_0002_m_000008
>>> -----
>>> Diagnostic Messages for this Task:
>>> Exception from container-launch:
>>> org.apache.hadoop.util.Shell$ExitCodeException:
>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
>>> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>>> at
>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:744)
>>>
>>>
>>> FAILED: Execution Error, return code 2 from
>>> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
>>> MapReduce Jobs Launched:
>>> Job 0: Map: 3 Reduce: 1 Cumulative CPU: 24.58 sec HDFS Read:
>>> 593821601 HDFS Write: 14518009 SUCCESS
>>> Job 1: Map: 9 Cumulative CPU: 37.19 sec HDFS Read: 1342219615 HDFS
>>> Write: 821879 FAIL
>>> Total MapReduce CPU Time Spent: 1 minutes 1 seconds 770 msec
>>> hive (default)> exit;
>>>
>>>
>>>
>>> Regards,
>>> Sarfraz Rasheed Ramay (DIT)
>>> Dublin, Ireland.
>>>
>>
>>
Re: Hive shell code exception, urgent help needed
Posted by Sarfraz Ramay <sa...@gmail.com>.
Hi,
Thanks for your reply. Have been following links for the past two days now.
Finally got hadoop natively compiled. Let's see if that solves the problem.
Yes, increasing the memory was on my list but i think i tried that, didn't
work.
Memory can be issue as it is working perfectly fine for queries with
smaller result sets. Also it sometimes works for bigger queries as well.
Really strange behavior...Thanks for the tip again.
Regards,
Sarfraz Rasheed Ramay (DIT)
Dublin, Ireland.
On Fri, Jul 25, 2014 at 1:51 AM, Juan Martin Pampliega <jpampliega@gmail.com
> wrote:
> Hi,
> The actual useful part of the error is:
>
> Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
>
> If you do a search for this plus "EC2" in Google you will find a couple of
> results that point to memory exhaustion issues. You should try increasing
> the configurated memory size.
>
> Since you are using a t2.micro you should really try using a bigger Amazon
> instance size. This might probably be a lot more useful than trying
> different configurations.
> On Jul 24, 2014 7:08 AM, "Sarfraz Ramay" <sa...@gmail.com> wrote:
>
>> Can anyone please help with this ?
>>
>> [image: Inline image 1]
>>
>>
>> i followed the advice here
>> http://stackoverflow.com/questions/20390217/mapreduce-job-in-headless-environment-fails-n-times-due-to-am-container-exceptio
>>
>> and added to mapred-site.xml following properties but still getting the
>> same error.
>>
>> <property>
>> <name>mapred.child.java.opts</name>
>> <value>-Djava.awt.headless=true</value></property><!-- add headless to default -Xmx1024m --><property>
>> <name>yarn.app.mapreduce.am.command-opts</name>
>> <value>-Djava.awt.headless=true -Xmx1024m</value></property><property>
>> <name>yarn.app.mapreduce.am.admin-command-opts</name>
>> <value>-Djava.awt.headless=true</value></property
>>
>>
>>
>> Regards,
>> Sarfraz Rasheed Ramay (DIT)
>> Dublin, Ireland.
>>
>>
>> On Tue, Jul 22, 2014 at 8:19 AM, Sarfraz Ramay <sa...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I am using Hive 0.13.1 and Hadoop 2-2.0 on amazon EC2 t2.micro
>>> instances. I have 4 instances, master has the namenode and yarn,
>>> secondarynode is a separate instance and two slaves are on separate
>>> instances each.
>>>
>>> It was working fine till now but it started to break when i tried to run
>>> the following query on tpch generated 3GB data. same worked ok on 1GB
>>>
>>> SELECT
>>> l_orderkey
>>> , sum(l_extendedprice*(1-l_discount)) as revenue
>>> , o_orderdate
>>> , o_shippriority
>>> FROM
>>> customer c JOIN orders o
>>> ON (c.c_custkey = o.o_custkey)
>>> JOIN lineitem l
>>> on (l.l_orderkey = o.o_orderkey)
>>> WHERE
>>> o_orderdate < '1995-03-15' and l_shipdate > '1995-03-15'
>>> AND c.c_mktsegment = 'AUTOMOBILE'
>>> GROUP BY
>>> l_orderkey, o_orderdate, o_shippriority
>>> HAVING
>>> sum(l_extendedprice*(1-l_discount)) > 38500 --average revenue
>>> --LIMIT 10;
>>>
>>> i have tried many things but nothing seems to work. I am also attaching
>>> my mapred-site.xml and yarn-site.xml files for reference plus the error
>>> log. I have also tried to limit the memory settings in mapred-site.xml and
>>> yarn-site but nothing seems to be working. For full log details please find
>>> attached hive.log file. Please help!
>>>
>>> Hadoop job information for Stage-7: number of mappers: 9; number of
>>> reducers: 0
>>> 2014-07-22 06:39:31,643 Stage-7 map = 0%, reduce = 0%
>>> 2014-07-22 06:39:43,940 Stage-7 map = 6%, reduce = 0%, Cumulative CPU
>>> 5.34 sec
>>> 2014-07-22 06:39:45,002 Stage-7 map = 11%, reduce = 0%, Cumulative CPU
>>> 6.94 sec
>>> 2014-07-22 06:40:08,373 Stage-7 map = 17%, reduce = 0%, Cumulative CPU
>>> 12.6 sec
>>> 2014-07-22 06:40:10,417 Stage-7 map = 22%, reduce = 0%, Cumulative CPU
>>> 14.06 sec
>>> 2014-07-22 06:40:22,732 Stage-7 map = 28%, reduce = 0%, Cumulative CPU
>>> 24.46 sec
>>> 2014-07-22 06:40:25,843 Stage-7 map = 33%, reduce = 0%, Cumulative CPU
>>> 25.74 sec
>>> 2014-07-22 06:40:33,039 Stage-7 map = 44%, reduce = 0%, Cumulative CPU
>>> 33.32 sec
>>> 2014-07-22 06:40:38,709 Stage-7 map = 56%, reduce = 0%, Cumulative CPU
>>> 37.19 sec
>>> 2014-07-22 06:41:07,648 Stage-7 map = 61%, reduce = 0%, Cumulative CPU
>>> 42.83 sec
>>> 2014-07-22 06:41:15,900 Stage-7 map = 56%, reduce = 0%, Cumulative CPU
>>> 39.49 sec
>>> 2014-07-22 06:41:27,299 Stage-7 map = 67%, reduce = 0%, Cumulative CPU
>>> 46.07 sec
>>> 2014-07-22 06:41:28,342 Stage-7 map = 56%, reduce = 0%, Cumulative CPU
>>> 40.9 sec
>>> 2014-07-22 06:41:43,753 Stage-7 map = 61%, reduce = 0%, Cumulative CPU
>>> 42.84 sec
>>> 2014-07-22 06:41:45,801 Stage-7 map = 100%, reduce = 0%, Cumulative CPU
>>> 37.19 sec
>>> MapReduce Total cumulative CPU time: 37 seconds 190 msec
>>> Ended Job = job_1406011031680_0002 with errors
>>> Error during job, obtaining debugging information...
>>> Job Tracking URL:
>>> http://ec2-54-77-76-145.eu-west-1.compute.amazonaws.com:8088/proxy/application_1406011031680_0002/
>>> Examining task ID: task_1406011031680_0002_m_000001 (and more) from job
>>> job_1406011031680_0002
>>> Examining task ID: task_1406011031680_0002_m_000005 (and more) from job
>>> job_1406011031680_0002
>>>
>>> Task with the most failures(4):
>>> -----
>>> Task ID:
>>> task_1406011031680_0002_m_000008
>>>
>>> URL:
>>>
>>> http://ec2-54-77-76-145.eu-west-1.compute.amazonaws.com:8088/taskdetails.jsp?jobid=job_1406011031680_0002&tipid=task_1406011031680_0002_m_000008
>>> -----
>>> Diagnostic Messages for this Task:
>>> Exception from container-launch:
>>> org.apache.hadoop.util.Shell$ExitCodeException:
>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
>>> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>>> at
>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:744)
>>>
>>>
>>> FAILED: Execution Error, return code 2 from
>>> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
>>> MapReduce Jobs Launched:
>>> Job 0: Map: 3 Reduce: 1 Cumulative CPU: 24.58 sec HDFS Read:
>>> 593821601 HDFS Write: 14518009 SUCCESS
>>> Job 1: Map: 9 Cumulative CPU: 37.19 sec HDFS Read: 1342219615 HDFS
>>> Write: 821879 FAIL
>>> Total MapReduce CPU Time Spent: 1 minutes 1 seconds 770 msec
>>> hive (default)> exit;
>>>
>>>
>>>
>>> Regards,
>>> Sarfraz Rasheed Ramay (DIT)
>>> Dublin, Ireland.
>>>
>>
>>
Re: Hive shell code exception, urgent help needed
Posted by Juan Martin Pampliega <jp...@gmail.com>.
Hi,
The actual useful part of the error is:
Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask
If you do a search for this plus "EC2" in Google you will find a couple of
results that point to memory exhaustion issues. You should try increasing
the configurated memory size.
Since you are using a t2.micro you should really try using a bigger Amazon
instance size. This might probably be a lot more useful than trying
different configurations.
On Jul 24, 2014 7:08 AM, "Sarfraz Ramay" <sa...@gmail.com> wrote:
> Can anyone please help with this ?
>
> [image: Inline image 1]
>
>
> i followed the advice here
> http://stackoverflow.com/questions/20390217/mapreduce-job-in-headless-environment-fails-n-times-due-to-am-container-exceptio
>
> and added to mapred-site.xml following properties but still getting the
> same error.
>
> <property>
> <name>mapred.child.java.opts</name>
> <value>-Djava.awt.headless=true</value></property><!-- add headless to default -Xmx1024m --><property>
> <name>yarn.app.mapreduce.am.command-opts</name>
> <value>-Djava.awt.headless=true -Xmx1024m</value></property><property>
> <name>yarn.app.mapreduce.am.admin-command-opts</name>
> <value>-Djava.awt.headless=true</value></property
>
>
>
> Regards,
> Sarfraz Rasheed Ramay (DIT)
> Dublin, Ireland.
>
>
> On Tue, Jul 22, 2014 at 8:19 AM, Sarfraz Ramay <sa...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am using Hive 0.13.1 and Hadoop 2-2.0 on amazon EC2 t2.micro instances.
>> I have 4 instances, master has the namenode and yarn, secondarynode is a
>> separate instance and two slaves are on separate instances each.
>>
>> It was working fine till now but it started to break when i tried to run
>> the following query on tpch generated 3GB data. same worked ok on 1GB
>>
>> SELECT
>> l_orderkey
>> , sum(l_extendedprice*(1-l_discount)) as revenue
>> , o_orderdate
>> , o_shippriority
>> FROM
>> customer c JOIN orders o
>> ON (c.c_custkey = o.o_custkey)
>> JOIN lineitem l
>> on (l.l_orderkey = o.o_orderkey)
>> WHERE
>> o_orderdate < '1995-03-15' and l_shipdate > '1995-03-15'
>> AND c.c_mktsegment = 'AUTOMOBILE'
>> GROUP BY
>> l_orderkey, o_orderdate, o_shippriority
>> HAVING
>> sum(l_extendedprice*(1-l_discount)) > 38500 --average revenue
>> --LIMIT 10;
>>
>> i have tried many things but nothing seems to work. I am also attaching
>> my mapred-site.xml and yarn-site.xml files for reference plus the error
>> log. I have also tried to limit the memory settings in mapred-site.xml and
>> yarn-site but nothing seems to be working. For full log details please find
>> attached hive.log file. Please help!
>>
>> Hadoop job information for Stage-7: number of mappers: 9; number of
>> reducers: 0
>> 2014-07-22 06:39:31,643 Stage-7 map = 0%, reduce = 0%
>> 2014-07-22 06:39:43,940 Stage-7 map = 6%, reduce = 0%, Cumulative CPU
>> 5.34 sec
>> 2014-07-22 06:39:45,002 Stage-7 map = 11%, reduce = 0%, Cumulative CPU
>> 6.94 sec
>> 2014-07-22 06:40:08,373 Stage-7 map = 17%, reduce = 0%, Cumulative CPU
>> 12.6 sec
>> 2014-07-22 06:40:10,417 Stage-7 map = 22%, reduce = 0%, Cumulative CPU
>> 14.06 sec
>> 2014-07-22 06:40:22,732 Stage-7 map = 28%, reduce = 0%, Cumulative CPU
>> 24.46 sec
>> 2014-07-22 06:40:25,843 Stage-7 map = 33%, reduce = 0%, Cumulative CPU
>> 25.74 sec
>> 2014-07-22 06:40:33,039 Stage-7 map = 44%, reduce = 0%, Cumulative CPU
>> 33.32 sec
>> 2014-07-22 06:40:38,709 Stage-7 map = 56%, reduce = 0%, Cumulative CPU
>> 37.19 sec
>> 2014-07-22 06:41:07,648 Stage-7 map = 61%, reduce = 0%, Cumulative CPU
>> 42.83 sec
>> 2014-07-22 06:41:15,900 Stage-7 map = 56%, reduce = 0%, Cumulative CPU
>> 39.49 sec
>> 2014-07-22 06:41:27,299 Stage-7 map = 67%, reduce = 0%, Cumulative CPU
>> 46.07 sec
>> 2014-07-22 06:41:28,342 Stage-7 map = 56%, reduce = 0%, Cumulative CPU
>> 40.9 sec
>> 2014-07-22 06:41:43,753 Stage-7 map = 61%, reduce = 0%, Cumulative CPU
>> 42.84 sec
>> 2014-07-22 06:41:45,801 Stage-7 map = 100%, reduce = 0%, Cumulative CPU
>> 37.19 sec
>> MapReduce Total cumulative CPU time: 37 seconds 190 msec
>> Ended Job = job_1406011031680_0002 with errors
>> Error during job, obtaining debugging information...
>> Job Tracking URL:
>> http://ec2-54-77-76-145.eu-west-1.compute.amazonaws.com:8088/proxy/application_1406011031680_0002/
>> Examining task ID: task_1406011031680_0002_m_000001 (and more) from job
>> job_1406011031680_0002
>> Examining task ID: task_1406011031680_0002_m_000005 (and more) from job
>> job_1406011031680_0002
>>
>> Task with the most failures(4):
>> -----
>> Task ID:
>> task_1406011031680_0002_m_000008
>>
>> URL:
>>
>> http://ec2-54-77-76-145.eu-west-1.compute.amazonaws.com:8088/taskdetails.jsp?jobid=job_1406011031680_0002&tipid=task_1406011031680_0002_m_000008
>> -----
>> Diagnostic Messages for this Task:
>> Exception from container-launch:
>> org.apache.hadoop.util.Shell$ExitCodeException:
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
>> at org.apache.hadoop.util.Shell.run(Shell.java:379)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>>
>>
>> FAILED: Execution Error, return code 2 from
>> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
>> MapReduce Jobs Launched:
>> Job 0: Map: 3 Reduce: 1 Cumulative CPU: 24.58 sec HDFS Read:
>> 593821601 HDFS Write: 14518009 SUCCESS
>> Job 1: Map: 9 Cumulative CPU: 37.19 sec HDFS Read: 1342219615 HDFS
>> Write: 821879 FAIL
>> Total MapReduce CPU Time Spent: 1 minutes 1 seconds 770 msec
>> hive (default)> exit;
>>
>>
>>
>> Regards,
>> Sarfraz Rasheed Ramay (DIT)
>> Dublin, Ireland.
>>
>
>
Re: Hive shell code exception, urgent help needed
Posted by Sarfraz Ramay <sa...@gmail.com>.
Can anyone please help with this ?
[image: Inline image 1]
i followed the advice here
http://stackoverflow.com/questions/20390217/mapreduce-job-in-headless-environment-fails-n-times-due-to-am-container-exceptio
and added to mapred-site.xml following properties but still getting the
same error.
<property>
<name>mapred.child.java.opts</name>
<value>-Djava.awt.headless=true</value></property><!-- add
headless to default -Xmx1024m --><property>
<name>yarn.app.mapreduce.am.command-opts</name>
<value>-Djava.awt.headless=true -Xmx1024m</value></property><property>
<name>yarn.app.mapreduce.am.admin-command-opts</name>
<value>-Djava.awt.headless=true</value></property
Regards,
Sarfraz Rasheed Ramay (DIT)
Dublin, Ireland.
On Tue, Jul 22, 2014 at 8:19 AM, Sarfraz Ramay <sa...@gmail.com>
wrote:
> Hi,
>
> I am using Hive 0.13.1 and Hadoop 2-2.0 on amazon EC2 t2.micro instances.
> I have 4 instances, master has the namenode and yarn, secondarynode is a
> separate instance and two slaves are on separate instances each.
>
> It was working fine till now but it started to break when i tried to run
> the following query on tpch generated 3GB data. same worked ok on 1GB
>
> SELECT
> l_orderkey
> , sum(l_extendedprice*(1-l_discount)) as revenue
> , o_orderdate
> , o_shippriority
> FROM
> customer c JOIN orders o
> ON (c.c_custkey = o.o_custkey)
> JOIN lineitem l
> on (l.l_orderkey = o.o_orderkey)
> WHERE
> o_orderdate < '1995-03-15' and l_shipdate > '1995-03-15'
> AND c.c_mktsegment = 'AUTOMOBILE'
> GROUP BY
> l_orderkey, o_orderdate, o_shippriority
> HAVING
> sum(l_extendedprice*(1-l_discount)) > 38500 --average revenue
> --LIMIT 10;
>
> i have tried many things but nothing seems to work. I am also attaching my
> mapred-site.xml and yarn-site.xml files for reference plus the error log. I
> have also tried to limit the memory settings in mapred-site.xml and
> yarn-site but nothing seems to be working. For full log details please find
> attached hive.log file. Please help!
>
> Hadoop job information for Stage-7: number of mappers: 9; number of
> reducers: 0
> 2014-07-22 06:39:31,643 Stage-7 map = 0%, reduce = 0%
> 2014-07-22 06:39:43,940 Stage-7 map = 6%, reduce = 0%, Cumulative CPU
> 5.34 sec
> 2014-07-22 06:39:45,002 Stage-7 map = 11%, reduce = 0%, Cumulative CPU
> 6.94 sec
> 2014-07-22 06:40:08,373 Stage-7 map = 17%, reduce = 0%, Cumulative CPU
> 12.6 sec
> 2014-07-22 06:40:10,417 Stage-7 map = 22%, reduce = 0%, Cumulative CPU
> 14.06 sec
> 2014-07-22 06:40:22,732 Stage-7 map = 28%, reduce = 0%, Cumulative CPU
> 24.46 sec
> 2014-07-22 06:40:25,843 Stage-7 map = 33%, reduce = 0%, Cumulative CPU
> 25.74 sec
> 2014-07-22 06:40:33,039 Stage-7 map = 44%, reduce = 0%, Cumulative CPU
> 33.32 sec
> 2014-07-22 06:40:38,709 Stage-7 map = 56%, reduce = 0%, Cumulative CPU
> 37.19 sec
> 2014-07-22 06:41:07,648 Stage-7 map = 61%, reduce = 0%, Cumulative CPU
> 42.83 sec
> 2014-07-22 06:41:15,900 Stage-7 map = 56%, reduce = 0%, Cumulative CPU
> 39.49 sec
> 2014-07-22 06:41:27,299 Stage-7 map = 67%, reduce = 0%, Cumulative CPU
> 46.07 sec
> 2014-07-22 06:41:28,342 Stage-7 map = 56%, reduce = 0%, Cumulative CPU
> 40.9 sec
> 2014-07-22 06:41:43,753 Stage-7 map = 61%, reduce = 0%, Cumulative CPU
> 42.84 sec
> 2014-07-22 06:41:45,801 Stage-7 map = 100%, reduce = 0%, Cumulative CPU
> 37.19 sec
> MapReduce Total cumulative CPU time: 37 seconds 190 msec
> Ended Job = job_1406011031680_0002 with errors
> Error during job, obtaining debugging information...
> Job Tracking URL:
> http://ec2-54-77-76-145.eu-west-1.compute.amazonaws.com:8088/proxy/application_1406011031680_0002/
> Examining task ID: task_1406011031680_0002_m_000001 (and more) from job
> job_1406011031680_0002
> Examining task ID: task_1406011031680_0002_m_000005 (and more) from job
> job_1406011031680_0002
>
> Task with the most failures(4):
> -----
> Task ID:
> task_1406011031680_0002_m_000008
>
> URL:
>
> http://ec2-54-77-76-145.eu-west-1.compute.amazonaws.com:8088/taskdetails.jsp?jobid=job_1406011031680_0002&tipid=task_1406011031680_0002_m_000008
> -----
> Diagnostic Messages for this Task:
> Exception from container-launch:
> org.apache.hadoop.util.Shell$ExitCodeException:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
> at org.apache.hadoop.util.Shell.run(Shell.java:379)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
>
>
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> MapReduce Jobs Launched:
> Job 0: Map: 3 Reduce: 1 Cumulative CPU: 24.58 sec HDFS Read:
> 593821601 HDFS Write: 14518009 SUCCESS
> Job 1: Map: 9 Cumulative CPU: 37.19 sec HDFS Read: 1342219615 HDFS
> Write: 821879 FAIL
> Total MapReduce CPU Time Spent: 1 minutes 1 seconds 770 msec
> hive (default)> exit;
>
>
>
> Regards,
> Sarfraz Rasheed Ramay (DIT)
> Dublin, Ireland.
>