You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Elaine Gan <el...@gmo.jp> on 2012/12/12 10:43:31 UTC
help on failed MR jobs (big hive files)
Hi,
I'm trying to run a program on Hadoop.
[Input] tsv file
My program does the following.
(1) Load tsv into hive
load data local inpath 'tsvfile' overwrite into table A partitioned by xx
(2) insert overwrite table B select a, b, c from table A where datediff(to_date(from_unixtime(unix_timestamp('${logdate}'))), request_date) <= 30
(3) Running Mahout
In step 2, i am trying to retrieve data from hive for the past month.
My hadoop work always stopped here.
When i check through my browser utility it says that
Diagnostic Info:
# of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201211291541_0262_m_001800
Task attempt_201211291541_0262_m_001800_0 failed to report status for 1802 seconds. Killing!
Error: Java heap space
Task attempt_201211291541_0262_m_001800_2 failed to report status for 1800 seconds. Killing!
Task attempt_201211291541_0262_m_001800_3 failed to report status for 1801 seconds. Killing!
Each hive table is big, around 6 GB.
(1) Is it too big to have around 6GB for each hive table?
(2) I've increased by HEAPSIZE to 50G,which i think is far more than enough. Any else
where i can do the tuning?
Thank you.
rei
Re: help on failed MR jobs (big hive files)
Posted by Elaine Gan <el...@gmo.jp>.
Dear Nitin and Mark,
Thanks for the prompt response.
(1) optimizing the query
I've tried using function in my where clause and just specify the date
range. It still doesnt work, so i guess i need to check around.
But still, this is a good advice. Will keep this in mind.
(2) heapsize
> > by the heapsize increase did you mean increase hive heapsize or hadoop
> > mapred heapsize ? You will need to increase the heapsize on mapred by
> > setting the property
> > set mapred.job.map.memory.mb=6000;
> > set mapred.job.reduce.memory.mb=4000;
I increase the hadoop heapsize, the one defined hadoop-env.sh
export HADOOP_HEAPSIZE=50000
I see, i will increase the size on the mapred.
(3) Partitioning
My datas are patitioned by request_date.
Each partition has around 6G of data, which means for one month it'll
have around 180G.
I have 10 datanodes, which means the total input are 180x10=1800G=1.8TB
(4) split size
> > If you got good size hadoop cluster then lower the split size and launch
> > many maps that way it will get executed quickly
My split size is 128MB. Is this consider big? I guess it's small enought?
And i have mapred.tasktracker.map.tasks.maximum=4.
Guess i will need to optimize this part.
I will update again on my result.
Thank you.
> Elaine,
> Nitin raises some good points.
>
> Continuing on the same lines, let's take a closer look at the query:
> insert overwrite table B select a, b, c from table A where
> datediff(to_date(from_unixtime(unix_timestamp('${logdate}'))),
> request_date) <= 30
>
> In the above query,
> "datediff(to_date(from_unixtime(unix_timestamp('${logdate}'))),
> request_date)" would cause this set of nested functions to be
> evaluated for every record in your 6 GB dataset on the server. It
> would be best if this computation was done in your client (bash script
> or java code issuing hive queries) so that the query that gets sent to
> the Hive server looks like: "request_date >= '2012-01-01' and
> request_date < '2012-06-01'"
>
> That would shave off a lot of time. If the performance is still poor,
> consider partitioning your data (based on date?), also make sure you
> don't suffer from the small file problem:
> http://www.cloudera.com/blog/2009/02/the-small-files-problem/
>
> Good luck!
> Mark
>
>
> On Wed, Dec 12, 2012 at 11:36 PM, Nitin Pawar <ni...@gmail.com> wrote:
> > 6GB size is nothing. We have done it with few TB of data in hive.
> > Error you are seeing is on the hadoop side.
> >
> > You can always optimize your query based on the hadoop compute capacity you
> > have got and also based on the pattern in the data you will need to design
> > your schema.
> >
> > The problem here can be you have got a fucntion to execute in the where
> > clause. Can you try hard coding them to data range and see if you can get
> > any improvements.
> >
> > Alternatively if you can partition your data on date basis, smaller dataset
> > you will have to read.
> >
> > If you got good size hadoop cluster then lower the split size and launch
> > many maps that way it will get executed quickly
> >
> > by the heapsize increase did you mean increase hive heapsize or hadoop
> > mapred heapsize ? You will need to increase the heapsize on mapred by
> > setting the property
> > set mapred.job.map.memory.mb=6000;
> > set mapred.job.reduce.memory.mb=4000;
> >
> >
> >
> > On Wed, Dec 12, 2012 at 3:13 PM, Elaine Gan <el...@gmo.jp> wrote:
> >>
> >> Hi,
> >>
> >> I'm trying to run a program on Hadoop.
> >>
> >> [Input] tsv file
> >>
> >> My program does the following.
> >> (1) Load tsv into hive
> >> load data local inpath 'tsvfile' overwrite into table A partitioned
> >> by xx
> >> (2) insert overwrite table B select a, b, c from table A where
> >> datediff(to_date(from_unixtime(unix_timestamp('${logdate}'))), request_date)
> >> <= 30
> >> (3) Running Mahout
> >>
> >> In step 2, i am trying to retrieve data from hive for the past month.
> >> My hadoop work always stopped here.
> >> When i check through my browser utility it says that
> >>
> >> Diagnostic Info:
> >> # of failed Map Tasks exceeded allowed limit. FailedCount: 1.
> >> LastFailedTask: task_201211291541_0262_m_001800
> >>
> >> Task attempt_201211291541_0262_m_001800_0 failed to report status for 1802
> >> seconds. Killing!
> >> Error: Java heap space
> >> Task attempt_201211291541_0262_m_001800_2 failed to report status for 1800
> >> seconds. Killing!
> >> Task attempt_201211291541_0262_m_001800_3 failed to report status for 1801
> >> seconds. Killing!
> >>
> >>
> >>
> >> Each hive table is big, around 6 GB.
> >>
> >> (1) Is it too big to have around 6GB for each hive table?
> >> (2) I've increased by HEAPSIZE to 50G,which i think is far more than
> >> enough. Any else
> >> where i can do the tuning?
> >>
> >>
> >> Thank you.
> >>
> >>
> >>
> >> rei
> >>
> >>
> >
> >
> >
> > --
> > Nitin Pawar
Re: help on failed MR jobs (big hive files)
Posted by Mark Grover <gr...@gmail.com>.
Elaine,
Nitin raises some good points.
Continuing on the same lines, let's take a closer look at the query:
insert overwrite table B select a, b, c from table A where
datediff(to_date(from_unixtime(unix_timestamp('${logdate}'))),
request_date) <= 30
In the above query,
"datediff(to_date(from_unixtime(unix_timestamp('${logdate}'))),
request_date)" would cause this set of nested functions to be
evaluated for every record in your 6 GB dataset on the server. It
would be best if this computation was done in your client (bash script
or java code issuing hive queries) so that the query that gets sent to
the Hive server looks like: "request_date >= '2012-01-01' and
request_date < '2012-06-01'"
That would shave off a lot of time. If the performance is still poor,
consider partitioning your data (based on date?), also make sure you
don't suffer from the small file problem:
http://www.cloudera.com/blog/2009/02/the-small-files-problem/
Good luck!
Mark
On Wed, Dec 12, 2012 at 11:36 PM, Nitin Pawar <ni...@gmail.com> wrote:
> 6GB size is nothing. We have done it with few TB of data in hive.
> Error you are seeing is on the hadoop side.
>
> You can always optimize your query based on the hadoop compute capacity you
> have got and also based on the pattern in the data you will need to design
> your schema.
>
> The problem here can be you have got a fucntion to execute in the where
> clause. Can you try hard coding them to data range and see if you can get
> any improvements.
>
> Alternatively if you can partition your data on date basis, smaller dataset
> you will have to read.
>
> If you got good size hadoop cluster then lower the split size and launch
> many maps that way it will get executed quickly
>
> by the heapsize increase did you mean increase hive heapsize or hadoop
> mapred heapsize ? You will need to increase the heapsize on mapred by
> setting the property
> set mapred.job.map.memory.mb=6000;
> set mapred.job.reduce.memory.mb=4000;
>
>
>
> On Wed, Dec 12, 2012 at 3:13 PM, Elaine Gan <el...@gmo.jp> wrote:
>>
>> Hi,
>>
>> I'm trying to run a program on Hadoop.
>>
>> [Input] tsv file
>>
>> My program does the following.
>> (1) Load tsv into hive
>> load data local inpath 'tsvfile' overwrite into table A partitioned
>> by xx
>> (2) insert overwrite table B select a, b, c from table A where
>> datediff(to_date(from_unixtime(unix_timestamp('${logdate}'))), request_date)
>> <= 30
>> (3) Running Mahout
>>
>> In step 2, i am trying to retrieve data from hive for the past month.
>> My hadoop work always stopped here.
>> When i check through my browser utility it says that
>>
>> Diagnostic Info:
>> # of failed Map Tasks exceeded allowed limit. FailedCount: 1.
>> LastFailedTask: task_201211291541_0262_m_001800
>>
>> Task attempt_201211291541_0262_m_001800_0 failed to report status for 1802
>> seconds. Killing!
>> Error: Java heap space
>> Task attempt_201211291541_0262_m_001800_2 failed to report status for 1800
>> seconds. Killing!
>> Task attempt_201211291541_0262_m_001800_3 failed to report status for 1801
>> seconds. Killing!
>>
>>
>>
>> Each hive table is big, around 6 GB.
>>
>> (1) Is it too big to have around 6GB for each hive table?
>> (2) I've increased by HEAPSIZE to 50G,which i think is far more than
>> enough. Any else
>> where i can do the tuning?
>>
>>
>> Thank you.
>>
>>
>>
>> rei
>>
>>
>
>
>
> --
> Nitin Pawar
Re: help on failed MR jobs (big hive files)
Posted by Nitin Pawar <ni...@gmail.com>.
6GB size is nothing. We have done it with few TB of data in hive.
Error you are seeing is on the hadoop side.
You can always optimize your query based on the hadoop compute capacity you
have got and also based on the pattern in the data you will need to design
your schema.
The problem here can be you have got a fucntion to execute in the where
clause. Can you try hard coding them to data range and see if you can get
any improvements.
Alternatively if you can partition your data on date basis, smaller dataset
you will have to read.
If you got good size hadoop cluster then lower the split size and launch
many maps that way it will get executed quickly
by the heapsize increase did you mean increase hive heapsize or hadoop
mapred heapsize ? You will need to increase the heapsize on mapred by
setting the property
set mapred.job.map.memory.mb=6000;
set mapred.job.reduce.memory.mb=4000;
On Wed, Dec 12, 2012 at 3:13 PM, Elaine Gan <el...@gmo.jp> wrote:
> Hi,
>
> I'm trying to run a program on Hadoop.
>
> [Input] tsv file
>
> My program does the following.
> (1) Load tsv into hive
> load data local inpath 'tsvfile' overwrite into table A partitioned
> by xx
> (2) insert overwrite table B select a, b, c from table A where
> datediff(to_date(from_unixtime(unix_timestamp('${logdate}'))),
> request_date) <= 30
> (3) Running Mahout
>
> In step 2, i am trying to retrieve data from hive for the past month.
> My hadoop work always stopped here.
> When i check through my browser utility it says that
>
> Diagnostic Info:
> # of failed Map Tasks exceeded allowed limit. FailedCount: 1.
> LastFailedTask: task_201211291541_0262_m_001800
>
> Task attempt_201211291541_0262_m_001800_0 failed to report status for 1802
> seconds. Killing!
> Error: Java heap space
> Task attempt_201211291541_0262_m_001800_2 failed to report status for 1800
> seconds. Killing!
> Task attempt_201211291541_0262_m_001800_3 failed to report status for 1801
> seconds. Killing!
>
>
>
> Each hive table is big, around 6 GB.
>
> (1) Is it too big to have around 6GB for each hive table?
> (2) I've increased by HEAPSIZE to 50G,which i think is far more than
> enough. Any else
> where i can do the tuning?
>
>
> Thank you.
>
>
>
> rei
>
>
>
--
Nitin Pawar