You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Yang <te...@gmail.com> on 2014/07/17 20:28:13 UTC
how to control hive log location on 0.13?
we just moved to hadoop2.0 (HDP2.1 distro). it turns out that the new hive
version generates a lot of logs into /tmp/ and is quickly creating the
danger of running out of our /tmp/ space.
I see these 2 different logs :
[myuser@mybox ~]$ ls -lt /tmp/myuser/
total 1988
-rw-rw-r-- 1 myuser myuser 191687 2014-07-17 11:17 hive.log
-rw-rw-r-- 1 myuser myuser 14472 2014-07-16 14:43
myuser_20140716143232_d76043ed-1c4b-42a0-bf0a-2816377a6a2a.log
-rw-rw-r-- 1 myuser myuser 14260 2014-07-16 14:04
myuser_20140716135353_de698da0-807f-4e3b-8b97-5af5064b55f2.log
-rw-rw-r-- 1 myuser myuser 14254 2014-07-16 13:42
myuser_20140716133838_208329bd-77bb-4981-a2e9-e747647d0704.log
from the doc at
https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-ErrorLogs
I can see that "per Hive session basis in /tmp/<user.name>/, but can be
configured in hive-site.xml
<https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration>
with
the hive.querylog.location property.",
but I tried to pass it to -hiveconf hive.querylog.location=/tmp/mycustomlogdir/
, doesn't seem to work; the hive.log location is not changed by this
approach either.
so how can I change the location of both the logs , by some per-script
params ? (i.e. we can't afford to change the system hive-site.xml or
/etc/hive/conf etc)
Thanks a lot
Yang
Re: how to control hive log location on 0.13?
Posted by Lefty Leverenz <le...@gmail.com>.
Thanks, Satish Mittal, I've added that information to the Error Logs section
<https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-ErrorLogs>
of the Getting Started wiki.
-- Lefty
On Fri, Jul 18, 2014 at 12:19 AM, Satish Mittal <sa...@inmobi.com>
wrote:
> You can configure the following property in
> $HIVE_HOME/conf/hive-log4j.properties:
>
> hive.log.dir=<your location>
>
> The default value of this property is ${java.io.tmpdir}/${user.name}.
>
> Thanks,
> Satish
>
>
> On Thu, Jul 17, 2014 at 11:58 PM, Yang <te...@gmail.com> wrote:
>
>> we just moved to hadoop2.0 (HDP2.1 distro). it turns out that the new
>> hive version generates a lot of logs into /tmp/ and is quickly creating the
>> danger of running out of our /tmp/ space.
>>
>>
>> I see these 2 different logs :
>>
>> [myuser@mybox ~]$ ls -lt /tmp/myuser/
>> total 1988
>> -rw-rw-r-- 1 myuser myuser 191687 2014-07-17 11:17 hive.log
>> -rw-rw-r-- 1 myuser myuser 14472 2014-07-16 14:43
>> myuser_20140716143232_d76043ed-1c4b-42a0-bf0a-2816377a6a2a.log
>> -rw-rw-r-- 1 myuser myuser 14260 2014-07-16 14:04
>> myuser_20140716135353_de698da0-807f-4e3b-8b97-5af5064b55f2.log
>> -rw-rw-r-- 1 myuser myuser 14254 2014-07-16 13:42
>> myuser_20140716133838_208329bd-77bb-4981-a2e9-e747647d0704.log
>>
>>
>>
>> from the doc at
>> https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-ErrorLogs
>> I can see that "per Hive session basis in /tmp/<user.name>/, but can be
>> configured in hive-site.xml
>> <https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration> with
>> the hive.querylog.location property.",
>> but I tried to pass it to -hiveconf hive.querylog.location=/tmp/mycustomlogdir/
>> , doesn't seem to work; the hive.log location is not changed by this
>> approach either.
>>
>> so how can I change the location of both the logs , by some per-script
>> params ? (i.e. we can't afford to change the system hive-site.xml or
>> /etc/hive/conf etc)
>>
>> Thanks a lot
>> Yang
>>
>
>
> _____________________________________________________________
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
Re: how to control hive log location on 0.13?
Posted by Lefty Leverenz <le...@gmail.com>.
Thanks André, I've added the sticky bit advice to Error Logs
<https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-ErrorLogs>
.
-- Lefty
On Fri, Jul 18, 2014 at 2:38 PM, Yang <te...@gmail.com> wrote:
> thanks guys. anybody knows what generates the log like "
> myuser_20140716143232_d76043ed-1c4b-42a0-bf0a-2816377a6a2a.log" ? I
> checked our application code, it doesn't generate this, looks from hive.
>
>
> On Fri, Jul 18, 2014 at 12:28 AM, Andre Araujo <ar...@pythian.com> wrote:
>
>> Make sure the directory you specify has the sticky bit set, otherwise
>> users will have permission problems:
>>
>> chmod 1777 <dir>
>>
>>
>> On 18 July 2014 14:19, Satish Mittal <sa...@inmobi.com> wrote:
>>
>>> You can configure the following property in
>>> $HIVE_HOME/conf/hive-log4j.properties:
>>>
>>> hive.log.dir=<your location>
>>>
>>> The default value of this property is ${java.io.tmpdir}/${user.name}.
>>>
>>> Thanks,
>>> Satish
>>>
>>>
>>> On Thu, Jul 17, 2014 at 11:58 PM, Yang <te...@gmail.com> wrote:
>>>
>>>> we just moved to hadoop2.0 (HDP2.1 distro). it turns out that the new
>>>> hive version generates a lot of logs into /tmp/ and is quickly creating the
>>>> danger of running out of our /tmp/ space.
>>>>
>>>>
>>>> I see these 2 different logs :
>>>>
>>>> [myuser@mybox ~]$ ls -lt /tmp/myuser/
>>>> total 1988
>>>> -rw-rw-r-- 1 myuser myuser 191687 2014-07-17 11:17 hive.log
>>>> -rw-rw-r-- 1 myuser myuser 14472 2014-07-16 14:43
>>>> myuser_20140716143232_d76043ed-1c4b-42a0-bf0a-2816377a6a2a.log
>>>> -rw-rw-r-- 1 myuser myuser 14260 2014-07-16 14:04
>>>> myuser_20140716135353_de698da0-807f-4e3b-8b97-5af5064b55f2.log
>>>> -rw-rw-r-- 1 myuser myuser 14254 2014-07-16 13:42
>>>> myuser_20140716133838_208329bd-77bb-4981-a2e9-e747647d0704.log
>>>>
>>>>
>>>>
>>>> from the doc at
>>>> https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-ErrorLogs
>>>> I can see that "per Hive session basis in /tmp/<user.name>/, but can
>>>> be configured in hive-site.xml
>>>> <https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration> with
>>>> the hive.querylog.location property.",
>>>> but I tried to pass it to -hiveconf hive.querylog.location=/tmp/mycustomlogdir/
>>>> , doesn't seem to work; the hive.log location is not changed by this
>>>> approach either.
>>>>
>>>> so how can I change the location of both the logs , by some per-script
>>>> params ? (i.e. we can't afford to change the system hive-site.xml or
>>>> /etc/hive/conf etc)
>>>>
>>>> Thanks a lot
>>>> Yang
>>>>
>>>
>>>
>>> _____________________________________________________________
>>> The information contained in this communication is intended solely for
>>> the use of the individual or entity to whom it is addressed and others
>>> authorized to receive it. It may contain confidential or legally privileged
>>> information. If you are not the intended recipient you are hereby notified
>>> that any disclosure, copying, distribution or taking any action in reliance
>>> on the contents of this information is strictly prohibited and may be
>>> unlawful. If you have received this communication in error, please notify
>>> us immediately by responding to this email and then delete it from your
>>> system. The firm is neither liable for the proper and complete transmission
>>> of the information contained in this communication nor for any delay in its
>>> receipt.
>>
>>
>>
>>
>> --
>> André Araújo
>> Big Data Consultant/Solutions Architect
>> The Pythian Group - Australia - www.pythian.com
>>
>> Office (calls from within Australia): 1300 366 021 x1270
>> Office (international): +61 2 8016 7000 x270 *OR* +1 613 565 8696
>> x1270
>> Mobile: +61 410 323 559
>> Fax: +61 2 9805 0544
>> IM: pythianaraujo @ AIM/MSN/Y! or araujo@pythian.com @ GTalk
>>
>> “Success is not about standing at the top, it's the steps you leave
>> behind.” — Iker Pou (rock climber)
>>
>> --
>>
>>
>>
>>
>
Re: how to control hive log location on 0.13?
Posted by Andre Araujo <ar...@pythian.com>.
Hi, Yang,
you're running your mapreduce jobs in Hadoop's local mode, and in that mode
all the Hive MR logging is handled through log4j on your local machine,
which is what this log file is about.
The log location and naming is controlled by the property
log4j.appender.FA.File in the Hive log4j properties file.
Cheers,
Andre
On 19 July 2014 15:24, Yang <te...@gmail.com> wrote:
> 2014-07-18 15:03:37,774 INFO mr.ExecDriver
> (SessionState.java:printInfo(537)) - Execution log at:
> /tmp/myuser/myuser_2014071815030
> 3_56bf6bb0-db30-4dbc-807c-9023ce4103f4.log
> 2014-07-18 15:03:37,864 WARN conf.Configuration
> (Configuration.java:loadProperty(2358)) -
> file:/tmp/myuser/hive_2014-07-18_15-03-30_423_
> 6799963466906099923-1/-local-10011/jobconf.xml:an attempt to override
> final parameter: mapreduce.job.end-notification.max.retry.interval; Igno
> ring.
> 2014-07-18 15:03:37,871 WARN conf.Configuration
> (Configuration.java:loadProperty(2358)) -
> file:/tmp/myuser/hive_2014-07-18_15-03-30_423_
> 6799963466906099923-1/-local-10011/jobconf.xml:an attempt to override
> final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
> 2014-07-18 15:03:37,951 INFO log.PerfLogger
> (PerfLogger.java:PerfLogBegin(108)) - <PERFLOG method=deserializePlan
> from=org.apache.hadoop.hive.
> ql.exec.Utilities>
> 2014-07-18 15:03:37,951 INFO exec.Utilities
> (Utilities.java:deserializePlan(822)) - Deserializing MapredLocalWork via
> kryo
> 2014-07-18 15:03:38,237 INFO log.PerfLogger
> (PerfLogger.java:PerfLogEnd(135)) - </PERFLOG method=deserializePlan
> start=1405721017951 end=14057
> 21018237 duration=286 from=org.apache.hadoop.hive.ql.exec.Utilities>
> 2014-07-18 15:03:38,246 INFO mr.MapredLocalTask
> (SessionState.java:printInfo(537)) - 2014-07-18 03:03:38 Starting to
> launch local task t
> o process map join; maximum memory = 4261937152
> 2014-07-18 15:03:38,261 INFO mr.MapredLocalTask
> (MapredLocalTask.java:initializeOperators(406)) - fetchoperator for
> null-subquery2:a-subquery2
> :dpkg_cntr:dpkg_wtransaction_p2_id_user_30m created
> 2014-07-18 15:03:38,263 INFO mr.MapredLocalTask
> (MapredLocalTask.java:initializeOperators(406)) - fetchoperator for
> null-subquery2:a-subquery2
> :dpkg:dpkg_wtransaction_p2_id_user_30m created
> 2014-07-18 15:03:38,264 INFO mr.MapredLocalTask
> (MapredLocalTask.java:initializeOperators(406)) - fetchoperator for
> null-subquery2:a-subquery2
> :xclick:b:wtrans_data_map_p2_30m created
> 2014-07-18 15:03:38,266 INFO mr.MapredLocalTask
> (MapredLocalTask.java:initializeOperators(406)) - fetchoperator for
> null-subquery1:a-subquery1
> :dpkg_cntr:dpkg_wtransaction_id_user_30m created
> 2014-07-18 15:03:38,268 INFO mr.MapredLocalTask
> (MapredLocalTask.java:initializeOperators(406)) - fetchoperator for
> null-subquery1:a-subquery1
> :dpkg:dpkg_wtransaction_id_user_30m created
> 2014-07-18 15:03:38,269 INFO mr.MapredLocalTask
> (MapredLocalTask.java:initializeOperators(406)) - fetchoperator for
> null-subquery1:a-subquery1
> :xclick:b:wtrans_data_map_30m created
> ---------------------------------------------
>
> whole bunch of stuff omitted here
>
> --------------------------------------
>
>
> --------------------------------------
> 2014-07-18 15:04:08,678 INFO exec.HashTableSinkOperator
> (HashTableSinkOperator.java:flushToFile(278)) - Temp URI for side table:
> file:/tmp/myuser/hive_2014-07-18_15-03-30_423_6799963466906099923-1/-local-10008/HashTable-Stage-2
> 2014-07-18 15:04:08,678 INFO exec.HashTableSinkOperator
> (SessionState.java:printInfo(537)) - 2014-07-18 03:04:08 Dump the
> side-table into file:
> file:/tmp/myuser/hive_2014-07-18_15-03-30_423_6799963466906099923-1/-local-10008/HashTable-Stage-2/MapJoin-mapfile11--.hashtable
> 2014-07-18 15:04:09,943 INFO exec.HashTableSinkOperator
> (SessionState.java:printInfo(537)) - 2014-07-18 03:04:09 Uploaded 1
> File to:
> file:/tmp/myuser/hive_2014-07-18_15-03-30_423_6799963466906099923-1/-local-10008/HashTable-Stage-2/MapJoin-mapfile11--.hashtable
> (58010217 bytes)
> 2014-07-18 15:04:09,943 INFO exec.HashTableSinkOperator
> (Operator.java:close(591)) - 6 Close done
> 2014-07-18 15:04:09,943 INFO exec.SelectOperator
> (Operator.java:close(591)) - 5 Close done
> 2014-07-18 15:04:09,943 INFO exec.TableScanOperator
> (Operator.java:close(591)) - 4 Close done
> 2014-07-18 15:04:09,951 INFO mapred.FileInputFormat
> (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
> 2014-07-18 15:04:10,008 INFO mapred.FileInputFormat
> (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
> 2014-07-18 15:04:11,526 INFO exec.HashTableSinkOperator
> (SessionState.java:printInfo(537)) - 2014-07-18 03:04:11 Processing
> rows:
> 200000 Hashtable size: 199999 Memory usage: 190041576
> percentage: 0.045
> 2014-07-18 15:04:11,950 INFO exec.HashTableSinkOperator
> (SessionState.java:printInfo(537)) - 2014-07-18 03:04:11 Processing
> rows:
> 300000 Hashtable size: 299999 Memory usage: 250890416
> percentage: 0.059
> 2014-07-18 15:04:12,456 INFO exec.HashTableSinkOperator
> (SessionState.java:printInfo(537)) - 2014-07-18 03:04:12 Processing
> rows:
> 400000 Hashtable size: 399999 Memory usage: 304697120
> percentage: 0.071
> 2014-07-18 15:04:12,744 INFO exec.TableScanOperator
> (Operator.java:close(574)) - 11 finished. closing...
> 2014-07-18 15:04:12,745 INFO exec.FilterOperator
> (Operator.java:close(574)) - 12 finished. closing...
> 2014-07-18 15:04:12,745 INFO exec.FilterOperator
> (Operator.java:logStats(838)) - PASSED:471403
> 2014-07-18 15:04:12,745 INFO exec.FilterOperator
> (Operator.java:logStats(838)) - FILTERED:278619
> 2014-07-18 15:04:12,745 INFO exec.SelectOperator
> (Operator.java:close(574)) - 13 finished. closing...
> 2014-07-18 15:04:12,745 INFO exec.HashTableSinkOperator
> (Operator.java:close(574)) - 14 finished. closing...
> 2014-07-18 15:04:12,745 INFO exec.HashTableSinkOperator
> (HashTableSinkOperator.java:flushToFile(278)) - Temp URI for side table:
> file:/tmp/myuser/hive_2014-07-18_15-03-30_423_6799963466906099923-1/-local-10008/HashTable-Stage-2
> 2014-07-18 15:04:12,745 INFO exec.HashTableSinkOperator
> (SessionState.java:printInfo(537)) - 2014-07-18 03:04:12 Dump the
> side-table into file:
> file:/tmp/myuser/hive_2014-07-18_15-03-30_423_6799963466906099923-1/-local-10008/HashTable-Stage-2/MapJoin-mapfile31--.hashtable
> 2014-07-18 15:04:13,836 INFO exec.HashTableSinkOperator
> (SessionState.java:printInfo(537)) - 2014-07-18 03:04:13 Uploaded 1
> File to:
> file:/tmp/myuser/hive_2014-07-18_15-03-30_423_6799963466906099923-1/-local-10008/HashTable-Stage-2/MapJoin-mapfile31--.hashtable
> (47251316 bytes)
> 2014-07-18 15:04:13,836 INFO exec.HashTableSinkOperator
> (Operator.java:close(591)) - 14 Close done
> 2014-07-18 15:04:13,836 INFO exec.SelectOperator
> (Operator.java:close(591)) - 13 Close done
> 2014-07-18 15:04:13,836 INFO exec.FilterOperator
> (Operator.java:close(591)) - 12 Close done
> 2014-07-18 15:04:13,836 INFO exec.TableScanOperator
> (Operator.java:close(591)) - 11 Close done
> 2014-07-18 15:04:13,837 INFO mr.MapredLocalTask
> (SessionState.java:printInfo(537)) - 2014-07-18 03:04:13 End of local
> task; Time Taken: 35.59 sec.
>
>
--
André Araújo
Big Data Consultant/Solutions Architect
The Pythian Group - Australia - www.pythian.com
Office (calls from within Australia): 1300 366 021 x1270
Office (international): +61 2 8016 7000 x270 *OR* +1 613 565 8696 x1270
Mobile: +61 410 323 559
Fax: +61 2 9805 0544
IM: pythianaraujo @ AIM/MSN/Y! or araujo@pythian.com @ GTalk
“Success is not about standing at the top, it's the steps you leave behind.”
— Iker Pou (rock climber)
--
--
Re: how to control hive log location on 0.13?
Posted by Yang <te...@gmail.com>.
2014-07-18 15:03:37,774 INFO mr.ExecDriver
(SessionState.java:printInfo(537)) - Execution log at:
/tmp/myuser/myuser_2014071815030
3_56bf6bb0-db30-4dbc-807c-9023ce4103f4.log
2014-07-18 15:03:37,864 WARN conf.Configuration
(Configuration.java:loadProperty(2358)) -
file:/tmp/myuser/hive_2014-07-18_15-03-30_423_
6799963466906099923-1/-local-10011/jobconf.xml:an attempt to override final
parameter: mapreduce.job.end-notification.max.retry.interval; Igno
ring.
2014-07-18 15:03:37,871 WARN conf.Configuration
(Configuration.java:loadProperty(2358)) -
file:/tmp/myuser/hive_2014-07-18_15-03-30_423_
6799963466906099923-1/-local-10011/jobconf.xml:an attempt to override final
parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2014-07-18 15:03:37,951 INFO log.PerfLogger
(PerfLogger.java:PerfLogBegin(108)) - <PERFLOG method=deserializePlan
from=org.apache.hadoop.hive.
ql.exec.Utilities>
2014-07-18 15:03:37,951 INFO exec.Utilities
(Utilities.java:deserializePlan(822)) - Deserializing MapredLocalWork via
kryo
2014-07-18 15:03:38,237 INFO log.PerfLogger
(PerfLogger.java:PerfLogEnd(135)) - </PERFLOG method=deserializePlan
start=1405721017951 end=14057
21018237 duration=286 from=org.apache.hadoop.hive.ql.exec.Utilities>
2014-07-18 15:03:38,246 INFO mr.MapredLocalTask
(SessionState.java:printInfo(537)) - 2014-07-18 03:03:38 Starting to
launch local task t
o process map join; maximum memory = 4261937152
2014-07-18 15:03:38,261 INFO mr.MapredLocalTask
(MapredLocalTask.java:initializeOperators(406)) - fetchoperator for
null-subquery2:a-subquery2
:dpkg_cntr:dpkg_wtransaction_p2_id_user_30m created
2014-07-18 15:03:38,263 INFO mr.MapredLocalTask
(MapredLocalTask.java:initializeOperators(406)) - fetchoperator for
null-subquery2:a-subquery2
:dpkg:dpkg_wtransaction_p2_id_user_30m created
2014-07-18 15:03:38,264 INFO mr.MapredLocalTask
(MapredLocalTask.java:initializeOperators(406)) - fetchoperator for
null-subquery2:a-subquery2
:xclick:b:wtrans_data_map_p2_30m created
2014-07-18 15:03:38,266 INFO mr.MapredLocalTask
(MapredLocalTask.java:initializeOperators(406)) - fetchoperator for
null-subquery1:a-subquery1
:dpkg_cntr:dpkg_wtransaction_id_user_30m created
2014-07-18 15:03:38,268 INFO mr.MapredLocalTask
(MapredLocalTask.java:initializeOperators(406)) - fetchoperator for
null-subquery1:a-subquery1
:dpkg:dpkg_wtransaction_id_user_30m created
2014-07-18 15:03:38,269 INFO mr.MapredLocalTask
(MapredLocalTask.java:initializeOperators(406)) - fetchoperator for
null-subquery1:a-subquery1
:xclick:b:wtrans_data_map_30m created
---------------------------------------------
whole bunch of stuff omitted here
--------------------------------------
--------------------------------------
2014-07-18 15:04:08,678 INFO exec.HashTableSinkOperator
(HashTableSinkOperator.java:flushToFile(278)) - Temp URI for side table:
file:/tmp/myuser/hive_2014-07-18_15-03-30_423_6799963466906099923-1/-local-10008/HashTable-Stage-2
2014-07-18 15:04:08,678 INFO exec.HashTableSinkOperator
(SessionState.java:printInfo(537)) - 2014-07-18 03:04:08 Dump the
side-table into file:
file:/tmp/myuser/hive_2014-07-18_15-03-30_423_6799963466906099923-1/-local-10008/HashTable-Stage-2/MapJoin-mapfile11--.hashtable
2014-07-18 15:04:09,943 INFO exec.HashTableSinkOperator
(SessionState.java:printInfo(537)) - 2014-07-18 03:04:09 Uploaded 1
File to:
file:/tmp/myuser/hive_2014-07-18_15-03-30_423_6799963466906099923-1/-local-10008/HashTable-Stage-2/MapJoin-mapfile11--.hashtable
(58010217 bytes)
2014-07-18 15:04:09,943 INFO exec.HashTableSinkOperator
(Operator.java:close(591)) - 6 Close done
2014-07-18 15:04:09,943 INFO exec.SelectOperator
(Operator.java:close(591)) - 5 Close done
2014-07-18 15:04:09,943 INFO exec.TableScanOperator
(Operator.java:close(591)) - 4 Close done
2014-07-18 15:04:09,951 INFO mapred.FileInputFormat
(FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2014-07-18 15:04:10,008 INFO mapred.FileInputFormat
(FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2014-07-18 15:04:11,526 INFO exec.HashTableSinkOperator
(SessionState.java:printInfo(537)) - 2014-07-18 03:04:11 Processing
rows:
200000 Hashtable size: 199999 Memory usage: 190041576
percentage: 0.045
2014-07-18 15:04:11,950 INFO exec.HashTableSinkOperator
(SessionState.java:printInfo(537)) - 2014-07-18 03:04:11 Processing
rows:
300000 Hashtable size: 299999 Memory usage: 250890416
percentage: 0.059
2014-07-18 15:04:12,456 INFO exec.HashTableSinkOperator
(SessionState.java:printInfo(537)) - 2014-07-18 03:04:12 Processing
rows:
400000 Hashtable size: 399999 Memory usage: 304697120
percentage: 0.071
2014-07-18 15:04:12,744 INFO exec.TableScanOperator
(Operator.java:close(574)) - 11 finished. closing...
2014-07-18 15:04:12,745 INFO exec.FilterOperator
(Operator.java:close(574)) - 12 finished. closing...
2014-07-18 15:04:12,745 INFO exec.FilterOperator
(Operator.java:logStats(838)) - PASSED:471403
2014-07-18 15:04:12,745 INFO exec.FilterOperator
(Operator.java:logStats(838)) - FILTERED:278619
2014-07-18 15:04:12,745 INFO exec.SelectOperator
(Operator.java:close(574)) - 13 finished. closing...
2014-07-18 15:04:12,745 INFO exec.HashTableSinkOperator
(Operator.java:close(574)) - 14 finished. closing...
2014-07-18 15:04:12,745 INFO exec.HashTableSinkOperator
(HashTableSinkOperator.java:flushToFile(278)) - Temp URI for side table:
file:/tmp/myuser/hive_2014-07-18_15-03-30_423_6799963466906099923-1/-local-10008/HashTable-Stage-2
2014-07-18 15:04:12,745 INFO exec.HashTableSinkOperator
(SessionState.java:printInfo(537)) - 2014-07-18 03:04:12 Dump the
side-table into file:
file:/tmp/myuser/hive_2014-07-18_15-03-30_423_6799963466906099923-1/-local-10008/HashTable-Stage-2/MapJoin-mapfile31--.hashtable
2014-07-18 15:04:13,836 INFO exec.HashTableSinkOperator
(SessionState.java:printInfo(537)) - 2014-07-18 03:04:13 Uploaded 1
File to:
file:/tmp/myuser/hive_2014-07-18_15-03-30_423_6799963466906099923-1/-local-10008/HashTable-Stage-2/MapJoin-mapfile31--.hashtable
(47251316 bytes)
2014-07-18 15:04:13,836 INFO exec.HashTableSinkOperator
(Operator.java:close(591)) - 14 Close done
2014-07-18 15:04:13,836 INFO exec.SelectOperator
(Operator.java:close(591)) - 13 Close done
2014-07-18 15:04:13,836 INFO exec.FilterOperator
(Operator.java:close(591)) - 12 Close done
2014-07-18 15:04:13,836 INFO exec.TableScanOperator
(Operator.java:close(591)) - 11 Close done
2014-07-18 15:04:13,837 INFO mr.MapredLocalTask
(SessionState.java:printInfo(537)) - 2014-07-18 03:04:13 End of local
task; Time Taken: 35.59 sec.
Re: how to control hive log location on 0.13?
Posted by Yang <te...@gmail.com>.
it's in /tmp/my_user/
the funny thing is that I already have a hive.log there.
On Fri, Jul 18, 2014 at 6:01 PM, Andre Araujo <ar...@pythian.com> wrote:
> and where is it located?
>
>
> On 19 July 2014 10:58, Andre Araujo <ar...@pythian.com> wrote:
>
>> Can you give us an excerpt of the contents of this log?
>>
>>
>> On 19 July 2014 04:38, Yang <te...@gmail.com> wrote:
>>
>>> thanks guys. anybody knows what generates the log like "
>>> myuser_20140716143232_d76043ed-1c4b-42a0-bf0a-2816377a6a2a.log" ? I
>>> checked our application code, it doesn't generate this, looks from hive.
>>>
>>>
>>> On Fri, Jul 18, 2014 at 12:28 AM, Andre Araujo <ar...@pythian.com>
>>> wrote:
>>>
>>>> Make sure the directory you specify has the sticky bit set, otherwise
>>>> users will have permission problems:
>>>>
>>>> chmod 1777 <dir>
>>>>
>>>>
>>>> On 18 July 2014 14:19, Satish Mittal <sa...@inmobi.com> wrote:
>>>>
>>>>> You can configure the following property in
>>>>> $HIVE_HOME/conf/hive-log4j.properties:
>>>>>
>>>>> hive.log.dir=<your location>
>>>>>
>>>>> The default value of this property is ${java.io.tmpdir}/${user.name}.
>>>>>
>>>>> Thanks,
>>>>> Satish
>>>>>
>>>>>
>>>>> On Thu, Jul 17, 2014 at 11:58 PM, Yang <te...@gmail.com> wrote:
>>>>>
>>>>>> we just moved to hadoop2.0 (HDP2.1 distro). it turns out that the new
>>>>>> hive version generates a lot of logs into /tmp/ and is quickly creating the
>>>>>> danger of running out of our /tmp/ space.
>>>>>>
>>>>>>
>>>>>> I see these 2 different logs :
>>>>>>
>>>>>> [myuser@mybox ~]$ ls -lt /tmp/myuser/
>>>>>> total 1988
>>>>>> -rw-rw-r-- 1 myuser myuser 191687 2014-07-17 11:17 hive.log
>>>>>> -rw-rw-r-- 1 myuser myuser 14472 2014-07-16 14:43
>>>>>> myuser_20140716143232_d76043ed-1c4b-42a0-bf0a-2816377a6a2a.log
>>>>>> -rw-rw-r-- 1 myuser myuser 14260 2014-07-16 14:04
>>>>>> myuser_20140716135353_de698da0-807f-4e3b-8b97-5af5064b55f2.log
>>>>>> -rw-rw-r-- 1 myuser myuser 14254 2014-07-16 13:42
>>>>>> myuser_20140716133838_208329bd-77bb-4981-a2e9-e747647d0704.log
>>>>>>
>>>>>>
>>>>>>
>>>>>> from the doc at
>>>>>> https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-ErrorLogs
>>>>>> I can see that "per Hive session basis in /tmp/<user.name>/, but can
>>>>>> be configured in hive-site.xml
>>>>>> <https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration> with
>>>>>> the hive.querylog.location property.",
>>>>>> but I tried to pass it to -hiveconf hive.querylog.location=/tmp/mycustomlogdir/
>>>>>> , doesn't seem to work; the hive.log location is not changed by this
>>>>>> approach either.
>>>>>>
>>>>>> so how can I change the location of both the logs , by some
>>>>>> per-script params ? (i.e. we can't afford to change the system
>>>>>> hive-site.xml or /etc/hive/conf etc)
>>>>>>
>>>>>> Thanks a lot
>>>>>> Yang
>>>>>>
>>>>>
>>>>>
>>>>> _____________________________________________________________
>>>>> The information contained in this communication is intended solely for
>>>>> the use of the individual or entity to whom it is addressed and others
>>>>> authorized to receive it. It may contain confidential or legally privileged
>>>>> information. If you are not the intended recipient you are hereby notified
>>>>> that any disclosure, copying, distribution or taking any action in reliance
>>>>> on the contents of this information is strictly prohibited and may be
>>>>> unlawful. If you have received this communication in error, please notify
>>>>> us immediately by responding to this email and then delete it from your
>>>>> system. The firm is neither liable for the proper and complete transmission
>>>>> of the information contained in this communication nor for any delay in its
>>>>> receipt.
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> André Araújo
>>>> Big Data Consultant/Solutions Architect
>>>> The Pythian Group - Australia - www.pythian.com
>>>>
>>>> Office (calls from within Australia): 1300 366 021 x1270
>>>> Office (international): +61 2 8016 7000 x270 *OR* +1 613 565 8696
>>>> x1270
>>>> Mobile: +61 410 323 559
>>>> Fax: +61 2 9805 0544
>>>> IM: pythianaraujo @ AIM/MSN/Y! or araujo@pythian.com @ GTalk
>>>>
>>>> “Success is not about standing at the top, it's the steps you leave
>>>> behind.” — Iker Pou (rock climber)
>>>>
>>>> --
>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> André Araújo
>> Big Data Consultant/Solutions Architect
>> The Pythian Group - Australia - www.pythian.com
>>
>> Office (calls from within Australia): 1300 366 021 x1270
>> Office (international): +61 2 8016 7000 x270 *OR* +1 613 565 8696
>> x1270
>> Mobile: +61 410 323 559
>> Fax: +61 2 9805 0544
>> IM: pythianaraujo @ AIM/MSN/Y! or araujo@pythian.com @ GTalk
>>
>> “Success is not about standing at the top, it's the steps you leave
>> behind.” — Iker Pou (rock climber)
>>
>
>
>
> --
> André Araújo
> Big Data Consultant/Solutions Architect
> The Pythian Group - Australia - www.pythian.com
>
> Office (calls from within Australia): 1300 366 021 x1270
> Office (international): +61 2 8016 7000 x270 *OR* +1 613 565 8696 x1270
> Mobile: +61 410 323 559
> Fax: +61 2 9805 0544
> IM: pythianaraujo @ AIM/MSN/Y! or araujo@pythian.com @ GTalk
>
> “Success is not about standing at the top, it's the steps you leave behind.”
> — Iker Pou (rock climber)
>
> --
>
>
>
>
Re: how to control hive log location on 0.13?
Posted by Andre Araujo <ar...@pythian.com>.
and where is it located?
On 19 July 2014 10:58, Andre Araujo <ar...@pythian.com> wrote:
> Can you give us an excerpt of the contents of this log?
>
>
> On 19 July 2014 04:38, Yang <te...@gmail.com> wrote:
>
>> thanks guys. anybody knows what generates the log like "
>> myuser_20140716143232_d76043ed-1c4b-42a0-bf0a-2816377a6a2a.log" ? I
>> checked our application code, it doesn't generate this, looks from hive.
>>
>>
>> On Fri, Jul 18, 2014 at 12:28 AM, Andre Araujo <ar...@pythian.com>
>> wrote:
>>
>>> Make sure the directory you specify has the sticky bit set, otherwise
>>> users will have permission problems:
>>>
>>> chmod 1777 <dir>
>>>
>>>
>>> On 18 July 2014 14:19, Satish Mittal <sa...@inmobi.com> wrote:
>>>
>>>> You can configure the following property in
>>>> $HIVE_HOME/conf/hive-log4j.properties:
>>>>
>>>> hive.log.dir=<your location>
>>>>
>>>> The default value of this property is ${java.io.tmpdir}/${user.name}.
>>>>
>>>> Thanks,
>>>> Satish
>>>>
>>>>
>>>> On Thu, Jul 17, 2014 at 11:58 PM, Yang <te...@gmail.com> wrote:
>>>>
>>>>> we just moved to hadoop2.0 (HDP2.1 distro). it turns out that the new
>>>>> hive version generates a lot of logs into /tmp/ and is quickly creating the
>>>>> danger of running out of our /tmp/ space.
>>>>>
>>>>>
>>>>> I see these 2 different logs :
>>>>>
>>>>> [myuser@mybox ~]$ ls -lt /tmp/myuser/
>>>>> total 1988
>>>>> -rw-rw-r-- 1 myuser myuser 191687 2014-07-17 11:17 hive.log
>>>>> -rw-rw-r-- 1 myuser myuser 14472 2014-07-16 14:43
>>>>> myuser_20140716143232_d76043ed-1c4b-42a0-bf0a-2816377a6a2a.log
>>>>> -rw-rw-r-- 1 myuser myuser 14260 2014-07-16 14:04
>>>>> myuser_20140716135353_de698da0-807f-4e3b-8b97-5af5064b55f2.log
>>>>> -rw-rw-r-- 1 myuser myuser 14254 2014-07-16 13:42
>>>>> myuser_20140716133838_208329bd-77bb-4981-a2e9-e747647d0704.log
>>>>>
>>>>>
>>>>>
>>>>> from the doc at
>>>>> https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-ErrorLogs
>>>>> I can see that "per Hive session basis in /tmp/<user.name>/, but can
>>>>> be configured in hive-site.xml
>>>>> <https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration> with
>>>>> the hive.querylog.location property.",
>>>>> but I tried to pass it to -hiveconf hive.querylog.location=/tmp/mycustomlogdir/
>>>>> , doesn't seem to work; the hive.log location is not changed by this
>>>>> approach either.
>>>>>
>>>>> so how can I change the location of both the logs , by some per-script
>>>>> params ? (i.e. we can't afford to change the system hive-site.xml or
>>>>> /etc/hive/conf etc)
>>>>>
>>>>> Thanks a lot
>>>>> Yang
>>>>>
>>>>
>>>>
>>>> _____________________________________________________________
>>>> The information contained in this communication is intended solely for
>>>> the use of the individual or entity to whom it is addressed and others
>>>> authorized to receive it. It may contain confidential or legally privileged
>>>> information. If you are not the intended recipient you are hereby notified
>>>> that any disclosure, copying, distribution or taking any action in reliance
>>>> on the contents of this information is strictly prohibited and may be
>>>> unlawful. If you have received this communication in error, please notify
>>>> us immediately by responding to this email and then delete it from your
>>>> system. The firm is neither liable for the proper and complete transmission
>>>> of the information contained in this communication nor for any delay in its
>>>> receipt.
>>>
>>>
>>>
>>>
>>> --
>>> André Araújo
>>> Big Data Consultant/Solutions Architect
>>> The Pythian Group - Australia - www.pythian.com
>>>
>>> Office (calls from within Australia): 1300 366 021 x1270
>>> Office (international): +61 2 8016 7000 x270 *OR* +1 613 565 8696
>>> x1270
>>> Mobile: +61 410 323 559
>>> Fax: +61 2 9805 0544
>>> IM: pythianaraujo @ AIM/MSN/Y! or araujo@pythian.com @ GTalk
>>>
>>> “Success is not about standing at the top, it's the steps you leave
>>> behind.” — Iker Pou (rock climber)
>>>
>>> --
>>>
>>>
>>>
>>>
>>
>
>
> --
> André Araújo
> Big Data Consultant/Solutions Architect
> The Pythian Group - Australia - www.pythian.com
>
> Office (calls from within Australia): 1300 366 021 x1270
> Office (international): +61 2 8016 7000 x270 *OR* +1 613 565 8696 x1270
> Mobile: +61 410 323 559
> Fax: +61 2 9805 0544
> IM: pythianaraujo @ AIM/MSN/Y! or araujo@pythian.com @ GTalk
>
> “Success is not about standing at the top, it's the steps you leave behind.”
> — Iker Pou (rock climber)
>
--
André Araújo
Big Data Consultant/Solutions Architect
The Pythian Group - Australia - www.pythian.com
Office (calls from within Australia): 1300 366 021 x1270
Office (international): +61 2 8016 7000 x270 *OR* +1 613 565 8696 x1270
Mobile: +61 410 323 559
Fax: +61 2 9805 0544
IM: pythianaraujo @ AIM/MSN/Y! or araujo@pythian.com @ GTalk
“Success is not about standing at the top, it's the steps you leave behind.”
— Iker Pou (rock climber)
--
--
Re: how to control hive log location on 0.13?
Posted by Andre Araujo <ar...@pythian.com>.
Can you give us an excerpt of the contents of this log?
On 19 July 2014 04:38, Yang <te...@gmail.com> wrote:
> thanks guys. anybody knows what generates the log like "
> myuser_20140716143232_d76043ed-1c4b-42a0-bf0a-2816377a6a2a.log" ? I
> checked our application code, it doesn't generate this, looks from hive.
>
>
> On Fri, Jul 18, 2014 at 12:28 AM, Andre Araujo <ar...@pythian.com> wrote:
>
>> Make sure the directory you specify has the sticky bit set, otherwise
>> users will have permission problems:
>>
>> chmod 1777 <dir>
>>
>>
>> On 18 July 2014 14:19, Satish Mittal <sa...@inmobi.com> wrote:
>>
>>> You can configure the following property in
>>> $HIVE_HOME/conf/hive-log4j.properties:
>>>
>>> hive.log.dir=<your location>
>>>
>>> The default value of this property is ${java.io.tmpdir}/${user.name}.
>>>
>>> Thanks,
>>> Satish
>>>
>>>
>>> On Thu, Jul 17, 2014 at 11:58 PM, Yang <te...@gmail.com> wrote:
>>>
>>>> we just moved to hadoop2.0 (HDP2.1 distro). it turns out that the new
>>>> hive version generates a lot of logs into /tmp/ and is quickly creating the
>>>> danger of running out of our /tmp/ space.
>>>>
>>>>
>>>> I see these 2 different logs :
>>>>
>>>> [myuser@mybox ~]$ ls -lt /tmp/myuser/
>>>> total 1988
>>>> -rw-rw-r-- 1 myuser myuser 191687 2014-07-17 11:17 hive.log
>>>> -rw-rw-r-- 1 myuser myuser 14472 2014-07-16 14:43
>>>> myuser_20140716143232_d76043ed-1c4b-42a0-bf0a-2816377a6a2a.log
>>>> -rw-rw-r-- 1 myuser myuser 14260 2014-07-16 14:04
>>>> myuser_20140716135353_de698da0-807f-4e3b-8b97-5af5064b55f2.log
>>>> -rw-rw-r-- 1 myuser myuser 14254 2014-07-16 13:42
>>>> myuser_20140716133838_208329bd-77bb-4981-a2e9-e747647d0704.log
>>>>
>>>>
>>>>
>>>> from the doc at
>>>> https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-ErrorLogs
>>>> I can see that "per Hive session basis in /tmp/<user.name>/, but can
>>>> be configured in hive-site.xml
>>>> <https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration> with
>>>> the hive.querylog.location property.",
>>>> but I tried to pass it to -hiveconf hive.querylog.location=/tmp/mycustomlogdir/
>>>> , doesn't seem to work; the hive.log location is not changed by this
>>>> approach either.
>>>>
>>>> so how can I change the location of both the logs , by some per-script
>>>> params ? (i.e. we can't afford to change the system hive-site.xml or
>>>> /etc/hive/conf etc)
>>>>
>>>> Thanks a lot
>>>> Yang
>>>>
>>>
>>>
>>> _____________________________________________________________
>>> The information contained in this communication is intended solely for
>>> the use of the individual or entity to whom it is addressed and others
>>> authorized to receive it. It may contain confidential or legally privileged
>>> information. If you are not the intended recipient you are hereby notified
>>> that any disclosure, copying, distribution or taking any action in reliance
>>> on the contents of this information is strictly prohibited and may be
>>> unlawful. If you have received this communication in error, please notify
>>> us immediately by responding to this email and then delete it from your
>>> system. The firm is neither liable for the proper and complete transmission
>>> of the information contained in this communication nor for any delay in its
>>> receipt.
>>
>>
>>
>>
>> --
>> André Araújo
>> Big Data Consultant/Solutions Architect
>> The Pythian Group - Australia - www.pythian.com
>>
>> Office (calls from within Australia): 1300 366 021 x1270
>> Office (international): +61 2 8016 7000 x270 *OR* +1 613 565 8696
>> x1270
>> Mobile: +61 410 323 559
>> Fax: +61 2 9805 0544
>> IM: pythianaraujo @ AIM/MSN/Y! or araujo@pythian.com @ GTalk
>>
>> “Success is not about standing at the top, it's the steps you leave
>> behind.” — Iker Pou (rock climber)
>>
>> --
>>
>>
>>
>>
>
--
André Araújo
Big Data Consultant/Solutions Architect
The Pythian Group - Australia - www.pythian.com
Office (calls from within Australia): 1300 366 021 x1270
Office (international): +61 2 8016 7000 x270 *OR* +1 613 565 8696 x1270
Mobile: +61 410 323 559
Fax: +61 2 9805 0544
IM: pythianaraujo @ AIM/MSN/Y! or araujo@pythian.com @ GTalk
“Success is not about standing at the top, it's the steps you leave behind.”
— Iker Pou (rock climber)
--
--
Re: how to control hive log location on 0.13?
Posted by Yang <te...@gmail.com>.
thanks guys. anybody knows what generates the log like "
myuser_20140716143232_d76043ed-1c4b-42a0-bf0a-2816377a6a2a.log" ? I checked
our application code, it doesn't generate this, looks from hive.
On Fri, Jul 18, 2014 at 12:28 AM, Andre Araujo <ar...@pythian.com> wrote:
> Make sure the directory you specify has the sticky bit set, otherwise
> users will have permission problems:
>
> chmod 1777 <dir>
>
>
> On 18 July 2014 14:19, Satish Mittal <sa...@inmobi.com> wrote:
>
>> You can configure the following property in
>> $HIVE_HOME/conf/hive-log4j.properties:
>>
>> hive.log.dir=<your location>
>>
>> The default value of this property is ${java.io.tmpdir}/${user.name}.
>>
>> Thanks,
>> Satish
>>
>>
>> On Thu, Jul 17, 2014 at 11:58 PM, Yang <te...@gmail.com> wrote:
>>
>>> we just moved to hadoop2.0 (HDP2.1 distro). it turns out that the new
>>> hive version generates a lot of logs into /tmp/ and is quickly creating the
>>> danger of running out of our /tmp/ space.
>>>
>>>
>>> I see these 2 different logs :
>>>
>>> [myuser@mybox ~]$ ls -lt /tmp/myuser/
>>> total 1988
>>> -rw-rw-r-- 1 myuser myuser 191687 2014-07-17 11:17 hive.log
>>> -rw-rw-r-- 1 myuser myuser 14472 2014-07-16 14:43
>>> myuser_20140716143232_d76043ed-1c4b-42a0-bf0a-2816377a6a2a.log
>>> -rw-rw-r-- 1 myuser myuser 14260 2014-07-16 14:04
>>> myuser_20140716135353_de698da0-807f-4e3b-8b97-5af5064b55f2.log
>>> -rw-rw-r-- 1 myuser myuser 14254 2014-07-16 13:42
>>> myuser_20140716133838_208329bd-77bb-4981-a2e9-e747647d0704.log
>>>
>>>
>>>
>>> from the doc at
>>> https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-ErrorLogs
>>> I can see that "per Hive session basis in /tmp/<user.name>/, but can be
>>> configured in hive-site.xml
>>> <https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration> with
>>> the hive.querylog.location property.",
>>> but I tried to pass it to -hiveconf hive.querylog.location=/tmp/mycustomlogdir/
>>> , doesn't seem to work; the hive.log location is not changed by this
>>> approach either.
>>>
>>> so how can I change the location of both the logs , by some per-script
>>> params ? (i.e. we can't afford to change the system hive-site.xml or
>>> /etc/hive/conf etc)
>>>
>>> Thanks a lot
>>> Yang
>>>
>>
>>
>> _____________________________________________________________
>> The information contained in this communication is intended solely for
>> the use of the individual or entity to whom it is addressed and others
>> authorized to receive it. It may contain confidential or legally privileged
>> information. If you are not the intended recipient you are hereby notified
>> that any disclosure, copying, distribution or taking any action in reliance
>> on the contents of this information is strictly prohibited and may be
>> unlawful. If you have received this communication in error, please notify
>> us immediately by responding to this email and then delete it from your
>> system. The firm is neither liable for the proper and complete transmission
>> of the information contained in this communication nor for any delay in its
>> receipt.
>
>
>
>
> --
> André Araújo
> Big Data Consultant/Solutions Architect
> The Pythian Group - Australia - www.pythian.com
>
> Office (calls from within Australia): 1300 366 021 x1270
> Office (international): +61 2 8016 7000 x270 *OR* +1 613 565 8696 x1270
> Mobile: +61 410 323 559
> Fax: +61 2 9805 0544
> IM: pythianaraujo @ AIM/MSN/Y! or araujo@pythian.com @ GTalk
>
> “Success is not about standing at the top, it's the steps you leave behind.”
> — Iker Pou (rock climber)
>
> --
>
>
>
>
Re: how to control hive log location on 0.13?
Posted by Andre Araujo <ar...@pythian.com>.
Make sure the directory you specify has the sticky bit set, otherwise users
will have permission problems:
chmod 1777 <dir>
On 18 July 2014 14:19, Satish Mittal <sa...@inmobi.com> wrote:
> You can configure the following property in
> $HIVE_HOME/conf/hive-log4j.properties:
>
> hive.log.dir=<your location>
>
> The default value of this property is ${java.io.tmpdir}/${user.name}.
>
> Thanks,
> Satish
>
>
> On Thu, Jul 17, 2014 at 11:58 PM, Yang <te...@gmail.com> wrote:
>
>> we just moved to hadoop2.0 (HDP2.1 distro). it turns out that the new
>> hive version generates a lot of logs into /tmp/ and is quickly creating the
>> danger of running out of our /tmp/ space.
>>
>>
>> I see these 2 different logs :
>>
>> [myuser@mybox ~]$ ls -lt /tmp/myuser/
>> total 1988
>> -rw-rw-r-- 1 myuser myuser 191687 2014-07-17 11:17 hive.log
>> -rw-rw-r-- 1 myuser myuser 14472 2014-07-16 14:43
>> myuser_20140716143232_d76043ed-1c4b-42a0-bf0a-2816377a6a2a.log
>> -rw-rw-r-- 1 myuser myuser 14260 2014-07-16 14:04
>> myuser_20140716135353_de698da0-807f-4e3b-8b97-5af5064b55f2.log
>> -rw-rw-r-- 1 myuser myuser 14254 2014-07-16 13:42
>> myuser_20140716133838_208329bd-77bb-4981-a2e9-e747647d0704.log
>>
>>
>>
>> from the doc at
>> https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-ErrorLogs
>> I can see that "per Hive session basis in /tmp/<user.name>/, but can be
>> configured in hive-site.xml
>> <https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration> with
>> the hive.querylog.location property.",
>> but I tried to pass it to -hiveconf hive.querylog.location=/tmp/mycustomlogdir/
>> , doesn't seem to work; the hive.log location is not changed by this
>> approach either.
>>
>> so how can I change the location of both the logs , by some per-script
>> params ? (i.e. we can't afford to change the system hive-site.xml or
>> /etc/hive/conf etc)
>>
>> Thanks a lot
>> Yang
>>
>
>
> _____________________________________________________________
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
--
André Araújo
Big Data Consultant/Solutions Architect
The Pythian Group - Australia - www.pythian.com
Office (calls from within Australia): 1300 366 021 x1270
Office (international): +61 2 8016 7000 x270 *OR* +1 613 565 8696 x1270
Mobile: +61 410 323 559
Fax: +61 2 9805 0544
IM: pythianaraujo @ AIM/MSN/Y! or araujo@pythian.com @ GTalk
“Success is not about standing at the top, it's the steps you leave behind.”
— Iker Pou (rock climber)
--
--
Re: how to control hive log location on 0.13?
Posted by Satish Mittal <sa...@inmobi.com>.
You can configure the following property in
$HIVE_HOME/conf/hive-log4j.properties:
hive.log.dir=<your location>
The default value of this property is ${java.io.tmpdir}/${user.name}.
Thanks,
Satish
On Thu, Jul 17, 2014 at 11:58 PM, Yang <te...@gmail.com> wrote:
> we just moved to hadoop2.0 (HDP2.1 distro). it turns out that the new hive
> version generates a lot of logs into /tmp/ and is quickly creating the
> danger of running out of our /tmp/ space.
>
>
> I see these 2 different logs :
>
> [myuser@mybox ~]$ ls -lt /tmp/myuser/
> total 1988
> -rw-rw-r-- 1 myuser myuser 191687 2014-07-17 11:17 hive.log
> -rw-rw-r-- 1 myuser myuser 14472 2014-07-16 14:43
> myuser_20140716143232_d76043ed-1c4b-42a0-bf0a-2816377a6a2a.log
> -rw-rw-r-- 1 myuser myuser 14260 2014-07-16 14:04
> myuser_20140716135353_de698da0-807f-4e3b-8b97-5af5064b55f2.log
> -rw-rw-r-- 1 myuser myuser 14254 2014-07-16 13:42
> myuser_20140716133838_208329bd-77bb-4981-a2e9-e747647d0704.log
>
>
>
> from the doc at
> https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-ErrorLogs
> I can see that "per Hive session basis in /tmp/<user.name>/, but can be
> configured in hive-site.xml
> <https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration> with
> the hive.querylog.location property.",
> but I tried to pass it to -hiveconf hive.querylog.location=/tmp/mycustomlogdir/
> , doesn't seem to work; the hive.log location is not changed by this
> approach either.
>
> so how can I change the location of both the logs , by some per-script
> params ? (i.e. we can't afford to change the system hive-site.xml or
> /etc/hive/conf etc)
>
> Thanks a lot
> Yang
>
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.