You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Jasper Knulst <ja...@vlc.nl> on 2011/09/12 21:32:18 UTC
Managing Hive logging detail (per Hive statement) in /tmp/${user.name}/
Hi,
Does anyone know how I can reduce the level of logging of the Hive History
file which are created per Hive statement executed in the (default) location
of /tmp/${user.name}/ hive_job_log_hdfs_201109122109_675419678.txt. The
level of detail is very high by default but I can't find a way to tune it
down.
It looks like this:
-------------------------------
SessionStart SESSION_ID="hdfs_201109122110" TIME="1315854630381"
QueryStart QUERY_STRING="show partitions dart_facts"
QUERY_ID="hdfs_20110912211010_e574273f-40dc-4061-9d03-6e9a0c2db8c4"
TIME="1315854630789"
Counters
plan="{"queryId":"hdfs_20110912211010_e574273f-40dc-4061-9d03-6e9a0c2db8c4","queryType":null,"queryAttributes":{"queryString":"show
partitions
dart_facts"},"queryCounters":"null","stageGraph":{"nodeType":"STAGE","roots":"null","adjacencyList":"]"},"stageList":[{"stageId":"Stage-0","stageType":"DDL","stageAttributes":"null","stageCounters":"}","taskList":[{"taskId":"Stage-0_OTHER","taskType":"OTHER","taskAttributes":"null","taskCounters":"null","operatorGraph":"null","operatorList":"]","done":"false","started":"false"}],"done":"false","started":"false"}],"done":"false","started":"true"}"
TIME="1315854630797"
TaskStart TASK_NAME="org.apache.hadoop.hive.ql.exec.DDLTask"
TASK_ID="Stage-0"
QUERY_ID="hdfs_20110912211010_e574273f-40dc-4061-9d03-6e9a0c2db8c4"
TIME="1315854630805"
Counters
plan="{"queryId":"hdfs_20110912211010_e574273f-40dc-4061-9d03-6e9a0c2db8c4","queryType":null,"queryAttributes":{"queryString":"show
partitions
dart_facts"},"queryCounters":"null","stageGraph":{"nodeType":"STAGE","roots":"null","adjacencyList":"]"},"stageList":[{"stageId":"Stage-0","stageType":"DDL","stageAttributes":"null","stageCounters":"}","taskList":[{"taskId":"Stage-0_OTHER","taskType":"OTHER","taskAttributes":"null","taskCounters":"null","operatorGraph":"null","operatorList":"]","done":"false","started":"true"}],"done":"false","started":"true"}],"done":"false","started":"true"}"
TIME="1315854630806"
-----------------------------------------
It doesn't look like standard log4j logging to me and this logging does not
respond to the log4j properties in :
-/usr/lib/hive/conf/hive-log4j (these settings influence
the detail in HIVE_LOG_DIR/hive.log)
-/usr/lib/hive/conf/hive-exec-log4j (don't know where these
settings are meant for. Anyone ? )
I have to find a way to turn the logging in /tmp/${user.name}/ down. I ran a
large Hive job once that generated 8GB of logging this way which stalled the
server.
Thanks
--
Kind Regards,
Re: Managing Hive logging detail (per Hive statement) in /tmp/${user.name}/
Posted by Adriaan Tijsseling <ad...@tijsseling.com>.
Should be here:
./conf/hive-exec-log4j.properties
./conf/hive-log4j.properties
Cheers,
Adriaan
On 2011/09/12, at 21:32, Jasper Knulst wrote:
> Hi,
>
> Does anyone know how I can reduce the level of logging of the Hive History
> file which are created per Hive statement executed in the (default) location
> of /tmp/${user.name}/ hive_job_log_hdfs_201109122109_675419678.txt. The
> level of detail is very high by default but I can't find a way to tune it
> down.
>
>
> It looks like this:
> -------------------------------
> SessionStart SESSION_ID="hdfs_201109122110" TIME="1315854630381"
> QueryStart QUERY_STRING="show partitions dart_facts"
> QUERY_ID="hdfs_20110912211010_e574273f-40dc-4061-9d03-6e9a0c2db8c4"
> TIME="1315854630789"
> Counters
> plan="{"queryId":"hdfs_20110912211010_e574273f-40dc-4061-9d03-6e9a0c2db8c4","queryType":null,"queryAttributes":{"queryString":"show
> partitions
> dart_facts"},"queryCounters":"null","stageGraph":{"nodeType":"STAGE","roots":"null","adjacencyList":"]"},"stageList":[{"stageId":"Stage-0","stageType":"DDL","stageAttributes":"null","stageCounters":"}","taskList":[{"taskId":"Stage-0_OTHER","taskType":"OTHER","taskAttributes":"null","taskCounters":"null","operatorGraph":"null","operatorList":"]","done":"false","started":"false"}],"done":"false","started":"false"}],"done":"false","started":"true"}"
> TIME="1315854630797"
> TaskStart TASK_NAME="org.apache.hadoop.hive.ql.exec.DDLTask"
> TASK_ID="Stage-0"
> QUERY_ID="hdfs_20110912211010_e574273f-40dc-4061-9d03-6e9a0c2db8c4"
> TIME="1315854630805"
> Counters
> plan="{"queryId":"hdfs_20110912211010_e574273f-40dc-4061-9d03-6e9a0c2db8c4","queryType":null,"queryAttributes":{"queryString":"show
> partitions
> dart_facts"},"queryCounters":"null","stageGraph":{"nodeType":"STAGE","roots":"null","adjacencyList":"]"},"stageList":[{"stageId":"Stage-0","stageType":"DDL","stageAttributes":"null","stageCounters":"}","taskList":[{"taskId":"Stage-0_OTHER","taskType":"OTHER","taskAttributes":"null","taskCounters":"null","operatorGraph":"null","operatorList":"]","done":"false","started":"true"}],"done":"false","started":"true"}],"done":"false","started":"true"}"
> TIME="1315854630806"
> -----------------------------------------
>
> It doesn't look like standard log4j logging to me and this logging does not
> respond to the log4j properties in :
>
> -/usr/lib/hive/conf/hive-log4j (these settings influence
> the detail in HIVE_LOG_DIR/hive.log)
> -/usr/lib/hive/conf/hive-exec-log4j (don't know where these
> settings are meant for. Anyone ? )
>
> I have to find a way to turn the logging in /tmp/${user.name}/ down. I ran a
> large Hive job once that generated 8GB of logging this way which stalled the
> server.
>
> Thanks
>
> --
> Kind Regards,