You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Vincent Barat <vi...@gmail.com> on 2011/08/26 16:56:43 UTC
Re: Removing local job traces in local mode
Here is how I solved this issue (it was only related to log4j
configuration):
/* Deactivate most traces from Hadoop and PIG (keep ERROR) */
props.setProperty("log4j.logger.org.apache.hadoop", "ERROR");
props.setProperty("log4j.logger.org.apache.zookeeper", "ERROR");
props.setProperty("log4j.logger.org.apache.pig", "ERROR");
/*
* Keep traces from M/R launcher (percentage of advancement)
and job statistics (nb of record
* written etc.)
*/
props.setProperty(
"log4j.logger.org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher",
"INFO");
props.setProperty("log4j.logger.org.apache.pig.tools.pigstats.PigStats",
"INFO");
Le 27/06/11 13:21, Vincent Barat a écrit :
> Thanks, but it is actually not what I'm looking for.
>
> First, I use PIG from Java, not from the pig shell, so the "-d"
> option cannot be used.
> Second, I want to be able to choose between which trace I want,
> and which I don't, depending on the origin of the trace.
> I guess log4j can be configured for that, but I don't manage to
> configure it correctly.
>
> Any hint ?
>
> Le 27/06/11 12:49, Vincent a écrit :
>> You can try -d option:
>>
>> For example pig -d WARN -x local myscript.pig
>>
>> On Mon, Jun 27, 2011 at 2:46 PM, Vincent
>> Barat<vi...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> I'm using PIG in local mode during development phases, and I get
>>> bored
>>> about traces outputted by local jobs (the one attached).
>>> Is there a way to get rid of them?
>>>
>>> Thanks for your help.
>>>
>>> 2011-06-27 12:26:56,289 | WARN | Thread-55 | JobClient | Use
>>> GenericOptionsParser for parsing the arguments. Applications should
>>> implement Tool for the same.
>>> 2011-06-27 12:26:56,423 | INFO | Thread-55 | JvmMetrics | Cannot
>>> initialize JVM Metrics with processName=JobTracker, sessionId= -
>>> already
>>> initialized
>>> 2011-06-27 12:26:56,427 | INFO | Thread-55 | JvmMetrics | Cannot
>>> initialize JVM Metrics with processName=JobTracker, sessionId= -
>>> already
>>> initialized
>>> 2011-06-27 12:26:56,429 | INFO | Thread-55 | FileInputFormat |
>>> Total input
>>> paths to process : 1
>>> 2011-06-27 12:26:56,429 | INFO | Thread-55 | MapRedUtil | Total
>>> input
>>> paths to process : 1
>>> 2011-06-27 12:26:56,429 | INFO | Thread-55 | MapRedUtil | Total
>>> input
>>> paths (combined) to process : 1
>>> 2011-06-27 12:26:56,614 | INFO | Thread-64 | JvmMetrics | Cannot
>>> initialize JVM Metrics with processName=JobTracker, sessionId= -
>>> already
>>> initialized
>>> 2011-06-27 12:26:56,616 | INFO | Thread-64 | FileInputFormat |
>>> Total input
>>> paths to process : 1
>>> 2011-06-27 12:26:56,616 | INFO | Thread-64 | MapRedUtil | Total
>>> input
>>> paths to process : 1
>>> 2011-06-27 12:26:56,616 | INFO | Thread-64 | MapRedUtil | Total
>>> input
>>> paths (combined) to process : 1
>>> 2011-06-27 12:26:56,653 | INFO | Thread-64 | JvmMetrics | Cannot
>>> initialize JVM Metrics with processName=JobTracker, sessionId= -
>>> already
>>> initialized
>>> 2011-06-27 12:26:56,657 | INFO | Thread-64 | JvmMetrics | Cannot
>>> initialize JVM Metrics with processName=JobTracker, sessionId= -
>>> already
>>> initialized
>>> 2011-06-27 12:26:56,658 | INFO | Thread-64 | MapTask |
>>> io.sort.mb = 100
>>> 2011-06-27 12:26:56,751 | INFO | Thread-64 | MapTask | data
>>> buffer =
>>> 79691776/99614720
>>> 2011-06-27 12:26:56,751 | INFO | Thread-64 | MapTask |
>>> record buffer =
>>> 262144/327680
>>> 2011-06-27 12:26:56,775 | INFO | Thread-64 | MapTask |
>>> Starting flush
>>> of map output
>>> 2011-06-27 12:26:56,785 | INFO | Thread-64 | MapTask |
>>> Finished spill 0
>>> 2011-06-27 12:26:56,790 | INFO | Thread-64 | LocalJobRunner |
>>> 2011-06-27 12:26:56,790 | INFO | Thread-64 | TaskRunner | Task
>>> 'attempt_local_0004_m_000000_**0' done.
>>> 2011-06-27 12:26:56,795 | INFO | Thread-64 | JvmMetrics | Cannot
>>> initialize JVM Metrics with processName=JobTracker, sessionId= -
>>> already
>>> initialized
>>> 2011-06-27 12:26:56,797 | INFO | Thread-64 | JvmMetrics | Cannot
>>> initialize JVM Metrics with processName=JobTracker, sessionId= -
>>> already
>>> initialized
>>> 2011-06-27 12:26:56,797 | INFO | Thread-64 | LocalJobRunner |
>>> 2011-06-27 12:26:56,798 | INFO | Thread-64 | Merger |
>>> Merging 1 sorted
>>> segments
>>> 2011-06-27 12:26:56,799 | INFO | Thread-64 | Merger | Down
>>> to the last
>>> merge-pass, with 1 segments left of total size: 350 bytes
>>> 2011-06-27 12:26:56,799 | INFO | Thread-64 | LocalJobRunner |
>>> 2011-06-27 12:26:56,800 | INFO | Thread-64 | JvmMetrics | Cannot
>>> initialize JVM Metrics with processName=JobTracker, sessionId= -
>>> already
>>> initialized
>>> 2011-06-27 12:26:56,847 | INFO | Thread-64 | TaskRunner |
>>> Task:attempt_local_0004_r_**000000_0 is done. And is in the
>>> process of
>>> commiting
>>> 2011-06-27 12:26:56,848 | INFO | Thread-64 | JvmMetrics | Cannot
>>> initialize JVM Metrics with processName=JobTracker, sessionId= -
>>> already
>>> initialized
>>> 2011-06-27 12:26:56,849 | INFO | Thread-64 | LocalJobRunner |
>>> 2011-06-27 12:26:56,849 | INFO | Thread-64 | TaskRunner | Task
>>> attempt_local_0004_r_000000_0 is allowed to commit now
>>> 2011-06-27 12:26:56,850 | INFO | Thread-64 | JvmMetrics | Cannot
>>> initialize JVM Metrics with processName=JobTracker, sessionId= -
>>> already
>>> initialized
>>> 2011-06-27 12:26:56,853 | INFO | Thread-64 |
>>> FileOutputCommitter | Saved
>>> output of task 'attempt_local_0004_r_000000_**0' to
>>> file:/tmp/temp-791598942/**tmp1907259615
>>> 2011-06-27 12:26:56,853 | INFO | Thread-64 | LocalJobRunner |
>>> reduce>
>>> reduce
>>> 2011-06-27 12:26:56,854 | INFO | Thread-64 | TaskRunner | Task
>>> 'attempt_local_0004_r_000000_**0' done.
>>>
>>>
>
Re: Removing local job traces in local mode
Posted by Ashutosh Chauhan <ha...@apache.org>.
Vincent,
Glad that you were able to solve the issue. Ideally, one should be able to
configure log4j externally through log4j.properties config file and not by
setting them explicitly in code. Did you try that?
Ashutosh
On Fri, Aug 26, 2011 at 07:56, Vincent Barat <vi...@gmail.com>wrote:
> Here is how I solved this issue (it was only related to log4j
> configuration):
>
> /* Deactivate most traces from Hadoop and PIG (keep ERROR) */
> props.setProperty("log4j.**logger.org.apache.hadoop", "ERROR");
> props.setProperty("log4j.**logger.org.apache.zookeeper", "ERROR");
> props.setProperty("log4j.**logger.org.apache.pig", "ERROR");
>
> /*
> * Keep traces from M/R launcher (percentage of advancement) and job
> statistics (nb of record
> * written etc.)
> */
> props.setProperty(
> "log4j.logger.org.apache.pig.**backend.hadoop.**executionengine.**
> mapReduceLayer.**MapReduceLauncher",
> "INFO");
> props.setProperty("log4j.**logger.org.apache.pig.tools.**pigstats.PigStats",
> "INFO");
>
>
> Le 27/06/11 13:21, Vincent Barat a écrit :
>
> Thanks, but it is actually not what I'm looking for.
>>
>> First, I use PIG from Java, not from the pig shell, so the "-d" option
>> cannot be used.
>> Second, I want to be able to choose between which trace I want, and which
>> I don't, depending on the origin of the trace.
>> I guess log4j can be configured for that, but I don't manage to configure
>> it correctly.
>>
>> Any hint ?
>>
>> Le 27/06/11 12:49, Vincent a écrit :
>>
>>> You can try -d option:
>>>
>>> For example pig -d WARN -x local myscript.pig
>>>
>>> On Mon, Jun 27, 2011 at 2:46 PM, Vincent Barat<vi...@gmail.com>*
>>> *wrote:
>>>
>>> Hi,
>>>>
>>>> I'm using PIG in local mode during development phases, and I get bored
>>>> about traces outputted by local jobs (the one attached).
>>>> Is there a way to get rid of them?
>>>>
>>>> Thanks for your help.
>>>>
>>>> 2011-06-27 12:26:56,289 | WARN | Thread-55 | JobClient | Use
>>>> GenericOptionsParser for parsing the arguments. Applications should
>>>> implement Tool for the same.
>>>> 2011-06-27 12:26:56,423 | INFO | Thread-55 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,427 | INFO | Thread-55 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,429 | INFO | Thread-55 | FileInputFormat | Total
>>>> input
>>>> paths to process : 1
>>>> 2011-06-27 12:26:56,429 | INFO | Thread-55 | MapRedUtil | Total input
>>>> paths to process : 1
>>>> 2011-06-27 12:26:56,429 | INFO | Thread-55 | MapRedUtil | Total input
>>>> paths (combined) to process : 1
>>>> 2011-06-27 12:26:56,614 | INFO | Thread-64 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,616 | INFO | Thread-64 | FileInputFormat | Total
>>>> input
>>>> paths to process : 1
>>>> 2011-06-27 12:26:56,616 | INFO | Thread-64 | MapRedUtil | Total input
>>>> paths to process : 1
>>>> 2011-06-27 12:26:56,616 | INFO | Thread-64 | MapRedUtil | Total input
>>>> paths (combined) to process : 1
>>>> 2011-06-27 12:26:56,653 | INFO | Thread-64 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,657 | INFO | Thread-64 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,658 | INFO | Thread-64 | MapTask | io.sort.mb =
>>>> 100
>>>> 2011-06-27 12:26:56,751 | INFO | Thread-64 | MapTask | data buffer =
>>>> 79691776/99614720
>>>> 2011-06-27 12:26:56,751 | INFO | Thread-64 | MapTask | record buffer
>>>> =
>>>> 262144/327680
>>>> 2011-06-27 12:26:56,775 | INFO | Thread-64 | MapTask | Starting
>>>> flush
>>>> of map output
>>>> 2011-06-27 12:26:56,785 | INFO | Thread-64 | MapTask | Finished
>>>> spill 0
>>>> 2011-06-27 12:26:56,790 | INFO | Thread-64 | LocalJobRunner |
>>>> 2011-06-27 12:26:56,790 | INFO | Thread-64 | TaskRunner | Task
>>>> 'attempt_local_0004_m_000000_****0' done.
>>>> 2011-06-27 12:26:56,795 | INFO | Thread-64 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,797 | INFO | Thread-64 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,797 | INFO | Thread-64 | LocalJobRunner |
>>>> 2011-06-27 12:26:56,798 | INFO | Thread-64 | Merger | Merging 1
>>>> sorted
>>>> segments
>>>> 2011-06-27 12:26:56,799 | INFO | Thread-64 | Merger | Down to the
>>>> last
>>>> merge-pass, with 1 segments left of total size: 350 bytes
>>>> 2011-06-27 12:26:56,799 | INFO | Thread-64 | LocalJobRunner |
>>>> 2011-06-27 12:26:56,800 | INFO | Thread-64 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,847 | INFO | Thread-64 | TaskRunner |
>>>> Task:attempt_local_0004_r_****000000_0 is done. And is in the process
>>>> of
>>>> commiting
>>>> 2011-06-27 12:26:56,848 | INFO | Thread-64 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,849 | INFO | Thread-64 | LocalJobRunner |
>>>> 2011-06-27 12:26:56,849 | INFO | Thread-64 | TaskRunner | Task
>>>> attempt_local_0004_r_000000_0 is allowed to commit now
>>>> 2011-06-27 12:26:56,850 | INFO | Thread-64 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,853 | INFO | Thread-64 | FileOutputCommitter |
>>>> Saved
>>>> output of task 'attempt_local_0004_r_000000_****0' to
>>>> file:/tmp/temp-791598942/****tmp1907259615
>>>> 2011-06-27 12:26:56,853 | INFO | Thread-64 | LocalJobRunner | reduce>
>>>> reduce
>>>> 2011-06-27 12:26:56,854 | INFO | Thread-64 | TaskRunner | Task
>>>> 'attempt_local_0004_r_000000_****0' done.
>>>>
>>>>
>>>>
>>