You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Vincent Barat <vi...@gmail.com> on 2011/06/27 12:46:12 UTC
Removing local job traces in local mode
Hi,
I'm using PIG in local mode during development phases, and I get
bored about traces outputted by local jobs (the one attached).
Is there a way to get rid of them?
Thanks for your help.
2011-06-27 12:26:56,289 | WARN | Thread-55 | JobClient | Use
GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2011-06-27 12:26:56,423 | INFO | Thread-55 | JvmMetrics | Cannot
initialize JVM Metrics with processName=JobTracker, sessionId= -
already initialized
2011-06-27 12:26:56,427 | INFO | Thread-55 | JvmMetrics | Cannot
initialize JVM Metrics with processName=JobTracker, sessionId= -
already initialized
2011-06-27 12:26:56,429 | INFO | Thread-55 | FileInputFormat |
Total input paths to process : 1
2011-06-27 12:26:56,429 | INFO | Thread-55 | MapRedUtil | Total
input paths to process : 1
2011-06-27 12:26:56,429 | INFO | Thread-55 | MapRedUtil | Total
input paths (combined) to process : 1
2011-06-27 12:26:56,614 | INFO | Thread-64 | JvmMetrics | Cannot
initialize JVM Metrics with processName=JobTracker, sessionId= -
already initialized
2011-06-27 12:26:56,616 | INFO | Thread-64 | FileInputFormat |
Total input paths to process : 1
2011-06-27 12:26:56,616 | INFO | Thread-64 | MapRedUtil | Total
input paths to process : 1
2011-06-27 12:26:56,616 | INFO | Thread-64 | MapRedUtil | Total
input paths (combined) to process : 1
2011-06-27 12:26:56,653 | INFO | Thread-64 | JvmMetrics | Cannot
initialize JVM Metrics with processName=JobTracker, sessionId= -
already initialized
2011-06-27 12:26:56,657 | INFO | Thread-64 | JvmMetrics | Cannot
initialize JVM Metrics with processName=JobTracker, sessionId= -
already initialized
2011-06-27 12:26:56,658 | INFO | Thread-64 | MapTask |
io.sort.mb = 100
2011-06-27 12:26:56,751 | INFO | Thread-64 | MapTask | data
buffer = 79691776/99614720
2011-06-27 12:26:56,751 | INFO | Thread-64 | MapTask | record
buffer = 262144/327680
2011-06-27 12:26:56,775 | INFO | Thread-64 | MapTask | Starting
flush of map output
2011-06-27 12:26:56,785 | INFO | Thread-64 | MapTask | Finished
spill 0
2011-06-27 12:26:56,790 | INFO | Thread-64 | LocalJobRunner |
2011-06-27 12:26:56,790 | INFO | Thread-64 | TaskRunner | Task
'attempt_local_0004_m_000000_0' done.
2011-06-27 12:26:56,795 | INFO | Thread-64 | JvmMetrics | Cannot
initialize JVM Metrics with processName=JobTracker, sessionId= -
already initialized
2011-06-27 12:26:56,797 | INFO | Thread-64 | JvmMetrics | Cannot
initialize JVM Metrics with processName=JobTracker, sessionId= -
already initialized
2011-06-27 12:26:56,797 | INFO | Thread-64 | LocalJobRunner |
2011-06-27 12:26:56,798 | INFO | Thread-64 | Merger | Merging 1
sorted segments
2011-06-27 12:26:56,799 | INFO | Thread-64 | Merger | Down to
the last merge-pass, with 1 segments left of total size: 350 bytes
2011-06-27 12:26:56,799 | INFO | Thread-64 | LocalJobRunner |
2011-06-27 12:26:56,800 | INFO | Thread-64 | JvmMetrics | Cannot
initialize JVM Metrics with processName=JobTracker, sessionId= -
already initialized
2011-06-27 12:26:56,847 | INFO | Thread-64 | TaskRunner |
Task:attempt_local_0004_r_000000_0 is done. And is in the process of
commiting
2011-06-27 12:26:56,848 | INFO | Thread-64 | JvmMetrics | Cannot
initialize JVM Metrics with processName=JobTracker, sessionId= -
already initialized
2011-06-27 12:26:56,849 | INFO | Thread-64 | LocalJobRunner |
2011-06-27 12:26:56,849 | INFO | Thread-64 | TaskRunner | Task
attempt_local_0004_r_000000_0 is allowed to commit now
2011-06-27 12:26:56,850 | INFO | Thread-64 | JvmMetrics | Cannot
initialize JVM Metrics with processName=JobTracker, sessionId= -
already initialized
2011-06-27 12:26:56,853 | INFO | Thread-64 | FileOutputCommitter |
Saved output of task 'attempt_local_0004_r_000000_0' to
file:/tmp/temp-791598942/tmp1907259615
2011-06-27 12:26:56,853 | INFO | Thread-64 | LocalJobRunner |
reduce > reduce
2011-06-27 12:26:56,854 | INFO | Thread-64 | TaskRunner | Task
'attempt_local_0004_r_000000_0' done.
Re: Removing local job traces in local mode
Posted by Ashutosh Chauhan <ha...@apache.org>.
Vincent,
Glad that you were able to solve the issue. Ideally, one should be able to
configure log4j externally through log4j.properties config file and not by
setting them explicitly in code. Did you try that?
Ashutosh
On Fri, Aug 26, 2011 at 07:56, Vincent Barat <vi...@gmail.com>wrote:
> Here is how I solved this issue (it was only related to log4j
> configuration):
>
> /* Deactivate most traces from Hadoop and PIG (keep ERROR) */
> props.setProperty("log4j.**logger.org.apache.hadoop", "ERROR");
> props.setProperty("log4j.**logger.org.apache.zookeeper", "ERROR");
> props.setProperty("log4j.**logger.org.apache.pig", "ERROR");
>
> /*
> * Keep traces from M/R launcher (percentage of advancement) and job
> statistics (nb of record
> * written etc.)
> */
> props.setProperty(
> "log4j.logger.org.apache.pig.**backend.hadoop.**executionengine.**
> mapReduceLayer.**MapReduceLauncher",
> "INFO");
> props.setProperty("log4j.**logger.org.apache.pig.tools.**pigstats.PigStats",
> "INFO");
>
>
> Le 27/06/11 13:21, Vincent Barat a écrit :
>
> Thanks, but it is actually not what I'm looking for.
>>
>> First, I use PIG from Java, not from the pig shell, so the "-d" option
>> cannot be used.
>> Second, I want to be able to choose between which trace I want, and which
>> I don't, depending on the origin of the trace.
>> I guess log4j can be configured for that, but I don't manage to configure
>> it correctly.
>>
>> Any hint ?
>>
>> Le 27/06/11 12:49, Vincent a écrit :
>>
>>> You can try -d option:
>>>
>>> For example pig -d WARN -x local myscript.pig
>>>
>>> On Mon, Jun 27, 2011 at 2:46 PM, Vincent Barat<vi...@gmail.com>*
>>> *wrote:
>>>
>>> Hi,
>>>>
>>>> I'm using PIG in local mode during development phases, and I get bored
>>>> about traces outputted by local jobs (the one attached).
>>>> Is there a way to get rid of them?
>>>>
>>>> Thanks for your help.
>>>>
>>>> 2011-06-27 12:26:56,289 | WARN | Thread-55 | JobClient | Use
>>>> GenericOptionsParser for parsing the arguments. Applications should
>>>> implement Tool for the same.
>>>> 2011-06-27 12:26:56,423 | INFO | Thread-55 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,427 | INFO | Thread-55 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,429 | INFO | Thread-55 | FileInputFormat | Total
>>>> input
>>>> paths to process : 1
>>>> 2011-06-27 12:26:56,429 | INFO | Thread-55 | MapRedUtil | Total input
>>>> paths to process : 1
>>>> 2011-06-27 12:26:56,429 | INFO | Thread-55 | MapRedUtil | Total input
>>>> paths (combined) to process : 1
>>>> 2011-06-27 12:26:56,614 | INFO | Thread-64 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,616 | INFO | Thread-64 | FileInputFormat | Total
>>>> input
>>>> paths to process : 1
>>>> 2011-06-27 12:26:56,616 | INFO | Thread-64 | MapRedUtil | Total input
>>>> paths to process : 1
>>>> 2011-06-27 12:26:56,616 | INFO | Thread-64 | MapRedUtil | Total input
>>>> paths (combined) to process : 1
>>>> 2011-06-27 12:26:56,653 | INFO | Thread-64 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,657 | INFO | Thread-64 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,658 | INFO | Thread-64 | MapTask | io.sort.mb =
>>>> 100
>>>> 2011-06-27 12:26:56,751 | INFO | Thread-64 | MapTask | data buffer =
>>>> 79691776/99614720
>>>> 2011-06-27 12:26:56,751 | INFO | Thread-64 | MapTask | record buffer
>>>> =
>>>> 262144/327680
>>>> 2011-06-27 12:26:56,775 | INFO | Thread-64 | MapTask | Starting
>>>> flush
>>>> of map output
>>>> 2011-06-27 12:26:56,785 | INFO | Thread-64 | MapTask | Finished
>>>> spill 0
>>>> 2011-06-27 12:26:56,790 | INFO | Thread-64 | LocalJobRunner |
>>>> 2011-06-27 12:26:56,790 | INFO | Thread-64 | TaskRunner | Task
>>>> 'attempt_local_0004_m_000000_****0' done.
>>>> 2011-06-27 12:26:56,795 | INFO | Thread-64 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,797 | INFO | Thread-64 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,797 | INFO | Thread-64 | LocalJobRunner |
>>>> 2011-06-27 12:26:56,798 | INFO | Thread-64 | Merger | Merging 1
>>>> sorted
>>>> segments
>>>> 2011-06-27 12:26:56,799 | INFO | Thread-64 | Merger | Down to the
>>>> last
>>>> merge-pass, with 1 segments left of total size: 350 bytes
>>>> 2011-06-27 12:26:56,799 | INFO | Thread-64 | LocalJobRunner |
>>>> 2011-06-27 12:26:56,800 | INFO | Thread-64 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,847 | INFO | Thread-64 | TaskRunner |
>>>> Task:attempt_local_0004_r_****000000_0 is done. And is in the process
>>>> of
>>>> commiting
>>>> 2011-06-27 12:26:56,848 | INFO | Thread-64 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,849 | INFO | Thread-64 | LocalJobRunner |
>>>> 2011-06-27 12:26:56,849 | INFO | Thread-64 | TaskRunner | Task
>>>> attempt_local_0004_r_000000_0 is allowed to commit now
>>>> 2011-06-27 12:26:56,850 | INFO | Thread-64 | JvmMetrics | Cannot
>>>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>>>> initialized
>>>> 2011-06-27 12:26:56,853 | INFO | Thread-64 | FileOutputCommitter |
>>>> Saved
>>>> output of task 'attempt_local_0004_r_000000_****0' to
>>>> file:/tmp/temp-791598942/****tmp1907259615
>>>> 2011-06-27 12:26:56,853 | INFO | Thread-64 | LocalJobRunner | reduce>
>>>> reduce
>>>> 2011-06-27 12:26:56,854 | INFO | Thread-64 | TaskRunner | Task
>>>> 'attempt_local_0004_r_000000_****0' done.
>>>>
>>>>
>>>>
>>
Re: Removing local job traces in local mode
Posted by Vincent Barat <vi...@gmail.com>.
Here is how I solved this issue (it was only related to log4j
configuration):
/* Deactivate most traces from Hadoop and PIG (keep ERROR) */
props.setProperty("log4j.logger.org.apache.hadoop", "ERROR");
props.setProperty("log4j.logger.org.apache.zookeeper", "ERROR");
props.setProperty("log4j.logger.org.apache.pig", "ERROR");
/*
* Keep traces from M/R launcher (percentage of advancement)
and job statistics (nb of record
* written etc.)
*/
props.setProperty(
"log4j.logger.org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher",
"INFO");
props.setProperty("log4j.logger.org.apache.pig.tools.pigstats.PigStats",
"INFO");
Le 27/06/11 13:21, Vincent Barat a écrit :
> Thanks, but it is actually not what I'm looking for.
>
> First, I use PIG from Java, not from the pig shell, so the "-d"
> option cannot be used.
> Second, I want to be able to choose between which trace I want,
> and which I don't, depending on the origin of the trace.
> I guess log4j can be configured for that, but I don't manage to
> configure it correctly.
>
> Any hint ?
>
> Le 27/06/11 12:49, Vincent a écrit :
>> You can try -d option:
>>
>> For example pig -d WARN -x local myscript.pig
>>
>> On Mon, Jun 27, 2011 at 2:46 PM, Vincent
>> Barat<vi...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> I'm using PIG in local mode during development phases, and I get
>>> bored
>>> about traces outputted by local jobs (the one attached).
>>> Is there a way to get rid of them?
>>>
>>> Thanks for your help.
>>>
>>> 2011-06-27 12:26:56,289 | WARN | Thread-55 | JobClient | Use
>>> GenericOptionsParser for parsing the arguments. Applications should
>>> implement Tool for the same.
>>> 2011-06-27 12:26:56,423 | INFO | Thread-55 | JvmMetrics | Cannot
>>> initialize JVM Metrics with processName=JobTracker, sessionId= -
>>> already
>>> initialized
>>> 2011-06-27 12:26:56,427 | INFO | Thread-55 | JvmMetrics | Cannot
>>> initialize JVM Metrics with processName=JobTracker, sessionId= -
>>> already
>>> initialized
>>> 2011-06-27 12:26:56,429 | INFO | Thread-55 | FileInputFormat |
>>> Total input
>>> paths to process : 1
>>> 2011-06-27 12:26:56,429 | INFO | Thread-55 | MapRedUtil | Total
>>> input
>>> paths to process : 1
>>> 2011-06-27 12:26:56,429 | INFO | Thread-55 | MapRedUtil | Total
>>> input
>>> paths (combined) to process : 1
>>> 2011-06-27 12:26:56,614 | INFO | Thread-64 | JvmMetrics | Cannot
>>> initialize JVM Metrics with processName=JobTracker, sessionId= -
>>> already
>>> initialized
>>> 2011-06-27 12:26:56,616 | INFO | Thread-64 | FileInputFormat |
>>> Total input
>>> paths to process : 1
>>> 2011-06-27 12:26:56,616 | INFO | Thread-64 | MapRedUtil | Total
>>> input
>>> paths to process : 1
>>> 2011-06-27 12:26:56,616 | INFO | Thread-64 | MapRedUtil | Total
>>> input
>>> paths (combined) to process : 1
>>> 2011-06-27 12:26:56,653 | INFO | Thread-64 | JvmMetrics | Cannot
>>> initialize JVM Metrics with processName=JobTracker, sessionId= -
>>> already
>>> initialized
>>> 2011-06-27 12:26:56,657 | INFO | Thread-64 | JvmMetrics | Cannot
>>> initialize JVM Metrics with processName=JobTracker, sessionId= -
>>> already
>>> initialized
>>> 2011-06-27 12:26:56,658 | INFO | Thread-64 | MapTask |
>>> io.sort.mb = 100
>>> 2011-06-27 12:26:56,751 | INFO | Thread-64 | MapTask | data
>>> buffer =
>>> 79691776/99614720
>>> 2011-06-27 12:26:56,751 | INFO | Thread-64 | MapTask |
>>> record buffer =
>>> 262144/327680
>>> 2011-06-27 12:26:56,775 | INFO | Thread-64 | MapTask |
>>> Starting flush
>>> of map output
>>> 2011-06-27 12:26:56,785 | INFO | Thread-64 | MapTask |
>>> Finished spill 0
>>> 2011-06-27 12:26:56,790 | INFO | Thread-64 | LocalJobRunner |
>>> 2011-06-27 12:26:56,790 | INFO | Thread-64 | TaskRunner | Task
>>> 'attempt_local_0004_m_000000_**0' done.
>>> 2011-06-27 12:26:56,795 | INFO | Thread-64 | JvmMetrics | Cannot
>>> initialize JVM Metrics with processName=JobTracker, sessionId= -
>>> already
>>> initialized
>>> 2011-06-27 12:26:56,797 | INFO | Thread-64 | JvmMetrics | Cannot
>>> initialize JVM Metrics with processName=JobTracker, sessionId= -
>>> already
>>> initialized
>>> 2011-06-27 12:26:56,797 | INFO | Thread-64 | LocalJobRunner |
>>> 2011-06-27 12:26:56,798 | INFO | Thread-64 | Merger |
>>> Merging 1 sorted
>>> segments
>>> 2011-06-27 12:26:56,799 | INFO | Thread-64 | Merger | Down
>>> to the last
>>> merge-pass, with 1 segments left of total size: 350 bytes
>>> 2011-06-27 12:26:56,799 | INFO | Thread-64 | LocalJobRunner |
>>> 2011-06-27 12:26:56,800 | INFO | Thread-64 | JvmMetrics | Cannot
>>> initialize JVM Metrics with processName=JobTracker, sessionId= -
>>> already
>>> initialized
>>> 2011-06-27 12:26:56,847 | INFO | Thread-64 | TaskRunner |
>>> Task:attempt_local_0004_r_**000000_0 is done. And is in the
>>> process of
>>> commiting
>>> 2011-06-27 12:26:56,848 | INFO | Thread-64 | JvmMetrics | Cannot
>>> initialize JVM Metrics with processName=JobTracker, sessionId= -
>>> already
>>> initialized
>>> 2011-06-27 12:26:56,849 | INFO | Thread-64 | LocalJobRunner |
>>> 2011-06-27 12:26:56,849 | INFO | Thread-64 | TaskRunner | Task
>>> attempt_local_0004_r_000000_0 is allowed to commit now
>>> 2011-06-27 12:26:56,850 | INFO | Thread-64 | JvmMetrics | Cannot
>>> initialize JVM Metrics with processName=JobTracker, sessionId= -
>>> already
>>> initialized
>>> 2011-06-27 12:26:56,853 | INFO | Thread-64 |
>>> FileOutputCommitter | Saved
>>> output of task 'attempt_local_0004_r_000000_**0' to
>>> file:/tmp/temp-791598942/**tmp1907259615
>>> 2011-06-27 12:26:56,853 | INFO | Thread-64 | LocalJobRunner |
>>> reduce>
>>> reduce
>>> 2011-06-27 12:26:56,854 | INFO | Thread-64 | TaskRunner | Task
>>> 'attempt_local_0004_r_000000_**0' done.
>>>
>>>
>
Re: Removing local job traces in local mode
Posted by Vincent Barat <vi...@gmail.com>.
Thanks, but it is actually not what I'm looking for.
First, I use PIG from Java, not from the pig shell, so the "-d"
option cannot be used.
Second, I want to be able to choose between which trace I want, and
which I don't, depending on the origin of the trace.
I guess log4j can be configured for that, but I don't manage to
configure it correctly.
Any hint ?
Le 27/06/11 12:49, Vincent a écrit :
> You can try -d option:
>
> For example pig -d WARN -x local myscript.pig
>
> On Mon, Jun 27, 2011 at 2:46 PM, Vincent Barat<vi...@gmail.com>wrote:
>
>> Hi,
>>
>> I'm using PIG in local mode during development phases, and I get bored
>> about traces outputted by local jobs (the one attached).
>> Is there a way to get rid of them?
>>
>> Thanks for your help.
>>
>> 2011-06-27 12:26:56,289 | WARN | Thread-55 | JobClient | Use
>> GenericOptionsParser for parsing the arguments. Applications should
>> implement Tool for the same.
>> 2011-06-27 12:26:56,423 | INFO | Thread-55 | JvmMetrics | Cannot
>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>> initialized
>> 2011-06-27 12:26:56,427 | INFO | Thread-55 | JvmMetrics | Cannot
>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>> initialized
>> 2011-06-27 12:26:56,429 | INFO | Thread-55 | FileInputFormat | Total input
>> paths to process : 1
>> 2011-06-27 12:26:56,429 | INFO | Thread-55 | MapRedUtil | Total input
>> paths to process : 1
>> 2011-06-27 12:26:56,429 | INFO | Thread-55 | MapRedUtil | Total input
>> paths (combined) to process : 1
>> 2011-06-27 12:26:56,614 | INFO | Thread-64 | JvmMetrics | Cannot
>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>> initialized
>> 2011-06-27 12:26:56,616 | INFO | Thread-64 | FileInputFormat | Total input
>> paths to process : 1
>> 2011-06-27 12:26:56,616 | INFO | Thread-64 | MapRedUtil | Total input
>> paths to process : 1
>> 2011-06-27 12:26:56,616 | INFO | Thread-64 | MapRedUtil | Total input
>> paths (combined) to process : 1
>> 2011-06-27 12:26:56,653 | INFO | Thread-64 | JvmMetrics | Cannot
>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>> initialized
>> 2011-06-27 12:26:56,657 | INFO | Thread-64 | JvmMetrics | Cannot
>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>> initialized
>> 2011-06-27 12:26:56,658 | INFO | Thread-64 | MapTask | io.sort.mb = 100
>> 2011-06-27 12:26:56,751 | INFO | Thread-64 | MapTask | data buffer =
>> 79691776/99614720
>> 2011-06-27 12:26:56,751 | INFO | Thread-64 | MapTask | record buffer =
>> 262144/327680
>> 2011-06-27 12:26:56,775 | INFO | Thread-64 | MapTask | Starting flush
>> of map output
>> 2011-06-27 12:26:56,785 | INFO | Thread-64 | MapTask | Finished spill 0
>> 2011-06-27 12:26:56,790 | INFO | Thread-64 | LocalJobRunner |
>> 2011-06-27 12:26:56,790 | INFO | Thread-64 | TaskRunner | Task
>> 'attempt_local_0004_m_000000_**0' done.
>> 2011-06-27 12:26:56,795 | INFO | Thread-64 | JvmMetrics | Cannot
>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>> initialized
>> 2011-06-27 12:26:56,797 | INFO | Thread-64 | JvmMetrics | Cannot
>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>> initialized
>> 2011-06-27 12:26:56,797 | INFO | Thread-64 | LocalJobRunner |
>> 2011-06-27 12:26:56,798 | INFO | Thread-64 | Merger | Merging 1 sorted
>> segments
>> 2011-06-27 12:26:56,799 | INFO | Thread-64 | Merger | Down to the last
>> merge-pass, with 1 segments left of total size: 350 bytes
>> 2011-06-27 12:26:56,799 | INFO | Thread-64 | LocalJobRunner |
>> 2011-06-27 12:26:56,800 | INFO | Thread-64 | JvmMetrics | Cannot
>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>> initialized
>> 2011-06-27 12:26:56,847 | INFO | Thread-64 | TaskRunner |
>> Task:attempt_local_0004_r_**000000_0 is done. And is in the process of
>> commiting
>> 2011-06-27 12:26:56,848 | INFO | Thread-64 | JvmMetrics | Cannot
>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>> initialized
>> 2011-06-27 12:26:56,849 | INFO | Thread-64 | LocalJobRunner |
>> 2011-06-27 12:26:56,849 | INFO | Thread-64 | TaskRunner | Task
>> attempt_local_0004_r_000000_0 is allowed to commit now
>> 2011-06-27 12:26:56,850 | INFO | Thread-64 | JvmMetrics | Cannot
>> initialize JVM Metrics with processName=JobTracker, sessionId= - already
>> initialized
>> 2011-06-27 12:26:56,853 | INFO | Thread-64 | FileOutputCommitter | Saved
>> output of task 'attempt_local_0004_r_000000_**0' to
>> file:/tmp/temp-791598942/**tmp1907259615
>> 2011-06-27 12:26:56,853 | INFO | Thread-64 | LocalJobRunner | reduce>
>> reduce
>> 2011-06-27 12:26:56,854 | INFO | Thread-64 | TaskRunner | Task
>> 'attempt_local_0004_r_000000_**0' done.
>>
>>
Re: Removing local job traces in local mode
Posted by Vincent <vi...@gmail.com>.
You can try -d option:
For example pig -d WARN -x local myscript.pig
On Mon, Jun 27, 2011 at 2:46 PM, Vincent Barat <vi...@gmail.com>wrote:
> Hi,
>
> I'm using PIG in local mode during development phases, and I get bored
> about traces outputted by local jobs (the one attached).
> Is there a way to get rid of them?
>
> Thanks for your help.
>
> 2011-06-27 12:26:56,289 | WARN | Thread-55 | JobClient | Use
> GenericOptionsParser for parsing the arguments. Applications should
> implement Tool for the same.
> 2011-06-27 12:26:56,423 | INFO | Thread-55 | JvmMetrics | Cannot
> initialize JVM Metrics with processName=JobTracker, sessionId= - already
> initialized
> 2011-06-27 12:26:56,427 | INFO | Thread-55 | JvmMetrics | Cannot
> initialize JVM Metrics with processName=JobTracker, sessionId= - already
> initialized
> 2011-06-27 12:26:56,429 | INFO | Thread-55 | FileInputFormat | Total input
> paths to process : 1
> 2011-06-27 12:26:56,429 | INFO | Thread-55 | MapRedUtil | Total input
> paths to process : 1
> 2011-06-27 12:26:56,429 | INFO | Thread-55 | MapRedUtil | Total input
> paths (combined) to process : 1
> 2011-06-27 12:26:56,614 | INFO | Thread-64 | JvmMetrics | Cannot
> initialize JVM Metrics with processName=JobTracker, sessionId= - already
> initialized
> 2011-06-27 12:26:56,616 | INFO | Thread-64 | FileInputFormat | Total input
> paths to process : 1
> 2011-06-27 12:26:56,616 | INFO | Thread-64 | MapRedUtil | Total input
> paths to process : 1
> 2011-06-27 12:26:56,616 | INFO | Thread-64 | MapRedUtil | Total input
> paths (combined) to process : 1
> 2011-06-27 12:26:56,653 | INFO | Thread-64 | JvmMetrics | Cannot
> initialize JVM Metrics with processName=JobTracker, sessionId= - already
> initialized
> 2011-06-27 12:26:56,657 | INFO | Thread-64 | JvmMetrics | Cannot
> initialize JVM Metrics with processName=JobTracker, sessionId= - already
> initialized
> 2011-06-27 12:26:56,658 | INFO | Thread-64 | MapTask | io.sort.mb = 100
> 2011-06-27 12:26:56,751 | INFO | Thread-64 | MapTask | data buffer =
> 79691776/99614720
> 2011-06-27 12:26:56,751 | INFO | Thread-64 | MapTask | record buffer =
> 262144/327680
> 2011-06-27 12:26:56,775 | INFO | Thread-64 | MapTask | Starting flush
> of map output
> 2011-06-27 12:26:56,785 | INFO | Thread-64 | MapTask | Finished spill 0
> 2011-06-27 12:26:56,790 | INFO | Thread-64 | LocalJobRunner |
> 2011-06-27 12:26:56,790 | INFO | Thread-64 | TaskRunner | Task
> 'attempt_local_0004_m_000000_**0' done.
> 2011-06-27 12:26:56,795 | INFO | Thread-64 | JvmMetrics | Cannot
> initialize JVM Metrics with processName=JobTracker, sessionId= - already
> initialized
> 2011-06-27 12:26:56,797 | INFO | Thread-64 | JvmMetrics | Cannot
> initialize JVM Metrics with processName=JobTracker, sessionId= - already
> initialized
> 2011-06-27 12:26:56,797 | INFO | Thread-64 | LocalJobRunner |
> 2011-06-27 12:26:56,798 | INFO | Thread-64 | Merger | Merging 1 sorted
> segments
> 2011-06-27 12:26:56,799 | INFO | Thread-64 | Merger | Down to the last
> merge-pass, with 1 segments left of total size: 350 bytes
> 2011-06-27 12:26:56,799 | INFO | Thread-64 | LocalJobRunner |
> 2011-06-27 12:26:56,800 | INFO | Thread-64 | JvmMetrics | Cannot
> initialize JVM Metrics with processName=JobTracker, sessionId= - already
> initialized
> 2011-06-27 12:26:56,847 | INFO | Thread-64 | TaskRunner |
> Task:attempt_local_0004_r_**000000_0 is done. And is in the process of
> commiting
> 2011-06-27 12:26:56,848 | INFO | Thread-64 | JvmMetrics | Cannot
> initialize JVM Metrics with processName=JobTracker, sessionId= - already
> initialized
> 2011-06-27 12:26:56,849 | INFO | Thread-64 | LocalJobRunner |
> 2011-06-27 12:26:56,849 | INFO | Thread-64 | TaskRunner | Task
> attempt_local_0004_r_000000_0 is allowed to commit now
> 2011-06-27 12:26:56,850 | INFO | Thread-64 | JvmMetrics | Cannot
> initialize JVM Metrics with processName=JobTracker, sessionId= - already
> initialized
> 2011-06-27 12:26:56,853 | INFO | Thread-64 | FileOutputCommitter | Saved
> output of task 'attempt_local_0004_r_000000_**0' to
> file:/tmp/temp-791598942/**tmp1907259615
> 2011-06-27 12:26:56,853 | INFO | Thread-64 | LocalJobRunner | reduce >
> reduce
> 2011-06-27 12:26:56,854 | INFO | Thread-64 | TaskRunner | Task
> 'attempt_local_0004_r_000000_**0' done.
>
>