You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by xeonmailinglist <xe...@gmail.com> on 2015/09/24 17:19:09 UTC
println in MapReduce job
Hi,
1.
I have this example of MapReduce [1], and I want to print info in
the stdout and in a log file. It seems that the logs isn’t print
anything. How can I make my class print these words?
2.
I also have set in the |yarn-site.xml| to retain log. Although the
logs are retained in the |/app-logs| dir, the |userlogs| dir is
deleted at the end of the job execution. How can I make MapReduce to
not delete files in the |userlogs| dir?
I am using Yarn.
Thanks,
[1] Wordcount exampla with just the map part.
|public class MyWordCount { public static class MyMap extends Mapper {
Log log = LogFactory.getLog(MyWordCount.class); private final static
IntWritable one = new IntWritable(1); private Text word = new Text();
public void map(LongWritable key, Text value, OutputCollector<Text,
IntWritable> output, Reporter reporter) throws IOException {
StringTokenizer itr = new StringTokenizer(value.toString());
System.out.println("HERRE"); log.info("HERRRRRE"); while
(itr.hasMoreTokens()) { word.set(itr.nextToken()); output.collect(word,
one); } } public void run(Context context) throws IOException,
InterruptedException { setup(context); try { while
(context.nextKeyValue()) { System.out.println("Key: " +
context.getCurrentKey() + " Value: " + context.getCurrentValue());
map(context.getCurrentKey(), context.getCurrentValue(), context); } }
finally { cleanup(context); } } public void cleanup(Mapper.Context
context) {} } |
[2] yarn-site.xml
|<!-- job history --> <property> <name>yarn.log-aggregation-enable</name>
<value>true</value> </property> <property>
<name>yarn.nodemanager.log.retain-seconds</name> <value>900000</value>
</property> <property> <name>yarn.nodemanager.remote-app-log-dir</name>
<value>/app-logs</value> </property> |
Re: println in MapReduce job
Posted by xeonmailinglist <xe...@gmail.com>.
I have found the error. It was a bug in my code.
On 09/24/2015 07:47 PM, xeonmailinglist wrote:
> No, I am looking to the right place. Here is the output of a maptask.
> I also thought that the print would come up, but it isn't showing.
> That is why I am asking.
>
>
> [1] Output from from one map task.
>
> ```
>
> Log Type: stderr
>
> Log Upload Time: 24-Sep-2015 12:45:19
>
> Log Length: 317
>
> Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
> It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
>
> Log Type: stdout
>
> Log Upload Time: 24-Sep-2015 12:45:19
>
> Log Length: 0
>
> Log Type: syslog
>
> Log Upload Time: 24-Sep-2015 12:45:19
>
> Log Length: 2604
>
> 2015-09-24 12:45:04,569 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 2015-09-24 12:45:05,139 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
> 2015-09-24 12:45:05,412 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
> 2015-09-24 12:45:05,413 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started
> 2015-09-24 12:45:05,462 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:
> 2015-09-24 12:45:05,463 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1443113036547_0001, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@1b5a082)
> 2015-09-24 12:45:05,847 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
> 2015-09-24 12:45:06,915 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /tmp/hadoop-temp/nm-local-dir/usercache/xubuntu/appcache/application_1443113036547_0001
> 2015-09-24 12:45:07,604 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
> 2015-09-24 12:45:09,402 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ]
> 2015-09-24 12:45:10,187 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hadoop-coc-1:9000/input1/b.txt:0+21
> 2015-09-24 12:45:10,812 INFO [main] org.apache.hadoop.mapred.Task: Task:attempt_1443113036547_0001_m_000000_0 is done. And is in the process of committing
> 2015-09-24 12:45:10,969 INFO [main] org.apache.hadoop.mapred.Task: Task attempt_1443113036547_0001_m_000000_0 is allowed to commit now
> 2015-09-24 12:45:10,993 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_1443113036547_0001_m_000000_0' to hdfs://192.168.10.110:9000/output1-1442847968/_temporary/1/task_1443113036547_0001_m_000000
> 2015-09-24 12:45:11,135 INFO [main] org.apache.hadoop.mapred.Task: Task 'attempt_1443113036547_0001_m_000000_0' done.
> 2015-09-24 12:45:11,135 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system...
> 2015-09-24 12:45:11,136 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system stopped.
> 2015-09-24 12:45:11,136 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system shutdown complete.
>
> ```
>
> On 09/24/2015 06:42 PM, Peyman Mohajerian wrote:
>> You log statement should show up in the container log for that Map
>> class within the data node. I'm guessing you aren't looking in the
>> right place.
>>
>> On Thu, Sep 24, 2015 at 9:54 AM, xeonmailinglist
>> <xeonmailinglist@gmail.com <ma...@gmail.com>> wrote:
>>
>> Does anyone know this question about logging in MapReduce. I
>> can't find an example or an explanation on how to print something
>> inside user map and reduce functions.
>>
>> On 09/24/2015 04:19 PM, xeonmailinglist wrote:
>>>
>>> Hi,
>>>
>>> 1.
>>>
>>> I have this example of MapReduce [1], and I want to print
>>> info in the stdout and in a log file. It seems that the logs
>>> isn’t print anything. How can I make my class print these words?
>>>
>>> 2.
>>>
>>> I also have set in the |yarn-site.xml| to retain log.
>>> Although the logs are retained in the |/app-logs| dir, the
>>> |userlogs| dir is deleted at the end of the job execution.
>>> How can I make MapReduce to not delete files in the
>>> |userlogs| dir?
>>>
>>> I am using Yarn.
>>>
>>> Thanks,
>>>
>>> [1] Wordcount exampla with just the map part.
>>>
>>> |public class MyWordCount { public static class MyMap extends
>>> Mapper { Log log = LogFactory.getLog(MyWordCount.class); private
>>> final static IntWritable one = new IntWritable(1); private Text
>>> word = new Text(); public void map(LongWritable key, Text value,
>>> OutputCollector<Text, IntWritable> output, Reporter reporter)
>>> throws IOException { StringTokenizer itr = new
>>> StringTokenizer(value.toString()); System.out.println("HERRE");
>>> log.info <http://log.info>("HERRRRRE"); while
>>> (itr.hasMoreTokens()) { word.set(itr.nextToken());
>>> output.collect(word, one); } } public void run(Context context)
>>> throws IOException, InterruptedException { setup(context); try {
>>> while (context.nextKeyValue()) { System.out.println("Key: " +
>>> context.getCurrentKey() + " Value: " +
>>> context.getCurrentValue()); map(context.getCurrentKey(),
>>> context.getCurrentValue(), context); } } finally {
>>> cleanup(context); } } public void cleanup(Mapper.Context
>>> context) {} } |
>>>
>>> [2] yarn-site.xml
>>>
>>> |<!-- job history --> <property>
>>> <name>yarn.log-aggregation-enable</name> <value>true</value>
>>> </property> <property>
>>> <name>yarn.nodemanager.log.retain-seconds</name>
>>> <value>900000</value> </property> <property>
>>> <name>yarn.nodemanager.remote-app-log-dir</name>
>>> <value>/app-logs</value> </property> |
>>>
>>
>>
>
Re: println in MapReduce job
Posted by xeonmailinglist <xe...@gmail.com>.
I have found the error. It was a bug in my code.
On 09/24/2015 07:47 PM, xeonmailinglist wrote:
> No, I am looking to the right place. Here is the output of a maptask.
> I also thought that the print would come up, but it isn't showing.
> That is why I am asking.
>
>
> [1] Output from from one map task.
>
> ```
>
> Log Type: stderr
>
> Log Upload Time: 24-Sep-2015 12:45:19
>
> Log Length: 317
>
> Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
> It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
>
> Log Type: stdout
>
> Log Upload Time: 24-Sep-2015 12:45:19
>
> Log Length: 0
>
> Log Type: syslog
>
> Log Upload Time: 24-Sep-2015 12:45:19
>
> Log Length: 2604
>
> 2015-09-24 12:45:04,569 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 2015-09-24 12:45:05,139 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
> 2015-09-24 12:45:05,412 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
> 2015-09-24 12:45:05,413 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started
> 2015-09-24 12:45:05,462 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:
> 2015-09-24 12:45:05,463 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1443113036547_0001, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@1b5a082)
> 2015-09-24 12:45:05,847 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
> 2015-09-24 12:45:06,915 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /tmp/hadoop-temp/nm-local-dir/usercache/xubuntu/appcache/application_1443113036547_0001
> 2015-09-24 12:45:07,604 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
> 2015-09-24 12:45:09,402 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ]
> 2015-09-24 12:45:10,187 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hadoop-coc-1:9000/input1/b.txt:0+21
> 2015-09-24 12:45:10,812 INFO [main] org.apache.hadoop.mapred.Task: Task:attempt_1443113036547_0001_m_000000_0 is done. And is in the process of committing
> 2015-09-24 12:45:10,969 INFO [main] org.apache.hadoop.mapred.Task: Task attempt_1443113036547_0001_m_000000_0 is allowed to commit now
> 2015-09-24 12:45:10,993 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_1443113036547_0001_m_000000_0' to hdfs://192.168.10.110:9000/output1-1442847968/_temporary/1/task_1443113036547_0001_m_000000
> 2015-09-24 12:45:11,135 INFO [main] org.apache.hadoop.mapred.Task: Task 'attempt_1443113036547_0001_m_000000_0' done.
> 2015-09-24 12:45:11,135 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system...
> 2015-09-24 12:45:11,136 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system stopped.
> 2015-09-24 12:45:11,136 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system shutdown complete.
>
> ```
>
> On 09/24/2015 06:42 PM, Peyman Mohajerian wrote:
>> You log statement should show up in the container log for that Map
>> class within the data node. I'm guessing you aren't looking in the
>> right place.
>>
>> On Thu, Sep 24, 2015 at 9:54 AM, xeonmailinglist
>> <xeonmailinglist@gmail.com <ma...@gmail.com>> wrote:
>>
>> Does anyone know this question about logging in MapReduce. I
>> can't find an example or an explanation on how to print something
>> inside user map and reduce functions.
>>
>> On 09/24/2015 04:19 PM, xeonmailinglist wrote:
>>>
>>> Hi,
>>>
>>> 1.
>>>
>>> I have this example of MapReduce [1], and I want to print
>>> info in the stdout and in a log file. It seems that the logs
>>> isn’t print anything. How can I make my class print these words?
>>>
>>> 2.
>>>
>>> I also have set in the |yarn-site.xml| to retain log.
>>> Although the logs are retained in the |/app-logs| dir, the
>>> |userlogs| dir is deleted at the end of the job execution.
>>> How can I make MapReduce to not delete files in the
>>> |userlogs| dir?
>>>
>>> I am using Yarn.
>>>
>>> Thanks,
>>>
>>> [1] Wordcount exampla with just the map part.
>>>
>>> |public class MyWordCount { public static class MyMap extends
>>> Mapper { Log log = LogFactory.getLog(MyWordCount.class); private
>>> final static IntWritable one = new IntWritable(1); private Text
>>> word = new Text(); public void map(LongWritable key, Text value,
>>> OutputCollector<Text, IntWritable> output, Reporter reporter)
>>> throws IOException { StringTokenizer itr = new
>>> StringTokenizer(value.toString()); System.out.println("HERRE");
>>> log.info <http://log.info>("HERRRRRE"); while
>>> (itr.hasMoreTokens()) { word.set(itr.nextToken());
>>> output.collect(word, one); } } public void run(Context context)
>>> throws IOException, InterruptedException { setup(context); try {
>>> while (context.nextKeyValue()) { System.out.println("Key: " +
>>> context.getCurrentKey() + " Value: " +
>>> context.getCurrentValue()); map(context.getCurrentKey(),
>>> context.getCurrentValue(), context); } } finally {
>>> cleanup(context); } } public void cleanup(Mapper.Context
>>> context) {} } |
>>>
>>> [2] yarn-site.xml
>>>
>>> |<!-- job history --> <property>
>>> <name>yarn.log-aggregation-enable</name> <value>true</value>
>>> </property> <property>
>>> <name>yarn.nodemanager.log.retain-seconds</name>
>>> <value>900000</value> </property> <property>
>>> <name>yarn.nodemanager.remote-app-log-dir</name>
>>> <value>/app-logs</value> </property> |
>>>
>>
>>
>
Re: println in MapReduce job
Posted by xeonmailinglist <xe...@gmail.com>.
I have found the error. It was a bug in my code.
On 09/24/2015 07:47 PM, xeonmailinglist wrote:
> No, I am looking to the right place. Here is the output of a maptask.
> I also thought that the print would come up, but it isn't showing.
> That is why I am asking.
>
>
> [1] Output from from one map task.
>
> ```
>
> Log Type: stderr
>
> Log Upload Time: 24-Sep-2015 12:45:19
>
> Log Length: 317
>
> Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
> It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
>
> Log Type: stdout
>
> Log Upload Time: 24-Sep-2015 12:45:19
>
> Log Length: 0
>
> Log Type: syslog
>
> Log Upload Time: 24-Sep-2015 12:45:19
>
> Log Length: 2604
>
> 2015-09-24 12:45:04,569 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 2015-09-24 12:45:05,139 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
> 2015-09-24 12:45:05,412 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
> 2015-09-24 12:45:05,413 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started
> 2015-09-24 12:45:05,462 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:
> 2015-09-24 12:45:05,463 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1443113036547_0001, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@1b5a082)
> 2015-09-24 12:45:05,847 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
> 2015-09-24 12:45:06,915 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /tmp/hadoop-temp/nm-local-dir/usercache/xubuntu/appcache/application_1443113036547_0001
> 2015-09-24 12:45:07,604 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
> 2015-09-24 12:45:09,402 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ]
> 2015-09-24 12:45:10,187 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hadoop-coc-1:9000/input1/b.txt:0+21
> 2015-09-24 12:45:10,812 INFO [main] org.apache.hadoop.mapred.Task: Task:attempt_1443113036547_0001_m_000000_0 is done. And is in the process of committing
> 2015-09-24 12:45:10,969 INFO [main] org.apache.hadoop.mapred.Task: Task attempt_1443113036547_0001_m_000000_0 is allowed to commit now
> 2015-09-24 12:45:10,993 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_1443113036547_0001_m_000000_0' to hdfs://192.168.10.110:9000/output1-1442847968/_temporary/1/task_1443113036547_0001_m_000000
> 2015-09-24 12:45:11,135 INFO [main] org.apache.hadoop.mapred.Task: Task 'attempt_1443113036547_0001_m_000000_0' done.
> 2015-09-24 12:45:11,135 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system...
> 2015-09-24 12:45:11,136 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system stopped.
> 2015-09-24 12:45:11,136 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system shutdown complete.
>
> ```
>
> On 09/24/2015 06:42 PM, Peyman Mohajerian wrote:
>> You log statement should show up in the container log for that Map
>> class within the data node. I'm guessing you aren't looking in the
>> right place.
>>
>> On Thu, Sep 24, 2015 at 9:54 AM, xeonmailinglist
>> <xeonmailinglist@gmail.com <ma...@gmail.com>> wrote:
>>
>> Does anyone know this question about logging in MapReduce. I
>> can't find an example or an explanation on how to print something
>> inside user map and reduce functions.
>>
>> On 09/24/2015 04:19 PM, xeonmailinglist wrote:
>>>
>>> Hi,
>>>
>>> 1.
>>>
>>> I have this example of MapReduce [1], and I want to print
>>> info in the stdout and in a log file. It seems that the logs
>>> isn’t print anything. How can I make my class print these words?
>>>
>>> 2.
>>>
>>> I also have set in the |yarn-site.xml| to retain log.
>>> Although the logs are retained in the |/app-logs| dir, the
>>> |userlogs| dir is deleted at the end of the job execution.
>>> How can I make MapReduce to not delete files in the
>>> |userlogs| dir?
>>>
>>> I am using Yarn.
>>>
>>> Thanks,
>>>
>>> [1] Wordcount exampla with just the map part.
>>>
>>> |public class MyWordCount { public static class MyMap extends
>>> Mapper { Log log = LogFactory.getLog(MyWordCount.class); private
>>> final static IntWritable one = new IntWritable(1); private Text
>>> word = new Text(); public void map(LongWritable key, Text value,
>>> OutputCollector<Text, IntWritable> output, Reporter reporter)
>>> throws IOException { StringTokenizer itr = new
>>> StringTokenizer(value.toString()); System.out.println("HERRE");
>>> log.info <http://log.info>("HERRRRRE"); while
>>> (itr.hasMoreTokens()) { word.set(itr.nextToken());
>>> output.collect(word, one); } } public void run(Context context)
>>> throws IOException, InterruptedException { setup(context); try {
>>> while (context.nextKeyValue()) { System.out.println("Key: " +
>>> context.getCurrentKey() + " Value: " +
>>> context.getCurrentValue()); map(context.getCurrentKey(),
>>> context.getCurrentValue(), context); } } finally {
>>> cleanup(context); } } public void cleanup(Mapper.Context
>>> context) {} } |
>>>
>>> [2] yarn-site.xml
>>>
>>> |<!-- job history --> <property>
>>> <name>yarn.log-aggregation-enable</name> <value>true</value>
>>> </property> <property>
>>> <name>yarn.nodemanager.log.retain-seconds</name>
>>> <value>900000</value> </property> <property>
>>> <name>yarn.nodemanager.remote-app-log-dir</name>
>>> <value>/app-logs</value> </property> |
>>>
>>
>>
>
Re: println in MapReduce job
Posted by xeonmailinglist <xe...@gmail.com>.
I have found the error. It was a bug in my code.
On 09/24/2015 07:47 PM, xeonmailinglist wrote:
> No, I am looking to the right place. Here is the output of a maptask.
> I also thought that the print would come up, but it isn't showing.
> That is why I am asking.
>
>
> [1] Output from from one map task.
>
> ```
>
> Log Type: stderr
>
> Log Upload Time: 24-Sep-2015 12:45:19
>
> Log Length: 317
>
> Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
> It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
>
> Log Type: stdout
>
> Log Upload Time: 24-Sep-2015 12:45:19
>
> Log Length: 0
>
> Log Type: syslog
>
> Log Upload Time: 24-Sep-2015 12:45:19
>
> Log Length: 2604
>
> 2015-09-24 12:45:04,569 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 2015-09-24 12:45:05,139 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
> 2015-09-24 12:45:05,412 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
> 2015-09-24 12:45:05,413 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started
> 2015-09-24 12:45:05,462 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:
> 2015-09-24 12:45:05,463 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1443113036547_0001, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@1b5a082)
> 2015-09-24 12:45:05,847 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
> 2015-09-24 12:45:06,915 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /tmp/hadoop-temp/nm-local-dir/usercache/xubuntu/appcache/application_1443113036547_0001
> 2015-09-24 12:45:07,604 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
> 2015-09-24 12:45:09,402 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ]
> 2015-09-24 12:45:10,187 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hadoop-coc-1:9000/input1/b.txt:0+21
> 2015-09-24 12:45:10,812 INFO [main] org.apache.hadoop.mapred.Task: Task:attempt_1443113036547_0001_m_000000_0 is done. And is in the process of committing
> 2015-09-24 12:45:10,969 INFO [main] org.apache.hadoop.mapred.Task: Task attempt_1443113036547_0001_m_000000_0 is allowed to commit now
> 2015-09-24 12:45:10,993 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_1443113036547_0001_m_000000_0' to hdfs://192.168.10.110:9000/output1-1442847968/_temporary/1/task_1443113036547_0001_m_000000
> 2015-09-24 12:45:11,135 INFO [main] org.apache.hadoop.mapred.Task: Task 'attempt_1443113036547_0001_m_000000_0' done.
> 2015-09-24 12:45:11,135 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system...
> 2015-09-24 12:45:11,136 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system stopped.
> 2015-09-24 12:45:11,136 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system shutdown complete.
>
> ```
>
> On 09/24/2015 06:42 PM, Peyman Mohajerian wrote:
>> You log statement should show up in the container log for that Map
>> class within the data node. I'm guessing you aren't looking in the
>> right place.
>>
>> On Thu, Sep 24, 2015 at 9:54 AM, xeonmailinglist
>> <xeonmailinglist@gmail.com <ma...@gmail.com>> wrote:
>>
>> Does anyone know this question about logging in MapReduce. I
>> can't find an example or an explanation on how to print something
>> inside user map and reduce functions.
>>
>> On 09/24/2015 04:19 PM, xeonmailinglist wrote:
>>>
>>> Hi,
>>>
>>> 1.
>>>
>>> I have this example of MapReduce [1], and I want to print
>>> info in the stdout and in a log file. It seems that the logs
>>> isn’t print anything. How can I make my class print these words?
>>>
>>> 2.
>>>
>>> I also have set in the |yarn-site.xml| to retain log.
>>> Although the logs are retained in the |/app-logs| dir, the
>>> |userlogs| dir is deleted at the end of the job execution.
>>> How can I make MapReduce to not delete files in the
>>> |userlogs| dir?
>>>
>>> I am using Yarn.
>>>
>>> Thanks,
>>>
>>> [1] Wordcount exampla with just the map part.
>>>
>>> |public class MyWordCount { public static class MyMap extends
>>> Mapper { Log log = LogFactory.getLog(MyWordCount.class); private
>>> final static IntWritable one = new IntWritable(1); private Text
>>> word = new Text(); public void map(LongWritable key, Text value,
>>> OutputCollector<Text, IntWritable> output, Reporter reporter)
>>> throws IOException { StringTokenizer itr = new
>>> StringTokenizer(value.toString()); System.out.println("HERRE");
>>> log.info <http://log.info>("HERRRRRE"); while
>>> (itr.hasMoreTokens()) { word.set(itr.nextToken());
>>> output.collect(word, one); } } public void run(Context context)
>>> throws IOException, InterruptedException { setup(context); try {
>>> while (context.nextKeyValue()) { System.out.println("Key: " +
>>> context.getCurrentKey() + " Value: " +
>>> context.getCurrentValue()); map(context.getCurrentKey(),
>>> context.getCurrentValue(), context); } } finally {
>>> cleanup(context); } } public void cleanup(Mapper.Context
>>> context) {} } |
>>>
>>> [2] yarn-site.xml
>>>
>>> |<!-- job history --> <property>
>>> <name>yarn.log-aggregation-enable</name> <value>true</value>
>>> </property> <property>
>>> <name>yarn.nodemanager.log.retain-seconds</name>
>>> <value>900000</value> </property> <property>
>>> <name>yarn.nodemanager.remote-app-log-dir</name>
>>> <value>/app-logs</value> </property> |
>>>
>>
>>
>
Re: println in MapReduce job
Posted by xeonmailinglist <xe...@gmail.com>.
No, I am looking to the right place. Here is the output of a maptask. I
also thought that the print would come up, but it isn't showing. That is
why I am asking.
[1] Output from from one map task.
```
Log Type: stderr
Log Upload Time: 24-Sep-2015 12:45:19
Log Length: 317
Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
Log Type: stdout
Log Upload Time: 24-Sep-2015 12:45:19
Log Length: 0
Log Type: syslog
Log Upload Time: 24-Sep-2015 12:45:19
Log Length: 2604
2015-09-24 12:45:04,569 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-09-24 12:45:05,139 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2015-09-24 12:45:05,412 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2015-09-24 12:45:05,413 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started
2015-09-24 12:45:05,462 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:
2015-09-24 12:45:05,463 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1443113036547_0001, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@1b5a082)
2015-09-24 12:45:05,847 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
2015-09-24 12:45:06,915 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /tmp/hadoop-temp/nm-local-dir/usercache/xubuntu/appcache/application_1443113036547_0001
2015-09-24 12:45:07,604 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2015-09-24 12:45:09,402 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2015-09-24 12:45:10,187 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hadoop-coc-1:9000/input1/b.txt:0+21
2015-09-24 12:45:10,812 INFO [main] org.apache.hadoop.mapred.Task: Task:attempt_1443113036547_0001_m_000000_0 is done. And is in the process of committing
2015-09-24 12:45:10,969 INFO [main] org.apache.hadoop.mapred.Task: Task attempt_1443113036547_0001_m_000000_0 is allowed to commit now
2015-09-24 12:45:10,993 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_1443113036547_0001_m_000000_0' to hdfs://192.168.10.110:9000/output1-1442847968/_temporary/1/task_1443113036547_0001_m_000000
2015-09-24 12:45:11,135 INFO [main] org.apache.hadoop.mapred.Task: Task 'attempt_1443113036547_0001_m_000000_0' done.
2015-09-24 12:45:11,135 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system...
2015-09-24 12:45:11,136 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system stopped.
2015-09-24 12:45:11,136 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system shutdown complete.
```
On 09/24/2015 06:42 PM, Peyman Mohajerian wrote:
> You log statement should show up in the container log for that Map
> class within the data node. I'm guessing you aren't looking in the
> right place.
>
> On Thu, Sep 24, 2015 at 9:54 AM, xeonmailinglist
> <xeonmailinglist@gmail.com <ma...@gmail.com>> wrote:
>
> Does anyone know this question about logging in MapReduce. I can't
> find an example or an explanation on how to print something inside
> user map and reduce functions.
>
> On 09/24/2015 04:19 PM, xeonmailinglist wrote:
>>
>> Hi,
>>
>> 1.
>>
>> I have this example of MapReduce [1], and I want to print
>> info in the stdout and in a log file. It seems that the logs
>> isn’t print anything. How can I make my class print these words?
>>
>> 2.
>>
>> I also have set in the |yarn-site.xml| to retain log.
>> Although the logs are retained in the |/app-logs| dir, the
>> |userlogs| dir is deleted at the end of the job execution.
>> How can I make MapReduce to not delete files in the
>> |userlogs| dir?
>>
>> I am using Yarn.
>>
>> Thanks,
>>
>> [1] Wordcount exampla with just the map part.
>>
>> |public class MyWordCount { public static class MyMap extends
>> Mapper { Log log = LogFactory.getLog(MyWordCount.class); private
>> final static IntWritable one = new IntWritable(1); private Text
>> word = new Text(); public void map(LongWritable key, Text value,
>> OutputCollector<Text, IntWritable> output, Reporter reporter)
>> throws IOException { StringTokenizer itr = new
>> StringTokenizer(value.toString()); System.out.println("HERRE");
>> log.info <http://log.info>("HERRRRRE"); while
>> (itr.hasMoreTokens()) { word.set(itr.nextToken());
>> output.collect(word, one); } } public void run(Context context)
>> throws IOException, InterruptedException { setup(context); try {
>> while (context.nextKeyValue()) { System.out.println("Key: " +
>> context.getCurrentKey() + " Value: " +
>> context.getCurrentValue()); map(context.getCurrentKey(),
>> context.getCurrentValue(), context); } } finally {
>> cleanup(context); } } public void cleanup(Mapper.Context context)
>> {} } |
>>
>> [2] yarn-site.xml
>>
>> |<!-- job history --> <property>
>> <name>yarn.log-aggregation-enable</name> <value>true</value>
>> </property> <property>
>> <name>yarn.nodemanager.log.retain-seconds</name>
>> <value>900000</value> </property> <property>
>> <name>yarn.nodemanager.remote-app-log-dir</name>
>> <value>/app-logs</value> </property> |
>>
>
>
Re: println in MapReduce job
Posted by xeonmailinglist <xe...@gmail.com>.
No, I am looking to the right place. Here is the output of a maptask. I
also thought that the print would come up, but it isn't showing. That is
why I am asking.
[1] Output from from one map task.
```
Log Type: stderr
Log Upload Time: 24-Sep-2015 12:45:19
Log Length: 317
Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
Log Type: stdout
Log Upload Time: 24-Sep-2015 12:45:19
Log Length: 0
Log Type: syslog
Log Upload Time: 24-Sep-2015 12:45:19
Log Length: 2604
2015-09-24 12:45:04,569 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-09-24 12:45:05,139 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2015-09-24 12:45:05,412 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2015-09-24 12:45:05,413 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started
2015-09-24 12:45:05,462 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:
2015-09-24 12:45:05,463 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1443113036547_0001, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@1b5a082)
2015-09-24 12:45:05,847 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
2015-09-24 12:45:06,915 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /tmp/hadoop-temp/nm-local-dir/usercache/xubuntu/appcache/application_1443113036547_0001
2015-09-24 12:45:07,604 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2015-09-24 12:45:09,402 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2015-09-24 12:45:10,187 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hadoop-coc-1:9000/input1/b.txt:0+21
2015-09-24 12:45:10,812 INFO [main] org.apache.hadoop.mapred.Task: Task:attempt_1443113036547_0001_m_000000_0 is done. And is in the process of committing
2015-09-24 12:45:10,969 INFO [main] org.apache.hadoop.mapred.Task: Task attempt_1443113036547_0001_m_000000_0 is allowed to commit now
2015-09-24 12:45:10,993 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_1443113036547_0001_m_000000_0' to hdfs://192.168.10.110:9000/output1-1442847968/_temporary/1/task_1443113036547_0001_m_000000
2015-09-24 12:45:11,135 INFO [main] org.apache.hadoop.mapred.Task: Task 'attempt_1443113036547_0001_m_000000_0' done.
2015-09-24 12:45:11,135 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system...
2015-09-24 12:45:11,136 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system stopped.
2015-09-24 12:45:11,136 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system shutdown complete.
```
On 09/24/2015 06:42 PM, Peyman Mohajerian wrote:
> You log statement should show up in the container log for that Map
> class within the data node. I'm guessing you aren't looking in the
> right place.
>
> On Thu, Sep 24, 2015 at 9:54 AM, xeonmailinglist
> <xeonmailinglist@gmail.com <ma...@gmail.com>> wrote:
>
> Does anyone know this question about logging in MapReduce. I can't
> find an example or an explanation on how to print something inside
> user map and reduce functions.
>
> On 09/24/2015 04:19 PM, xeonmailinglist wrote:
>>
>> Hi,
>>
>> 1.
>>
>> I have this example of MapReduce [1], and I want to print
>> info in the stdout and in a log file. It seems that the logs
>> isn’t print anything. How can I make my class print these words?
>>
>> 2.
>>
>> I also have set in the |yarn-site.xml| to retain log.
>> Although the logs are retained in the |/app-logs| dir, the
>> |userlogs| dir is deleted at the end of the job execution.
>> How can I make MapReduce to not delete files in the
>> |userlogs| dir?
>>
>> I am using Yarn.
>>
>> Thanks,
>>
>> [1] Wordcount exampla with just the map part.
>>
>> |public class MyWordCount { public static class MyMap extends
>> Mapper { Log log = LogFactory.getLog(MyWordCount.class); private
>> final static IntWritable one = new IntWritable(1); private Text
>> word = new Text(); public void map(LongWritable key, Text value,
>> OutputCollector<Text, IntWritable> output, Reporter reporter)
>> throws IOException { StringTokenizer itr = new
>> StringTokenizer(value.toString()); System.out.println("HERRE");
>> log.info <http://log.info>("HERRRRRE"); while
>> (itr.hasMoreTokens()) { word.set(itr.nextToken());
>> output.collect(word, one); } } public void run(Context context)
>> throws IOException, InterruptedException { setup(context); try {
>> while (context.nextKeyValue()) { System.out.println("Key: " +
>> context.getCurrentKey() + " Value: " +
>> context.getCurrentValue()); map(context.getCurrentKey(),
>> context.getCurrentValue(), context); } } finally {
>> cleanup(context); } } public void cleanup(Mapper.Context context)
>> {} } |
>>
>> [2] yarn-site.xml
>>
>> |<!-- job history --> <property>
>> <name>yarn.log-aggregation-enable</name> <value>true</value>
>> </property> <property>
>> <name>yarn.nodemanager.log.retain-seconds</name>
>> <value>900000</value> </property> <property>
>> <name>yarn.nodemanager.remote-app-log-dir</name>
>> <value>/app-logs</value> </property> |
>>
>
>
Re: println in MapReduce job
Posted by xeonmailinglist <xe...@gmail.com>.
No, I am looking to the right place. Here is the output of a maptask. I
also thought that the print would come up, but it isn't showing. That is
why I am asking.
[1] Output from from one map task.
```
Log Type: stderr
Log Upload Time: 24-Sep-2015 12:45:19
Log Length: 317
Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
Log Type: stdout
Log Upload Time: 24-Sep-2015 12:45:19
Log Length: 0
Log Type: syslog
Log Upload Time: 24-Sep-2015 12:45:19
Log Length: 2604
2015-09-24 12:45:04,569 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-09-24 12:45:05,139 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2015-09-24 12:45:05,412 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2015-09-24 12:45:05,413 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started
2015-09-24 12:45:05,462 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:
2015-09-24 12:45:05,463 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1443113036547_0001, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@1b5a082)
2015-09-24 12:45:05,847 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
2015-09-24 12:45:06,915 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /tmp/hadoop-temp/nm-local-dir/usercache/xubuntu/appcache/application_1443113036547_0001
2015-09-24 12:45:07,604 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2015-09-24 12:45:09,402 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2015-09-24 12:45:10,187 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hadoop-coc-1:9000/input1/b.txt:0+21
2015-09-24 12:45:10,812 INFO [main] org.apache.hadoop.mapred.Task: Task:attempt_1443113036547_0001_m_000000_0 is done. And is in the process of committing
2015-09-24 12:45:10,969 INFO [main] org.apache.hadoop.mapred.Task: Task attempt_1443113036547_0001_m_000000_0 is allowed to commit now
2015-09-24 12:45:10,993 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_1443113036547_0001_m_000000_0' to hdfs://192.168.10.110:9000/output1-1442847968/_temporary/1/task_1443113036547_0001_m_000000
2015-09-24 12:45:11,135 INFO [main] org.apache.hadoop.mapred.Task: Task 'attempt_1443113036547_0001_m_000000_0' done.
2015-09-24 12:45:11,135 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system...
2015-09-24 12:45:11,136 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system stopped.
2015-09-24 12:45:11,136 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system shutdown complete.
```
On 09/24/2015 06:42 PM, Peyman Mohajerian wrote:
> You log statement should show up in the container log for that Map
> class within the data node. I'm guessing you aren't looking in the
> right place.
>
> On Thu, Sep 24, 2015 at 9:54 AM, xeonmailinglist
> <xeonmailinglist@gmail.com <ma...@gmail.com>> wrote:
>
> Does anyone know this question about logging in MapReduce. I can't
> find an example or an explanation on how to print something inside
> user map and reduce functions.
>
> On 09/24/2015 04:19 PM, xeonmailinglist wrote:
>>
>> Hi,
>>
>> 1.
>>
>> I have this example of MapReduce [1], and I want to print
>> info in the stdout and in a log file. It seems that the logs
>> isn’t print anything. How can I make my class print these words?
>>
>> 2.
>>
>> I also have set in the |yarn-site.xml| to retain log.
>> Although the logs are retained in the |/app-logs| dir, the
>> |userlogs| dir is deleted at the end of the job execution.
>> How can I make MapReduce to not delete files in the
>> |userlogs| dir?
>>
>> I am using Yarn.
>>
>> Thanks,
>>
>> [1] Wordcount exampla with just the map part.
>>
>> |public class MyWordCount { public static class MyMap extends
>> Mapper { Log log = LogFactory.getLog(MyWordCount.class); private
>> final static IntWritable one = new IntWritable(1); private Text
>> word = new Text(); public void map(LongWritable key, Text value,
>> OutputCollector<Text, IntWritable> output, Reporter reporter)
>> throws IOException { StringTokenizer itr = new
>> StringTokenizer(value.toString()); System.out.println("HERRE");
>> log.info <http://log.info>("HERRRRRE"); while
>> (itr.hasMoreTokens()) { word.set(itr.nextToken());
>> output.collect(word, one); } } public void run(Context context)
>> throws IOException, InterruptedException { setup(context); try {
>> while (context.nextKeyValue()) { System.out.println("Key: " +
>> context.getCurrentKey() + " Value: " +
>> context.getCurrentValue()); map(context.getCurrentKey(),
>> context.getCurrentValue(), context); } } finally {
>> cleanup(context); } } public void cleanup(Mapper.Context context)
>> {} } |
>>
>> [2] yarn-site.xml
>>
>> |<!-- job history --> <property>
>> <name>yarn.log-aggregation-enable</name> <value>true</value>
>> </property> <property>
>> <name>yarn.nodemanager.log.retain-seconds</name>
>> <value>900000</value> </property> <property>
>> <name>yarn.nodemanager.remote-app-log-dir</name>
>> <value>/app-logs</value> </property> |
>>
>
>
Re: println in MapReduce job
Posted by xeonmailinglist <xe...@gmail.com>.
No, I am looking to the right place. Here is the output of a maptask. I
also thought that the print would come up, but it isn't showing. That is
why I am asking.
[1] Output from from one map task.
```
Log Type: stderr
Log Upload Time: 24-Sep-2015 12:45:19
Log Length: 317
Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
Log Type: stdout
Log Upload Time: 24-Sep-2015 12:45:19
Log Length: 0
Log Type: syslog
Log Upload Time: 24-Sep-2015 12:45:19
Log Length: 2604
2015-09-24 12:45:04,569 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-09-24 12:45:05,139 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2015-09-24 12:45:05,412 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2015-09-24 12:45:05,413 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started
2015-09-24 12:45:05,462 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:
2015-09-24 12:45:05,463 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1443113036547_0001, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@1b5a082)
2015-09-24 12:45:05,847 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
2015-09-24 12:45:06,915 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /tmp/hadoop-temp/nm-local-dir/usercache/xubuntu/appcache/application_1443113036547_0001
2015-09-24 12:45:07,604 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2015-09-24 12:45:09,402 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2015-09-24 12:45:10,187 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hadoop-coc-1:9000/input1/b.txt:0+21
2015-09-24 12:45:10,812 INFO [main] org.apache.hadoop.mapred.Task: Task:attempt_1443113036547_0001_m_000000_0 is done. And is in the process of committing
2015-09-24 12:45:10,969 INFO [main] org.apache.hadoop.mapred.Task: Task attempt_1443113036547_0001_m_000000_0 is allowed to commit now
2015-09-24 12:45:10,993 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_1443113036547_0001_m_000000_0' to hdfs://192.168.10.110:9000/output1-1442847968/_temporary/1/task_1443113036547_0001_m_000000
2015-09-24 12:45:11,135 INFO [main] org.apache.hadoop.mapred.Task: Task 'attempt_1443113036547_0001_m_000000_0' done.
2015-09-24 12:45:11,135 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system...
2015-09-24 12:45:11,136 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system stopped.
2015-09-24 12:45:11,136 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system shutdown complete.
```
On 09/24/2015 06:42 PM, Peyman Mohajerian wrote:
> You log statement should show up in the container log for that Map
> class within the data node. I'm guessing you aren't looking in the
> right place.
>
> On Thu, Sep 24, 2015 at 9:54 AM, xeonmailinglist
> <xeonmailinglist@gmail.com <ma...@gmail.com>> wrote:
>
> Does anyone know this question about logging in MapReduce. I can't
> find an example or an explanation on how to print something inside
> user map and reduce functions.
>
> On 09/24/2015 04:19 PM, xeonmailinglist wrote:
>>
>> Hi,
>>
>> 1.
>>
>> I have this example of MapReduce [1], and I want to print
>> info in the stdout and in a log file. It seems that the logs
>> isn’t print anything. How can I make my class print these words?
>>
>> 2.
>>
>> I also have set in the |yarn-site.xml| to retain log.
>> Although the logs are retained in the |/app-logs| dir, the
>> |userlogs| dir is deleted at the end of the job execution.
>> How can I make MapReduce to not delete files in the
>> |userlogs| dir?
>>
>> I am using Yarn.
>>
>> Thanks,
>>
>> [1] Wordcount exampla with just the map part.
>>
>> |public class MyWordCount { public static class MyMap extends
>> Mapper { Log log = LogFactory.getLog(MyWordCount.class); private
>> final static IntWritable one = new IntWritable(1); private Text
>> word = new Text(); public void map(LongWritable key, Text value,
>> OutputCollector<Text, IntWritable> output, Reporter reporter)
>> throws IOException { StringTokenizer itr = new
>> StringTokenizer(value.toString()); System.out.println("HERRE");
>> log.info <http://log.info>("HERRRRRE"); while
>> (itr.hasMoreTokens()) { word.set(itr.nextToken());
>> output.collect(word, one); } } public void run(Context context)
>> throws IOException, InterruptedException { setup(context); try {
>> while (context.nextKeyValue()) { System.out.println("Key: " +
>> context.getCurrentKey() + " Value: " +
>> context.getCurrentValue()); map(context.getCurrentKey(),
>> context.getCurrentValue(), context); } } finally {
>> cleanup(context); } } public void cleanup(Mapper.Context context)
>> {} } |
>>
>> [2] yarn-site.xml
>>
>> |<!-- job history --> <property>
>> <name>yarn.log-aggregation-enable</name> <value>true</value>
>> </property> <property>
>> <name>yarn.nodemanager.log.retain-seconds</name>
>> <value>900000</value> </property> <property>
>> <name>yarn.nodemanager.remote-app-log-dir</name>
>> <value>/app-logs</value> </property> |
>>
>
>
Re: println in MapReduce job
Posted by Peyman Mohajerian <mo...@gmail.com>.
You log statement should show up in the container log for that Map class
within the data node. I'm guessing you aren't looking in the right place.
On Thu, Sep 24, 2015 at 9:54 AM, xeonmailinglist <xe...@gmail.com>
wrote:
> Does anyone know this question about logging in MapReduce. I can't find an
> example or an explanation on how to print something inside user map and
> reduce functions.
>
> On 09/24/2015 04:19 PM, xeonmailinglist wrote:
>
> Hi,
>
> 1.
>
> I have this example of MapReduce [1], and I want to print info in the
> stdout and in a log file. It seems that the logs isn’t print anything. How
> can I make my class print these words?
> 2.
>
> I also have set in the yarn-site.xml to retain log. Although the logs
> are retained in the /app-logs dir, the userlogs dir is deleted at the
> end of the job execution. How can I make MapReduce to not delete files in
> the userlogs dir?
>
> I am using Yarn.
>
> Thanks,
>
> [1] Wordcount exampla with just the map part.
>
>
> public class MyWordCount {
> public static class MyMap extends Mapper {
> Log log = LogFactory.getLog(MyWordCount.class);
> private final static IntWritable one = new IntWritable(1);
> private Text word = new Text();
>
> public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
> StringTokenizer itr = new StringTokenizer(value.toString());
> System.out.println("HERRE");
> log.info("HERRRRRE");
> while (itr.hasMoreTokens()) {
> word.set(itr.nextToken());
> output.collect(word, one);
> }
> }
>
> public void run(Context context) throws IOException, InterruptedException {
> setup(context);
> try {
> while (context.nextKeyValue()) {
> System.out.println("Key: " + context.getCurrentKey() + " Value: " + context.getCurrentValue());
> map(context.getCurrentKey(), context.getCurrentValue(), context);
> }
> } finally {
> cleanup(context);
> }
> }
>
> public void cleanup(Mapper.Context context) {}
> }
>
> [2] yarn-site.xml
>
> <!-- job history -->
> <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property>
> <property> <name>yarn.nodemanager.log.retain-seconds</name> <value>900000</value> </property>
> <property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/app-logs</value> </property>
>
>
>
>
>
Re: println in MapReduce job
Posted by Peyman Mohajerian <mo...@gmail.com>.
You log statement should show up in the container log for that Map class
within the data node. I'm guessing you aren't looking in the right place.
On Thu, Sep 24, 2015 at 9:54 AM, xeonmailinglist <xe...@gmail.com>
wrote:
> Does anyone know this question about logging in MapReduce. I can't find an
> example or an explanation on how to print something inside user map and
> reduce functions.
>
> On 09/24/2015 04:19 PM, xeonmailinglist wrote:
>
> Hi,
>
> 1.
>
> I have this example of MapReduce [1], and I want to print info in the
> stdout and in a log file. It seems that the logs isn’t print anything. How
> can I make my class print these words?
> 2.
>
> I also have set in the yarn-site.xml to retain log. Although the logs
> are retained in the /app-logs dir, the userlogs dir is deleted at the
> end of the job execution. How can I make MapReduce to not delete files in
> the userlogs dir?
>
> I am using Yarn.
>
> Thanks,
>
> [1] Wordcount exampla with just the map part.
>
>
> public class MyWordCount {
> public static class MyMap extends Mapper {
> Log log = LogFactory.getLog(MyWordCount.class);
> private final static IntWritable one = new IntWritable(1);
> private Text word = new Text();
>
> public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
> StringTokenizer itr = new StringTokenizer(value.toString());
> System.out.println("HERRE");
> log.info("HERRRRRE");
> while (itr.hasMoreTokens()) {
> word.set(itr.nextToken());
> output.collect(word, one);
> }
> }
>
> public void run(Context context) throws IOException, InterruptedException {
> setup(context);
> try {
> while (context.nextKeyValue()) {
> System.out.println("Key: " + context.getCurrentKey() + " Value: " + context.getCurrentValue());
> map(context.getCurrentKey(), context.getCurrentValue(), context);
> }
> } finally {
> cleanup(context);
> }
> }
>
> public void cleanup(Mapper.Context context) {}
> }
>
> [2] yarn-site.xml
>
> <!-- job history -->
> <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property>
> <property> <name>yarn.nodemanager.log.retain-seconds</name> <value>900000</value> </property>
> <property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/app-logs</value> </property>
>
>
>
>
>
Re: println in MapReduce job
Posted by Peyman Mohajerian <mo...@gmail.com>.
You log statement should show up in the container log for that Map class
within the data node. I'm guessing you aren't looking in the right place.
On Thu, Sep 24, 2015 at 9:54 AM, xeonmailinglist <xe...@gmail.com>
wrote:
> Does anyone know this question about logging in MapReduce. I can't find an
> example or an explanation on how to print something inside user map and
> reduce functions.
>
> On 09/24/2015 04:19 PM, xeonmailinglist wrote:
>
> Hi,
>
> 1.
>
> I have this example of MapReduce [1], and I want to print info in the
> stdout and in a log file. It seems that the logs isn’t print anything. How
> can I make my class print these words?
> 2.
>
> I also have set in the yarn-site.xml to retain log. Although the logs
> are retained in the /app-logs dir, the userlogs dir is deleted at the
> end of the job execution. How can I make MapReduce to not delete files in
> the userlogs dir?
>
> I am using Yarn.
>
> Thanks,
>
> [1] Wordcount exampla with just the map part.
>
>
> public class MyWordCount {
> public static class MyMap extends Mapper {
> Log log = LogFactory.getLog(MyWordCount.class);
> private final static IntWritable one = new IntWritable(1);
> private Text word = new Text();
>
> public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
> StringTokenizer itr = new StringTokenizer(value.toString());
> System.out.println("HERRE");
> log.info("HERRRRRE");
> while (itr.hasMoreTokens()) {
> word.set(itr.nextToken());
> output.collect(word, one);
> }
> }
>
> public void run(Context context) throws IOException, InterruptedException {
> setup(context);
> try {
> while (context.nextKeyValue()) {
> System.out.println("Key: " + context.getCurrentKey() + " Value: " + context.getCurrentValue());
> map(context.getCurrentKey(), context.getCurrentValue(), context);
> }
> } finally {
> cleanup(context);
> }
> }
>
> public void cleanup(Mapper.Context context) {}
> }
>
> [2] yarn-site.xml
>
> <!-- job history -->
> <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property>
> <property> <name>yarn.nodemanager.log.retain-seconds</name> <value>900000</value> </property>
> <property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/app-logs</value> </property>
>
>
>
>
>
Re: println in MapReduce job
Posted by Peyman Mohajerian <mo...@gmail.com>.
You log statement should show up in the container log for that Map class
within the data node. I'm guessing you aren't looking in the right place.
On Thu, Sep 24, 2015 at 9:54 AM, xeonmailinglist <xe...@gmail.com>
wrote:
> Does anyone know this question about logging in MapReduce. I can't find an
> example or an explanation on how to print something inside user map and
> reduce functions.
>
> On 09/24/2015 04:19 PM, xeonmailinglist wrote:
>
> Hi,
>
> 1.
>
> I have this example of MapReduce [1], and I want to print info in the
> stdout and in a log file. It seems that the logs isn’t print anything. How
> can I make my class print these words?
> 2.
>
> I also have set in the yarn-site.xml to retain log. Although the logs
> are retained in the /app-logs dir, the userlogs dir is deleted at the
> end of the job execution. How can I make MapReduce to not delete files in
> the userlogs dir?
>
> I am using Yarn.
>
> Thanks,
>
> [1] Wordcount exampla with just the map part.
>
>
> public class MyWordCount {
> public static class MyMap extends Mapper {
> Log log = LogFactory.getLog(MyWordCount.class);
> private final static IntWritable one = new IntWritable(1);
> private Text word = new Text();
>
> public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
> StringTokenizer itr = new StringTokenizer(value.toString());
> System.out.println("HERRE");
> log.info("HERRRRRE");
> while (itr.hasMoreTokens()) {
> word.set(itr.nextToken());
> output.collect(word, one);
> }
> }
>
> public void run(Context context) throws IOException, InterruptedException {
> setup(context);
> try {
> while (context.nextKeyValue()) {
> System.out.println("Key: " + context.getCurrentKey() + " Value: " + context.getCurrentValue());
> map(context.getCurrentKey(), context.getCurrentValue(), context);
> }
> } finally {
> cleanup(context);
> }
> }
>
> public void cleanup(Mapper.Context context) {}
> }
>
> [2] yarn-site.xml
>
> <!-- job history -->
> <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property>
> <property> <name>yarn.nodemanager.log.retain-seconds</name> <value>900000</value> </property>
> <property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/app-logs</value> </property>
>
>
>
>
>
Re: println in MapReduce job
Posted by xeonmailinglist <xe...@gmail.com>.
Does anyone know this question about logging in MapReduce. I can't find
an example or an explanation on how to print something inside user map
and reduce functions.
On 09/24/2015 04:19 PM, xeonmailinglist wrote:
>
> Hi,
>
> 1.
>
> I have this example of MapReduce [1], and I want to print info in
> the stdout and in a log file. It seems that the logs isn’t print
> anything. How can I make my class print these words?
>
> 2.
>
> I also have set in the |yarn-site.xml| to retain log. Although the
> logs are retained in the |/app-logs| dir, the |userlogs| dir is
> deleted at the end of the job execution. How can I make MapReduce
> to not delete files in the |userlogs| dir?
>
> I am using Yarn.
>
> Thanks,
>
> [1] Wordcount exampla with just the map part.
>
> |public class MyWordCount { public static class MyMap extends Mapper {
> Log log = LogFactory.getLog(MyWordCount.class); private final static
> IntWritable one = new IntWritable(1); private Text word = new Text();
> public void map(LongWritable key, Text value, OutputCollector<Text,
> IntWritable> output, Reporter reporter) throws IOException {
> StringTokenizer itr = new StringTokenizer(value.toString());
> System.out.println("HERRE"); log.info("HERRRRRE"); while
> (itr.hasMoreTokens()) { word.set(itr.nextToken());
> output.collect(word, one); } } public void run(Context context) throws
> IOException, InterruptedException { setup(context); try { while
> (context.nextKeyValue()) { System.out.println("Key: " +
> context.getCurrentKey() + " Value: " + context.getCurrentValue());
> map(context.getCurrentKey(), context.getCurrentValue(), context); } }
> finally { cleanup(context); } } public void cleanup(Mapper.Context
> context) {} } |
>
> [2] yarn-site.xml
>
> |<!-- job history --> <property>
> <name>yarn.log-aggregation-enable</name> <value>true</value>
> </property> <property>
> <name>yarn.nodemanager.log.retain-seconds</name> <value>900000</value>
> </property> <property>
> <name>yarn.nodemanager.remote-app-log-dir</name>
> <value>/app-logs</value> </property> |
>
Re: println in MapReduce job
Posted by Rich Haase <rh...@pandora.com>.
To unsubscribe from this list send an email to user-unsubscribe@hadoop.apache.org<ma...@hadoop.apache.org>.
https://hadoop.apache.org/mailing_lists.html
On Sep 24, 2015, at 9:40 AM, sukesh kumar <s7...@gmail.com>> wrote:
unsubscribe
On Thu, Sep 24, 2015 at 8:49 PM, xeonmailinglist <xe...@gmail.com>> wrote:
Hi,
1.
I have this example of MapReduce [1], and I want to print info in the stdout and in a log file. It seems that the logs isn’t print anything. How can I make my class print these words?
2.
I also have set in the yarn-site.xml to retain log. Although the logs are retained in the /app-logs dir, the userlogs dir is deleted at the end of the job execution. How can I make MapReduce to not delete files in the userlogs dir?
I am using Yarn.
Thanks,
[1] Wordcount exampla with just the map part.
public class MyWordCount {
public static class MyMap extends Mapper {
Log log = LogFactory.getLog(MyWordCount.class);
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
StringTokenizer itr = new StringTokenizer(value.toString());
System.out.println("HERRE");
log.info<http://log.info/>("HERRRRRE");
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
output.collect(word, one);
}
}
public void run(Context context) throws IOException, InterruptedException {
setup(context);
try {
while (context.nextKeyValue()) {
System.out.println("Key: " + context.getCurrentKey() + " Value: " + context.getCurrentValue());
map(context.getCurrentKey(), context.getCurrentValue(), context);
}
} finally {
cleanup(context);
}
}
public void cleanup(Mapper.Context context) {}
}
[2] yarn-site.xml
<!-- job history -->
<property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property>
<property> <name>yarn.nodemanager.log.retain-seconds</name> <value>900000</value> </property>
<property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/app-logs</value> </property>
--
Thanks & Best Regards
Sukesh Kumar
Re: println in MapReduce job
Posted by Rich Haase <rh...@pandora.com>.
To unsubscribe from this list send an email to user-unsubscribe@hadoop.apache.org<ma...@hadoop.apache.org>.
https://hadoop.apache.org/mailing_lists.html
On Sep 24, 2015, at 9:40 AM, sukesh kumar <s7...@gmail.com>> wrote:
unsubscribe
On Thu, Sep 24, 2015 at 8:49 PM, xeonmailinglist <xe...@gmail.com>> wrote:
Hi,
1.
I have this example of MapReduce [1], and I want to print info in the stdout and in a log file. It seems that the logs isn’t print anything. How can I make my class print these words?
2.
I also have set in the yarn-site.xml to retain log. Although the logs are retained in the /app-logs dir, the userlogs dir is deleted at the end of the job execution. How can I make MapReduce to not delete files in the userlogs dir?
I am using Yarn.
Thanks,
[1] Wordcount exampla with just the map part.
public class MyWordCount {
public static class MyMap extends Mapper {
Log log = LogFactory.getLog(MyWordCount.class);
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
StringTokenizer itr = new StringTokenizer(value.toString());
System.out.println("HERRE");
log.info<http://log.info/>("HERRRRRE");
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
output.collect(word, one);
}
}
public void run(Context context) throws IOException, InterruptedException {
setup(context);
try {
while (context.nextKeyValue()) {
System.out.println("Key: " + context.getCurrentKey() + " Value: " + context.getCurrentValue());
map(context.getCurrentKey(), context.getCurrentValue(), context);
}
} finally {
cleanup(context);
}
}
public void cleanup(Mapper.Context context) {}
}
[2] yarn-site.xml
<!-- job history -->
<property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property>
<property> <name>yarn.nodemanager.log.retain-seconds</name> <value>900000</value> </property>
<property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/app-logs</value> </property>
--
Thanks & Best Regards
Sukesh Kumar
Re: println in MapReduce job
Posted by Rich Haase <rh...@pandora.com>.
To unsubscribe from this list send an email to user-unsubscribe@hadoop.apache.org<ma...@hadoop.apache.org>.
https://hadoop.apache.org/mailing_lists.html
On Sep 24, 2015, at 9:40 AM, sukesh kumar <s7...@gmail.com>> wrote:
unsubscribe
On Thu, Sep 24, 2015 at 8:49 PM, xeonmailinglist <xe...@gmail.com>> wrote:
Hi,
1.
I have this example of MapReduce [1], and I want to print info in the stdout and in a log file. It seems that the logs isn’t print anything. How can I make my class print these words?
2.
I also have set in the yarn-site.xml to retain log. Although the logs are retained in the /app-logs dir, the userlogs dir is deleted at the end of the job execution. How can I make MapReduce to not delete files in the userlogs dir?
I am using Yarn.
Thanks,
[1] Wordcount exampla with just the map part.
public class MyWordCount {
public static class MyMap extends Mapper {
Log log = LogFactory.getLog(MyWordCount.class);
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
StringTokenizer itr = new StringTokenizer(value.toString());
System.out.println("HERRE");
log.info<http://log.info/>("HERRRRRE");
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
output.collect(word, one);
}
}
public void run(Context context) throws IOException, InterruptedException {
setup(context);
try {
while (context.nextKeyValue()) {
System.out.println("Key: " + context.getCurrentKey() + " Value: " + context.getCurrentValue());
map(context.getCurrentKey(), context.getCurrentValue(), context);
}
} finally {
cleanup(context);
}
}
public void cleanup(Mapper.Context context) {}
}
[2] yarn-site.xml
<!-- job history -->
<property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property>
<property> <name>yarn.nodemanager.log.retain-seconds</name> <value>900000</value> </property>
<property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/app-logs</value> </property>
--
Thanks & Best Regards
Sukesh Kumar
Re: println in MapReduce job
Posted by Rich Haase <rh...@pandora.com>.
To unsubscribe from this list send an email to user-unsubscribe@hadoop.apache.org<ma...@hadoop.apache.org>.
https://hadoop.apache.org/mailing_lists.html
On Sep 24, 2015, at 9:40 AM, sukesh kumar <s7...@gmail.com>> wrote:
unsubscribe
On Thu, Sep 24, 2015 at 8:49 PM, xeonmailinglist <xe...@gmail.com>> wrote:
Hi,
1.
I have this example of MapReduce [1], and I want to print info in the stdout and in a log file. It seems that the logs isn’t print anything. How can I make my class print these words?
2.
I also have set in the yarn-site.xml to retain log. Although the logs are retained in the /app-logs dir, the userlogs dir is deleted at the end of the job execution. How can I make MapReduce to not delete files in the userlogs dir?
I am using Yarn.
Thanks,
[1] Wordcount exampla with just the map part.
public class MyWordCount {
public static class MyMap extends Mapper {
Log log = LogFactory.getLog(MyWordCount.class);
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
StringTokenizer itr = new StringTokenizer(value.toString());
System.out.println("HERRE");
log.info<http://log.info/>("HERRRRRE");
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
output.collect(word, one);
}
}
public void run(Context context) throws IOException, InterruptedException {
setup(context);
try {
while (context.nextKeyValue()) {
System.out.println("Key: " + context.getCurrentKey() + " Value: " + context.getCurrentValue());
map(context.getCurrentKey(), context.getCurrentValue(), context);
}
} finally {
cleanup(context);
}
}
public void cleanup(Mapper.Context context) {}
}
[2] yarn-site.xml
<!-- job history -->
<property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property>
<property> <name>yarn.nodemanager.log.retain-seconds</name> <value>900000</value> </property>
<property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/app-logs</value> </property>
--
Thanks & Best Regards
Sukesh Kumar
Re: println in MapReduce job
Posted by sukesh kumar <s7...@gmail.com>.
unsubscribe
On Thu, Sep 24, 2015 at 8:49 PM, xeonmailinglist <xe...@gmail.com>
wrote:
> Hi,
>
> 1.
>
> I have this example of MapReduce [1], and I want to print info in the
> stdout and in a log file. It seems that the logs isn’t print anything. How
> can I make my class print these words?
> 2.
>
> I also have set in the yarn-site.xml to retain log. Although the logs
> are retained in the /app-logs dir, the userlogs dir is deleted at the
> end of the job execution. How can I make MapReduce to not delete files in
> the userlogs dir?
>
> I am using Yarn.
>
> Thanks,
>
> [1] Wordcount exampla with just the map part.
>
>
> public class MyWordCount {
> public static class MyMap extends Mapper {
> Log log = LogFactory.getLog(MyWordCount.class);
> private final static IntWritable one = new IntWritable(1);
> private Text word = new Text();
>
> public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
> StringTokenizer itr = new StringTokenizer(value.toString());
> System.out.println("HERRE");
> log.info("HERRRRRE");
> while (itr.hasMoreTokens()) {
> word.set(itr.nextToken());
> output.collect(word, one);
> }
> }
>
> public void run(Context context) throws IOException, InterruptedException {
> setup(context);
> try {
> while (context.nextKeyValue()) {
> System.out.println("Key: " + context.getCurrentKey() + " Value: " + context.getCurrentValue());
> map(context.getCurrentKey(), context.getCurrentValue(), context);
> }
> } finally {
> cleanup(context);
> }
> }
>
> public void cleanup(Mapper.Context context) {}
> }
>
> [2] yarn-site.xml
>
> <!-- job history -->
> <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property>
> <property> <name>yarn.nodemanager.log.retain-seconds</name> <value>900000</value> </property>
> <property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/app-logs</value> </property>
>
>
>
--
Thanks & Best Regards
Sukesh Kumar
Re: println in MapReduce job
Posted by sukesh kumar <s7...@gmail.com>.
unsubscribe
On Thu, Sep 24, 2015 at 8:49 PM, xeonmailinglist <xe...@gmail.com>
wrote:
> Hi,
>
> 1.
>
> I have this example of MapReduce [1], and I want to print info in the
> stdout and in a log file. It seems that the logs isn’t print anything. How
> can I make my class print these words?
> 2.
>
> I also have set in the yarn-site.xml to retain log. Although the logs
> are retained in the /app-logs dir, the userlogs dir is deleted at the
> end of the job execution. How can I make MapReduce to not delete files in
> the userlogs dir?
>
> I am using Yarn.
>
> Thanks,
>
> [1] Wordcount exampla with just the map part.
>
>
> public class MyWordCount {
> public static class MyMap extends Mapper {
> Log log = LogFactory.getLog(MyWordCount.class);
> private final static IntWritable one = new IntWritable(1);
> private Text word = new Text();
>
> public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
> StringTokenizer itr = new StringTokenizer(value.toString());
> System.out.println("HERRE");
> log.info("HERRRRRE");
> while (itr.hasMoreTokens()) {
> word.set(itr.nextToken());
> output.collect(word, one);
> }
> }
>
> public void run(Context context) throws IOException, InterruptedException {
> setup(context);
> try {
> while (context.nextKeyValue()) {
> System.out.println("Key: " + context.getCurrentKey() + " Value: " + context.getCurrentValue());
> map(context.getCurrentKey(), context.getCurrentValue(), context);
> }
> } finally {
> cleanup(context);
> }
> }
>
> public void cleanup(Mapper.Context context) {}
> }
>
> [2] yarn-site.xml
>
> <!-- job history -->
> <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property>
> <property> <name>yarn.nodemanager.log.retain-seconds</name> <value>900000</value> </property>
> <property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/app-logs</value> </property>
>
>
>
--
Thanks & Best Regards
Sukesh Kumar
Re: println in MapReduce job
Posted by xeonmailinglist <xe...@gmail.com>.
Does anyone know this question about logging in MapReduce. I can't find
an example or an explanation on how to print something inside user map
and reduce functions.
On 09/24/2015 04:19 PM, xeonmailinglist wrote:
>
> Hi,
>
> 1.
>
> I have this example of MapReduce [1], and I want to print info in
> the stdout and in a log file. It seems that the logs isn’t print
> anything. How can I make my class print these words?
>
> 2.
>
> I also have set in the |yarn-site.xml| to retain log. Although the
> logs are retained in the |/app-logs| dir, the |userlogs| dir is
> deleted at the end of the job execution. How can I make MapReduce
> to not delete files in the |userlogs| dir?
>
> I am using Yarn.
>
> Thanks,
>
> [1] Wordcount exampla with just the map part.
>
> |public class MyWordCount { public static class MyMap extends Mapper {
> Log log = LogFactory.getLog(MyWordCount.class); private final static
> IntWritable one = new IntWritable(1); private Text word = new Text();
> public void map(LongWritable key, Text value, OutputCollector<Text,
> IntWritable> output, Reporter reporter) throws IOException {
> StringTokenizer itr = new StringTokenizer(value.toString());
> System.out.println("HERRE"); log.info("HERRRRRE"); while
> (itr.hasMoreTokens()) { word.set(itr.nextToken());
> output.collect(word, one); } } public void run(Context context) throws
> IOException, InterruptedException { setup(context); try { while
> (context.nextKeyValue()) { System.out.println("Key: " +
> context.getCurrentKey() + " Value: " + context.getCurrentValue());
> map(context.getCurrentKey(), context.getCurrentValue(), context); } }
> finally { cleanup(context); } } public void cleanup(Mapper.Context
> context) {} } |
>
> [2] yarn-site.xml
>
> |<!-- job history --> <property>
> <name>yarn.log-aggregation-enable</name> <value>true</value>
> </property> <property>
> <name>yarn.nodemanager.log.retain-seconds</name> <value>900000</value>
> </property> <property>
> <name>yarn.nodemanager.remote-app-log-dir</name>
> <value>/app-logs</value> </property> |
>
Re: println in MapReduce job
Posted by xeonmailinglist <xe...@gmail.com>.
Does anyone know this question about logging in MapReduce. I can't find
an example or an explanation on how to print something inside user map
and reduce functions.
On 09/24/2015 04:19 PM, xeonmailinglist wrote:
>
> Hi,
>
> 1.
>
> I have this example of MapReduce [1], and I want to print info in
> the stdout and in a log file. It seems that the logs isn’t print
> anything. How can I make my class print these words?
>
> 2.
>
> I also have set in the |yarn-site.xml| to retain log. Although the
> logs are retained in the |/app-logs| dir, the |userlogs| dir is
> deleted at the end of the job execution. How can I make MapReduce
> to not delete files in the |userlogs| dir?
>
> I am using Yarn.
>
> Thanks,
>
> [1] Wordcount exampla with just the map part.
>
> |public class MyWordCount { public static class MyMap extends Mapper {
> Log log = LogFactory.getLog(MyWordCount.class); private final static
> IntWritable one = new IntWritable(1); private Text word = new Text();
> public void map(LongWritable key, Text value, OutputCollector<Text,
> IntWritable> output, Reporter reporter) throws IOException {
> StringTokenizer itr = new StringTokenizer(value.toString());
> System.out.println("HERRE"); log.info("HERRRRRE"); while
> (itr.hasMoreTokens()) { word.set(itr.nextToken());
> output.collect(word, one); } } public void run(Context context) throws
> IOException, InterruptedException { setup(context); try { while
> (context.nextKeyValue()) { System.out.println("Key: " +
> context.getCurrentKey() + " Value: " + context.getCurrentValue());
> map(context.getCurrentKey(), context.getCurrentValue(), context); } }
> finally { cleanup(context); } } public void cleanup(Mapper.Context
> context) {} } |
>
> [2] yarn-site.xml
>
> |<!-- job history --> <property>
> <name>yarn.log-aggregation-enable</name> <value>true</value>
> </property> <property>
> <name>yarn.nodemanager.log.retain-seconds</name> <value>900000</value>
> </property> <property>
> <name>yarn.nodemanager.remote-app-log-dir</name>
> <value>/app-logs</value> </property> |
>
Re: println in MapReduce job
Posted by sukesh kumar <s7...@gmail.com>.
unsubscribe
On Thu, Sep 24, 2015 at 8:49 PM, xeonmailinglist <xe...@gmail.com>
wrote:
> Hi,
>
> 1.
>
> I have this example of MapReduce [1], and I want to print info in the
> stdout and in a log file. It seems that the logs isn’t print anything. How
> can I make my class print these words?
> 2.
>
> I also have set in the yarn-site.xml to retain log. Although the logs
> are retained in the /app-logs dir, the userlogs dir is deleted at the
> end of the job execution. How can I make MapReduce to not delete files in
> the userlogs dir?
>
> I am using Yarn.
>
> Thanks,
>
> [1] Wordcount exampla with just the map part.
>
>
> public class MyWordCount {
> public static class MyMap extends Mapper {
> Log log = LogFactory.getLog(MyWordCount.class);
> private final static IntWritable one = new IntWritable(1);
> private Text word = new Text();
>
> public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
> StringTokenizer itr = new StringTokenizer(value.toString());
> System.out.println("HERRE");
> log.info("HERRRRRE");
> while (itr.hasMoreTokens()) {
> word.set(itr.nextToken());
> output.collect(word, one);
> }
> }
>
> public void run(Context context) throws IOException, InterruptedException {
> setup(context);
> try {
> while (context.nextKeyValue()) {
> System.out.println("Key: " + context.getCurrentKey() + " Value: " + context.getCurrentValue());
> map(context.getCurrentKey(), context.getCurrentValue(), context);
> }
> } finally {
> cleanup(context);
> }
> }
>
> public void cleanup(Mapper.Context context) {}
> }
>
> [2] yarn-site.xml
>
> <!-- job history -->
> <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property>
> <property> <name>yarn.nodemanager.log.retain-seconds</name> <value>900000</value> </property>
> <property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/app-logs</value> </property>
>
>
>
--
Thanks & Best Regards
Sukesh Kumar
Re: println in MapReduce job
Posted by xeonmailinglist <xe...@gmail.com>.
Does anyone know this question about logging in MapReduce. I can't find
an example or an explanation on how to print something inside user map
and reduce functions.
On 09/24/2015 04:19 PM, xeonmailinglist wrote:
>
> Hi,
>
> 1.
>
> I have this example of MapReduce [1], and I want to print info in
> the stdout and in a log file. It seems that the logs isn’t print
> anything. How can I make my class print these words?
>
> 2.
>
> I also have set in the |yarn-site.xml| to retain log. Although the
> logs are retained in the |/app-logs| dir, the |userlogs| dir is
> deleted at the end of the job execution. How can I make MapReduce
> to not delete files in the |userlogs| dir?
>
> I am using Yarn.
>
> Thanks,
>
> [1] Wordcount exampla with just the map part.
>
> |public class MyWordCount { public static class MyMap extends Mapper {
> Log log = LogFactory.getLog(MyWordCount.class); private final static
> IntWritable one = new IntWritable(1); private Text word = new Text();
> public void map(LongWritable key, Text value, OutputCollector<Text,
> IntWritable> output, Reporter reporter) throws IOException {
> StringTokenizer itr = new StringTokenizer(value.toString());
> System.out.println("HERRE"); log.info("HERRRRRE"); while
> (itr.hasMoreTokens()) { word.set(itr.nextToken());
> output.collect(word, one); } } public void run(Context context) throws
> IOException, InterruptedException { setup(context); try { while
> (context.nextKeyValue()) { System.out.println("Key: " +
> context.getCurrentKey() + " Value: " + context.getCurrentValue());
> map(context.getCurrentKey(), context.getCurrentValue(), context); } }
> finally { cleanup(context); } } public void cleanup(Mapper.Context
> context) {} } |
>
> [2] yarn-site.xml
>
> |<!-- job history --> <property>
> <name>yarn.log-aggregation-enable</name> <value>true</value>
> </property> <property>
> <name>yarn.nodemanager.log.retain-seconds</name> <value>900000</value>
> </property> <property>
> <name>yarn.nodemanager.remote-app-log-dir</name>
> <value>/app-logs</value> </property> |
>
Re: println in MapReduce job
Posted by sukesh kumar <s7...@gmail.com>.
unsubscribe
On Thu, Sep 24, 2015 at 8:49 PM, xeonmailinglist <xe...@gmail.com>
wrote:
> Hi,
>
> 1.
>
> I have this example of MapReduce [1], and I want to print info in the
> stdout and in a log file. It seems that the logs isn’t print anything. How
> can I make my class print these words?
> 2.
>
> I also have set in the yarn-site.xml to retain log. Although the logs
> are retained in the /app-logs dir, the userlogs dir is deleted at the
> end of the job execution. How can I make MapReduce to not delete files in
> the userlogs dir?
>
> I am using Yarn.
>
> Thanks,
>
> [1] Wordcount exampla with just the map part.
>
>
> public class MyWordCount {
> public static class MyMap extends Mapper {
> Log log = LogFactory.getLog(MyWordCount.class);
> private final static IntWritable one = new IntWritable(1);
> private Text word = new Text();
>
> public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
> StringTokenizer itr = new StringTokenizer(value.toString());
> System.out.println("HERRE");
> log.info("HERRRRRE");
> while (itr.hasMoreTokens()) {
> word.set(itr.nextToken());
> output.collect(word, one);
> }
> }
>
> public void run(Context context) throws IOException, InterruptedException {
> setup(context);
> try {
> while (context.nextKeyValue()) {
> System.out.println("Key: " + context.getCurrentKey() + " Value: " + context.getCurrentValue());
> map(context.getCurrentKey(), context.getCurrentValue(), context);
> }
> } finally {
> cleanup(context);
> }
> }
>
> public void cleanup(Mapper.Context context) {}
> }
>
> [2] yarn-site.xml
>
> <!-- job history -->
> <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property>
> <property> <name>yarn.nodemanager.log.retain-seconds</name> <value>900000</value> </property>
> <property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/app-logs</value> </property>
>
>
>
--
Thanks & Best Regards
Sukesh Kumar