You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by xeonmailinglist-gmail <xe...@gmail.com> on 2015/06/26 18:13:46 UTC
Invalid key type in the map task
Hi,
I have this map class that is accepting input files with a key as
LongWritable and a value of Text.
The input file is in [1]. Here we can see that it contains a key as a
Long (I think) and bytes as value.
In [2], it is my map class. The goal of the map class is to read the
input data, and sent it out as it is.
I was expecting that the key was the index number, and the value was the
gibberish. But, I am having a strange error [3], as the key is getting
there a s Text, and not the index.
Can anyone tell me why I get this error, and how I can solve it?
[1] Input file
|xubuntu@hadoop-coc-1:~/Programs/hadoop$ hdfs dfs -cat /output1-1434970707/part-m-00000
15/06/26 12:01:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
0 SEQ"org.apache.hadoop.io.BytesWritable"org.apache.hadoop.io.BytesWritableI���#HA��u��BZ��n��tX eX����zZi��ndFw�{>�H�\� ��Bi�'�gb�Y�a�*4�7;���uL_��gs���`��*
187 ����tAj#�5*nX�b"��>�˥k�c2�)�1z�1���6�l���Q�be3'$�G��!{�� u����z@�t����v���r�gj8fq�P��.�m ���y��n��!���_w�U8e�����v0���S���2c�_�ջ�R\x*��
�܁���|x
U�3���l���qaq� )�N��k�L��a�2�ն���k NL
404 ��)i�g A�2)L��W,���ܺ�V�w��M7ڭCϣ����ZI�� be��$�f[���W�7��5�6.}�ѽb� ����C��%|�h�lj�D}@F�"�(K��xh�'6
|
[2] My map class
|
/** Identity mapper set by the user. */
public static class MyFullyIndentityMapperWebdataScan
extends Mapper<LongWritable, Text, LongWritable, Text>{
// private LongWritable word = new LongWritable();
private LongWritable word = new LongWritable();
private Text rvalue = new Text();
public void map(LongWritable key, Text value, Context context
) throws IOException, InterruptedException {
System.out.println("1: " + key.getClass().getName() + " " + value.getClass().getName());
System.out.println("2: " + context.getCurrentKey().getClass() + " " + context.getCurrentValue().getClass());
context.write(key, value);
}
}
|
[3] Output of execution
|
Log Type: stdout
Log Upload Time: 26-Jun-2015 11:58:53
Log Length: 138
1: org.apache.hadoop.io.LongWritable org.apache.hadoop.io.Text 2: class org.apache.hadoop.io.LongWritable class org.apache.hadoop.io.Text
Log Type: syslog
Log Upload Time: 26-Jun-2015 11:58:53
Log Length: 4367
Showing 4096 bytes of 4367 total. Click here for the full log.
operties from hadoop-metrics2.properties 2015-06-26 11:58:32,118 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2015-06-26 11:58:32,118 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started 2015-06-26 11:58:32,128 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens: 2015-06-26 11:58:32,128 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1435332416394_0009, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@5d7657) 2015-06-26 11:58:32,208 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now. 2015-06-26 11:58:32,439 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /tmp/hadoop-temp/nm-local-dir/usercache/xubuntu/appcache/application_1435332416394_0009 2015-06-26 11:58:32,611 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 2015-06-26 11:58:33,038 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2015-06-26 11:58:33,180 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hadoop-coc-1:9000/output1-1434970707/part-m-00000:0+17853 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2015-06-26 11:58:33,255 INFO [main] org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2015-06-26 11:58:33,269 INFO [main] org.apache.hadoop.mapred.MapTask: Starting flush of map output 2015-06-26 11:58:33,276 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.Text, received org.apache.hadoop.io.LongWritable at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1069) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:712) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89) at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112) at org.apache.hadoop.mapred.examples.MyWebDataScan$MyFullyIndentityMapperWebdataScan.map(MyWebDataScan.java:144) at org.apache.hadoop.mapred.examples.MyWebDataScan$MyFullyIndentityMapperWebdataScan.map(MyWebDataScan.java:131) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
|
--
--
Thanks,
Re: Invalid key type in the map task
Posted by Dieter De Witte <dr...@gmail.com>.
You need to show the driver class as well. Are you using textinputformat?
Are you aware that this standard inputformat will take as a value the
complete line (until newline separator), the key in that case is the
bitoffset in the file and definitely not the number you assume it will be.
2015-06-26 18:13 GMT+02:00 xeonmailinglist-gmail <xe...@gmail.com>
:
> Hi,
>
> I have this map class that is accepting input files with a key as
> LongWritable and a value of Text.
>
> The input file is in [1]. Here we can see that it contains a key as a Long
> (I think) and bytes as value.
> In [2], it is my map class. The goal of the map class is to read the input
> data, and sent it out as it is.
> I was expecting that the key was the index number, and the value was the
> gibberish. But, I am having a strange error [3], as the key is getting
> there a s Text, and not the index.
>
> Can anyone tell me why I get this error, and how I can solve it?
>
> [1] Input file
>
> xubuntu@hadoop-coc-1:~/Programs/hadoop$ hdfs dfs -cat /output1-1434970707/part-m-00000
> 15/06/26 12:01:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 0 SEQ "org.apache.hadoop.io.BytesWritable"org.apache.hadoop.io.BytesWritableI���#HA��u��BZ � � n�� tX eX����zZi��n dFw�{>�H� \� ��Bi�'�g b�Y�a �*4�7;���uL_��gs�� �`�� *
> 187 ���� tAj#�5*nX�b " ��>�˥k�c2�)�1z�1� ��6�l��� Q �be3'$ �G��!{� � u����z@�t����v���r�gj8fq�P��.�m ��� y ��n��!���_w�U8e�����v0���S���2c�_�ջ�R\x *��
> �܁���|x
> U�3���l���qa q� )�N ��k�L��a�2�ն���k NL
> 404 ��)i �g A�2 )L��W,���ܺ�V�w� �M7ڭCϣ����ZI�� be��$�f [�� � W� 7��5�6.}�ѽ b� ����C��%|�h�lj�D } @ F�"�(K ��xh�'6
>
> [2] My map class
>
>
> /** Identity mapper set by the user. */
> public static class MyFullyIndentityMapperWebdataScan
> extends Mapper<LongWritable, Text, LongWritable, Text>{
>
> // private LongWritable word = new LongWritable();
> private LongWritable word = new LongWritable();
> private Text rvalue = new Text();
>
> public void map(LongWritable key, Text value, Context context
> ) throws IOException, InterruptedException {
>
> System.out.println("1: " + key.getClass().getName() + " " + value.getClass().getName());
> System.out.println("2: " + context.getCurrentKey().getClass() + " " + context.getCurrentValue().getClass());
> context.write(key, value);
> }
> }
>
> [3] Output of execution
>
>
> Log Type: stdout
>
> Log Upload Time: 26-Jun-2015 11:58:53
>
> Log Length: 138
>
> 1: org.apache.hadoop.io.LongWritable org.apache.hadoop.io.Text 2: class org.apache.hadoop.io.LongWritable class org.apache.hadoop.io.Text
>
>
>
> Log Type: syslog
>
> Log Upload Time: 26-Jun-2015 11:58:53
>
> Log Length: 4367
>
> Showing 4096 bytes of 4367 total. Click here for the full log.
>
> operties from hadoop-metrics2.properties 2015-06-26 11:58:32,118 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2015-06-26 11:58:32,118 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started 2015-06-26 11:58:32,128 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens: 2015-06-26 11:58:32,128 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1435332416394_0009, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@5d7657) 2015-06-26 11:58:32,208 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now. 2015-06-26 11:58:32,439 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /tmp/hadoop-temp/nm-local-dir/usercache/xubuntu/appcache/application_1435332416394_0009 2015-06-26 11:58:32,611 INFO [main] org.apache.hadoop.conf.Configuration.deprecation:
> session.id is deprecated. Instead, use dfs.metrics.session-id 2015-06-26 11:58:33,038 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2015-06-26 11:58:33,180 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hadoop-coc-1:9000/output1-1434970707/part-m-00000:0+17853 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2015-06-26 11:58:33,255 INFO [main] org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2015-06-26 11:
> 5
> 8:33,269 INFO [main] org.apache.hadoop.mapred.MapTask: Starting flush of map output 2015-06-26 11:58:33,276 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.Text, received org.apache.hadoop.io.LongWritable at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1069) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:712) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89) at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112) at org.apache.hadoop.mapred.examples.MyWebDataScan$MyFullyIndentityMapperWebdataScan.map(MyWebDataScan.java:144) at org.apache.hadoop.mapred.examples.MyWebDataScan$MyFullyIndentityMapperWebdataScan.map(MyWebDataScan.java:131) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.
> h
> adoop.mapred.MapTask.runNewMapper(MapTask.java:784) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>
>
>
> --
> --
> Thanks,
>
>
Re: Invalid key type in the map task
Posted by Dieter De Witte <dr...@gmail.com>.
You need to show the driver class as well. Are you using textinputformat?
Are you aware that this standard inputformat will take as a value the
complete line (until newline separator), the key in that case is the
bitoffset in the file and definitely not the number you assume it will be.
2015-06-26 18:13 GMT+02:00 xeonmailinglist-gmail <xe...@gmail.com>
:
> Hi,
>
> I have this map class that is accepting input files with a key as
> LongWritable and a value of Text.
>
> The input file is in [1]. Here we can see that it contains a key as a Long
> (I think) and bytes as value.
> In [2], it is my map class. The goal of the map class is to read the input
> data, and sent it out as it is.
> I was expecting that the key was the index number, and the value was the
> gibberish. But, I am having a strange error [3], as the key is getting
> there a s Text, and not the index.
>
> Can anyone tell me why I get this error, and how I can solve it?
>
> [1] Input file
>
> xubuntu@hadoop-coc-1:~/Programs/hadoop$ hdfs dfs -cat /output1-1434970707/part-m-00000
> 15/06/26 12:01:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 0 SEQ "org.apache.hadoop.io.BytesWritable"org.apache.hadoop.io.BytesWritableI���#HA��u��BZ � � n�� tX eX����zZi��n dFw�{>�H� \� ��Bi�'�g b�Y�a �*4�7;���uL_��gs�� �`�� *
> 187 ���� tAj#�5*nX�b " ��>�˥k�c2�)�1z�1� ��6�l��� Q �be3'$ �G��!{� � u����z@�t����v���r�gj8fq�P��.�m ��� y ��n��!���_w�U8e�����v0���S���2c�_�ջ�R\x *��
> �܁���|x
> U�3���l���qa q� )�N ��k�L��a�2�ն���k NL
> 404 ��)i �g A�2 )L��W,���ܺ�V�w� �M7ڭCϣ����ZI�� be��$�f [�� � W� 7��5�6.}�ѽ b� ����C��%|�h�lj�D } @ F�"�(K ��xh�'6
>
> [2] My map class
>
>
> /** Identity mapper set by the user. */
> public static class MyFullyIndentityMapperWebdataScan
> extends Mapper<LongWritable, Text, LongWritable, Text>{
>
> // private LongWritable word = new LongWritable();
> private LongWritable word = new LongWritable();
> private Text rvalue = new Text();
>
> public void map(LongWritable key, Text value, Context context
> ) throws IOException, InterruptedException {
>
> System.out.println("1: " + key.getClass().getName() + " " + value.getClass().getName());
> System.out.println("2: " + context.getCurrentKey().getClass() + " " + context.getCurrentValue().getClass());
> context.write(key, value);
> }
> }
>
> [3] Output of execution
>
>
> Log Type: stdout
>
> Log Upload Time: 26-Jun-2015 11:58:53
>
> Log Length: 138
>
> 1: org.apache.hadoop.io.LongWritable org.apache.hadoop.io.Text 2: class org.apache.hadoop.io.LongWritable class org.apache.hadoop.io.Text
>
>
>
> Log Type: syslog
>
> Log Upload Time: 26-Jun-2015 11:58:53
>
> Log Length: 4367
>
> Showing 4096 bytes of 4367 total. Click here for the full log.
>
> operties from hadoop-metrics2.properties 2015-06-26 11:58:32,118 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2015-06-26 11:58:32,118 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started 2015-06-26 11:58:32,128 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens: 2015-06-26 11:58:32,128 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1435332416394_0009, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@5d7657) 2015-06-26 11:58:32,208 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now. 2015-06-26 11:58:32,439 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /tmp/hadoop-temp/nm-local-dir/usercache/xubuntu/appcache/application_1435332416394_0009 2015-06-26 11:58:32,611 INFO [main] org.apache.hadoop.conf.Configuration.deprecation:
> session.id is deprecated. Instead, use dfs.metrics.session-id 2015-06-26 11:58:33,038 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2015-06-26 11:58:33,180 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hadoop-coc-1:9000/output1-1434970707/part-m-00000:0+17853 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2015-06-26 11:58:33,255 INFO [main] org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2015-06-26 11:
> 5
> 8:33,269 INFO [main] org.apache.hadoop.mapred.MapTask: Starting flush of map output 2015-06-26 11:58:33,276 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.Text, received org.apache.hadoop.io.LongWritable at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1069) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:712) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89) at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112) at org.apache.hadoop.mapred.examples.MyWebDataScan$MyFullyIndentityMapperWebdataScan.map(MyWebDataScan.java:144) at org.apache.hadoop.mapred.examples.MyWebDataScan$MyFullyIndentityMapperWebdataScan.map(MyWebDataScan.java:131) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.
> h
> adoop.mapred.MapTask.runNewMapper(MapTask.java:784) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>
>
>
> --
> --
> Thanks,
>
>
Re: Invalid key type in the map task
Posted by Dieter De Witte <dr...@gmail.com>.
You need to show the driver class as well. Are you using textinputformat?
Are you aware that this standard inputformat will take as a value the
complete line (until newline separator), the key in that case is the
bitoffset in the file and definitely not the number you assume it will be.
2015-06-26 18:13 GMT+02:00 xeonmailinglist-gmail <xe...@gmail.com>
:
> Hi,
>
> I have this map class that is accepting input files with a key as
> LongWritable and a value of Text.
>
> The input file is in [1]. Here we can see that it contains a key as a Long
> (I think) and bytes as value.
> In [2], it is my map class. The goal of the map class is to read the input
> data, and sent it out as it is.
> I was expecting that the key was the index number, and the value was the
> gibberish. But, I am having a strange error [3], as the key is getting
> there a s Text, and not the index.
>
> Can anyone tell me why I get this error, and how I can solve it?
>
> [1] Input file
>
> xubuntu@hadoop-coc-1:~/Programs/hadoop$ hdfs dfs -cat /output1-1434970707/part-m-00000
> 15/06/26 12:01:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 0 SEQ "org.apache.hadoop.io.BytesWritable"org.apache.hadoop.io.BytesWritableI���#HA��u��BZ � � n�� tX eX����zZi��n dFw�{>�H� \� ��Bi�'�g b�Y�a �*4�7;���uL_��gs�� �`�� *
> 187 ���� tAj#�5*nX�b " ��>�˥k�c2�)�1z�1� ��6�l��� Q �be3'$ �G��!{� � u����z@�t����v���r�gj8fq�P��.�m ��� y ��n��!���_w�U8e�����v0���S���2c�_�ջ�R\x *��
> �܁���|x
> U�3���l���qa q� )�N ��k�L��a�2�ն���k NL
> 404 ��)i �g A�2 )L��W,���ܺ�V�w� �M7ڭCϣ����ZI�� be��$�f [�� � W� 7��5�6.}�ѽ b� ����C��%|�h�lj�D } @ F�"�(K ��xh�'6
>
> [2] My map class
>
>
> /** Identity mapper set by the user. */
> public static class MyFullyIndentityMapperWebdataScan
> extends Mapper<LongWritable, Text, LongWritable, Text>{
>
> // private LongWritable word = new LongWritable();
> private LongWritable word = new LongWritable();
> private Text rvalue = new Text();
>
> public void map(LongWritable key, Text value, Context context
> ) throws IOException, InterruptedException {
>
> System.out.println("1: " + key.getClass().getName() + " " + value.getClass().getName());
> System.out.println("2: " + context.getCurrentKey().getClass() + " " + context.getCurrentValue().getClass());
> context.write(key, value);
> }
> }
>
> [3] Output of execution
>
>
> Log Type: stdout
>
> Log Upload Time: 26-Jun-2015 11:58:53
>
> Log Length: 138
>
> 1: org.apache.hadoop.io.LongWritable org.apache.hadoop.io.Text 2: class org.apache.hadoop.io.LongWritable class org.apache.hadoop.io.Text
>
>
>
> Log Type: syslog
>
> Log Upload Time: 26-Jun-2015 11:58:53
>
> Log Length: 4367
>
> Showing 4096 bytes of 4367 total. Click here for the full log.
>
> operties from hadoop-metrics2.properties 2015-06-26 11:58:32,118 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2015-06-26 11:58:32,118 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started 2015-06-26 11:58:32,128 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens: 2015-06-26 11:58:32,128 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1435332416394_0009, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@5d7657) 2015-06-26 11:58:32,208 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now. 2015-06-26 11:58:32,439 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /tmp/hadoop-temp/nm-local-dir/usercache/xubuntu/appcache/application_1435332416394_0009 2015-06-26 11:58:32,611 INFO [main] org.apache.hadoop.conf.Configuration.deprecation:
> session.id is deprecated. Instead, use dfs.metrics.session-id 2015-06-26 11:58:33,038 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2015-06-26 11:58:33,180 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hadoop-coc-1:9000/output1-1434970707/part-m-00000:0+17853 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2015-06-26 11:58:33,255 INFO [main] org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2015-06-26 11:
> 5
> 8:33,269 INFO [main] org.apache.hadoop.mapred.MapTask: Starting flush of map output 2015-06-26 11:58:33,276 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.Text, received org.apache.hadoop.io.LongWritable at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1069) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:712) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89) at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112) at org.apache.hadoop.mapred.examples.MyWebDataScan$MyFullyIndentityMapperWebdataScan.map(MyWebDataScan.java:144) at org.apache.hadoop.mapred.examples.MyWebDataScan$MyFullyIndentityMapperWebdataScan.map(MyWebDataScan.java:131) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.
> h
> adoop.mapred.MapTask.runNewMapper(MapTask.java:784) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>
>
>
> --
> --
> Thanks,
>
>
Re: Invalid key type in the map task
Posted by Dieter De Witte <dr...@gmail.com>.
You need to show the driver class as well. Are you using textinputformat?
Are you aware that this standard inputformat will take as a value the
complete line (until newline separator), the key in that case is the
bitoffset in the file and definitely not the number you assume it will be.
2015-06-26 18:13 GMT+02:00 xeonmailinglist-gmail <xe...@gmail.com>
:
> Hi,
>
> I have this map class that is accepting input files with a key as
> LongWritable and a value of Text.
>
> The input file is in [1]. Here we can see that it contains a key as a Long
> (I think) and bytes as value.
> In [2], it is my map class. The goal of the map class is to read the input
> data, and sent it out as it is.
> I was expecting that the key was the index number, and the value was the
> gibberish. But, I am having a strange error [3], as the key is getting
> there a s Text, and not the index.
>
> Can anyone tell me why I get this error, and how I can solve it?
>
> [1] Input file
>
> xubuntu@hadoop-coc-1:~/Programs/hadoop$ hdfs dfs -cat /output1-1434970707/part-m-00000
> 15/06/26 12:01:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 0 SEQ "org.apache.hadoop.io.BytesWritable"org.apache.hadoop.io.BytesWritableI���#HA��u��BZ � � n�� tX eX����zZi��n dFw�{>�H� \� ��Bi�'�g b�Y�a �*4�7;���uL_��gs�� �`�� *
> 187 ���� tAj#�5*nX�b " ��>�˥k�c2�)�1z�1� ��6�l��� Q �be3'$ �G��!{� � u����z@�t����v���r�gj8fq�P��.�m ��� y ��n��!���_w�U8e�����v0���S���2c�_�ջ�R\x *��
> �܁���|x
> U�3���l���qa q� )�N ��k�L��a�2�ն���k NL
> 404 ��)i �g A�2 )L��W,���ܺ�V�w� �M7ڭCϣ����ZI�� be��$�f [�� � W� 7��5�6.}�ѽ b� ����C��%|�h�lj�D } @ F�"�(K ��xh�'6
>
> [2] My map class
>
>
> /** Identity mapper set by the user. */
> public static class MyFullyIndentityMapperWebdataScan
> extends Mapper<LongWritable, Text, LongWritable, Text>{
>
> // private LongWritable word = new LongWritable();
> private LongWritable word = new LongWritable();
> private Text rvalue = new Text();
>
> public void map(LongWritable key, Text value, Context context
> ) throws IOException, InterruptedException {
>
> System.out.println("1: " + key.getClass().getName() + " " + value.getClass().getName());
> System.out.println("2: " + context.getCurrentKey().getClass() + " " + context.getCurrentValue().getClass());
> context.write(key, value);
> }
> }
>
> [3] Output of execution
>
>
> Log Type: stdout
>
> Log Upload Time: 26-Jun-2015 11:58:53
>
> Log Length: 138
>
> 1: org.apache.hadoop.io.LongWritable org.apache.hadoop.io.Text 2: class org.apache.hadoop.io.LongWritable class org.apache.hadoop.io.Text
>
>
>
> Log Type: syslog
>
> Log Upload Time: 26-Jun-2015 11:58:53
>
> Log Length: 4367
>
> Showing 4096 bytes of 4367 total. Click here for the full log.
>
> operties from hadoop-metrics2.properties 2015-06-26 11:58:32,118 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2015-06-26 11:58:32,118 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started 2015-06-26 11:58:32,128 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens: 2015-06-26 11:58:32,128 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1435332416394_0009, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@5d7657) 2015-06-26 11:58:32,208 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now. 2015-06-26 11:58:32,439 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /tmp/hadoop-temp/nm-local-dir/usercache/xubuntu/appcache/application_1435332416394_0009 2015-06-26 11:58:32,611 INFO [main] org.apache.hadoop.conf.Configuration.deprecation:
> session.id is deprecated. Instead, use dfs.metrics.session-id 2015-06-26 11:58:33,038 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2015-06-26 11:58:33,180 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hadoop-coc-1:9000/output1-1434970707/part-m-00000:0+17853 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2015-06-26 11:58:33,255 INFO [main] org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2015-06-26 11:
> 5
> 8:33,269 INFO [main] org.apache.hadoop.mapred.MapTask: Starting flush of map output 2015-06-26 11:58:33,276 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.Text, received org.apache.hadoop.io.LongWritable at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1069) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:712) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89) at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112) at org.apache.hadoop.mapred.examples.MyWebDataScan$MyFullyIndentityMapperWebdataScan.map(MyWebDataScan.java:144) at org.apache.hadoop.mapred.examples.MyWebDataScan$MyFullyIndentityMapperWebdataScan.map(MyWebDataScan.java:131) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.
> h
> adoop.mapred.MapTask.runNewMapper(MapTask.java:784) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>
>
>
> --
> --
> Thanks,
>
>