You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Li Li <fa...@gmail.com> on 2014/06/25 10:29:41 UTC
TableInputFormatBase Cannot resolve the host name
I have a map reduce job using hbase table as input. when the job
starts, it says:
ERROR main org.apache.hadoop.hbase.mapreduce.TableInputFormatBase
Cannot resolve the host name for vc141/172.16.10.141 because of
javax.naming.CommunicationException: DNS error [Root exception is
java.net.PortUnreachableException: ICMP Port Unreachable]; remaining
name '141.10.16.172.in-addr.arpa'
the /etc/hosts file:
172.16.10.137 vc137
172.16.10.138 vc138
172.16.10.139 vc139
172.16.10.140 vc140
172.16.10.141 vc141
172.16.10.142 vc142
172.16.10.143 vc143
172.16.10.144 vc144
172.16.10.145 vc145
172.16.10.146 vc146
172.16.10.147 vc147
172.16.10.148 vc148
172.16.10.149 vc149
Re: TableInputFormatBase Cannot resolve the host name
Posted by Li Li <fa...@gmail.com>.
thanks, I got it.
On Wed, Jun 25, 2014 at 5:25 PM, Ted Yu <yu...@gmail.com> wrote:
> Please see https://issues.apache.org/jira/browse/HBASE-10906
>
> Cheers
>
> On Jun 25, 2014, at 2:12 AM, Li Li <fa...@gmail.com> wrote:
>
>> there is not any DNS server for me . you mean find name by ip? if no
>> DNS server, will it work correctly?
>>
>> On Wed, Jun 25, 2014 at 5:08 PM, Samir Ahmic <ah...@gmail.com> wrote:
>>> Do you use DNS server for name resolution ? Did you setup reverse DNS zone
>>> in your cluster ? I have seen this errors before when there is no reverse
>>> DNS setup. I believe that TableInputFormatBase class requires revers DNS
>>> name resolution.
>>>
>>> Regards
>>> Samir
>>>
>>>
>>> On Wed, Jun 25, 2014 at 10:57 AM, Li Li <fa...@gmail.com> wrote:
>>>
>>>> I have many map reduce jobs using hbase table as input. Others are all
>>>> correct.
>>>> This one is a little bit difference because it use both hdfs and hbase
>>>> as input source.
>>>> btw, even there are errors, the job can run successfully.
>>>>
>>>> My codes:
>>>>
>>>> 1. Hbase Table Mapper, mapper output key is Text and value is my custom
>>>> writable
>>>> public class UrlDedupHbaseMapper extends TableMapper<Text,
>>>> ExtractResultWritable> {
>>>>
>>>> @Override
>>>> public void map(NullWritable key, ExtractResultWritable value, Context
>>>> context)
>>>> throws IOException, InterruptedException {
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> 2. Hdfs Mapper, which read from sequence files, input key is
>>>> NullWritable, value is my custom writable. Mapper output key is Text
>>>> and value is my custome writable
>>>>
>>>> public class UrlDedupHbaseMapper extends TableMapper<Text,
>>>> ExtractResultWritable> {
>>>> public void map(ImmutableBytesWritable key, Result value, Context
>>>> context)
>>>> throws IOException, InterruptedException {
>>>> }
>>>> }
>>>>
>>>> 3. Reducer
>>>> public class UrlDedupReducer extends
>>>> Reducer<Text, ExtractResultWritable, NullWritable, ExtractResultWritable> {
>>>>
>>>> @Override
>>>> public void reduce(Text key, Iterable<ExtractResultWritable>
>>>> values, Context context)
>>>> throws IOException, InterruptedException {
>>>> }
>>>> }
>>>>
>>>> 4. driver class
>>>>
>>>> Configuration conf=getConf();
>>>> Configuration myConf = HBaseConfiguration.create(conf);
>>>> myConf.set("hbase.zookeeper.quorum", zkQuorum);
>>>> myConf.set("hbase.zookeeper.property.clientPort", zkPort);
>>>> myConf.set("mapred.child.java.opts", "-Xmx1g");
>>>> myConf.set("mapred.output.compress", "true");
>>>> Job job = new Job(myConf);
>>>> job.setJobName(UrlDedup.class.getSimpleName());
>>>>
>>>> job.setJarByClass(UrlDedup.class);
>>>> ArrayList<Path> fileList=new ArrayList<Path>();
>>>> FileSystem fs = FileSystem.get(conf);
>>>> Path inFile = new Path(args[0]);
>>>> FileStatus[] status = fs.listStatus(inFile);
>>>> for(FileStatus file:status){
>>>> fileList.add(file.getPath());
>>>> }
>>>>
>>>> for(Path path:fileList){
>>>> MultipleInputs.addInputPath(job, path,
>>>> SequenceFileInputFormat.class,UrlDedupHdfsMapper.class);
>>>> }
>>>> Scan urldbScan=new Scan();
>>>> urldbScan.setCaching(hbaseBlockCache);
>>>> urldbScan.setCacheBlocks(false);
>>>> urldbScan.addFamily(HbaseTools.CF_BT);
>>>>
>>>> job.setInputFormatClass(SequenceFileInputFormat.class);
>>>> TableMapReduceUtil.initTableMapperJob(HbaseTools.TB_URL_DB_BT,
>>>> urldbScan,
>>>> UrlDedupHbaseMapper.class, NullWritable.class,
>>>> ExtractResultWritable.class, job);
>>>> MultipleInputs.addInputPath(job, new Path("/null/path"),
>>>> TableInputFormat.class,
>>>> UrlDedupHbaseMapper.class);
>>>> job.setOutputFormatClass(SequenceFileOutputFormat.class);
>>>> FileOutputFormat.setOutputPath(job, new Path(args[1]));
>>>> job.setReducerClass(UrlDedupReducer.class);
>>>>
>>>> job.setOutputKeyClass(NullWritable.class);
>>>> job.setOutputValueClass(ExtractResultWritable.class);
>>>> job.setMapOutputKeyClass(Text.class);
>>>> job.setMapOutputValueClass(ExtractResultWritable.class);
>>>>
>>>> On Wed, Jun 25, 2014 at 4:49 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>> Can you ping vc141 from this machine ?
>>>>>
>>>>> Cheers
>>>>>
>>>>> On Jun 25, 2014, at 1:29 AM, Li Li <fa...@gmail.com> wrote:
>>>>>
>>>>>> I have a map reduce job using hbase table as input. when the job
>>>>>> starts, it says:
>>>>>>
>>>>>> ERROR main org.apache.hadoop.hbase.mapreduce.TableInputFormatBase
>>>>>> Cannot resolve the host name for vc141/172.16.10.141 because of
>>>>>> javax.naming.CommunicationException: DNS error [Root exception is
>>>>>> java.net.PortUnreachableException: ICMP Port Unreachable]; remaining
>>>>>> name '141.10.16.172.in-addr.arpa'
>>>>>>
>>>>>> the /etc/hosts file:
>>>>>> 172.16.10.137 vc137
>>>>>> 172.16.10.138 vc138
>>>>>> 172.16.10.139 vc139
>>>>>> 172.16.10.140 vc140
>>>>>> 172.16.10.141 vc141
>>>>>> 172.16.10.142 vc142
>>>>>> 172.16.10.143 vc143
>>>>>> 172.16.10.144 vc144
>>>>>> 172.16.10.145 vc145
>>>>>> 172.16.10.146 vc146
>>>>>> 172.16.10.147 vc147
>>>>>> 172.16.10.148 vc148
>>>>>> 172.16.10.149 vc149
>>>>
Re: TableInputFormatBase Cannot resolve the host name
Posted by Ted Yu <yu...@gmail.com>.
Please see https://issues.apache.org/jira/browse/HBASE-10906
Cheers
On Jun 25, 2014, at 2:12 AM, Li Li <fa...@gmail.com> wrote:
> there is not any DNS server for me . you mean find name by ip? if no
> DNS server, will it work correctly?
>
> On Wed, Jun 25, 2014 at 5:08 PM, Samir Ahmic <ah...@gmail.com> wrote:
>> Do you use DNS server for name resolution ? Did you setup reverse DNS zone
>> in your cluster ? I have seen this errors before when there is no reverse
>> DNS setup. I believe that TableInputFormatBase class requires revers DNS
>> name resolution.
>>
>> Regards
>> Samir
>>
>>
>> On Wed, Jun 25, 2014 at 10:57 AM, Li Li <fa...@gmail.com> wrote:
>>
>>> I have many map reduce jobs using hbase table as input. Others are all
>>> correct.
>>> This one is a little bit difference because it use both hdfs and hbase
>>> as input source.
>>> btw, even there are errors, the job can run successfully.
>>>
>>> My codes:
>>>
>>> 1. Hbase Table Mapper, mapper output key is Text and value is my custom
>>> writable
>>> public class UrlDedupHbaseMapper extends TableMapper<Text,
>>> ExtractResultWritable> {
>>>
>>> @Override
>>> public void map(NullWritable key, ExtractResultWritable value, Context
>>> context)
>>> throws IOException, InterruptedException {
>>>
>>> }
>>>
>>> }
>>>
>>> 2. Hdfs Mapper, which read from sequence files, input key is
>>> NullWritable, value is my custom writable. Mapper output key is Text
>>> and value is my custome writable
>>>
>>> public class UrlDedupHbaseMapper extends TableMapper<Text,
>>> ExtractResultWritable> {
>>> public void map(ImmutableBytesWritable key, Result value, Context
>>> context)
>>> throws IOException, InterruptedException {
>>> }
>>> }
>>>
>>> 3. Reducer
>>> public class UrlDedupReducer extends
>>> Reducer<Text, ExtractResultWritable, NullWritable, ExtractResultWritable> {
>>>
>>> @Override
>>> public void reduce(Text key, Iterable<ExtractResultWritable>
>>> values, Context context)
>>> throws IOException, InterruptedException {
>>> }
>>> }
>>>
>>> 4. driver class
>>>
>>> Configuration conf=getConf();
>>> Configuration myConf = HBaseConfiguration.create(conf);
>>> myConf.set("hbase.zookeeper.quorum", zkQuorum);
>>> myConf.set("hbase.zookeeper.property.clientPort", zkPort);
>>> myConf.set("mapred.child.java.opts", "-Xmx1g");
>>> myConf.set("mapred.output.compress", "true");
>>> Job job = new Job(myConf);
>>> job.setJobName(UrlDedup.class.getSimpleName());
>>>
>>> job.setJarByClass(UrlDedup.class);
>>> ArrayList<Path> fileList=new ArrayList<Path>();
>>> FileSystem fs = FileSystem.get(conf);
>>> Path inFile = new Path(args[0]);
>>> FileStatus[] status = fs.listStatus(inFile);
>>> for(FileStatus file:status){
>>> fileList.add(file.getPath());
>>> }
>>>
>>> for(Path path:fileList){
>>> MultipleInputs.addInputPath(job, path,
>>> SequenceFileInputFormat.class,UrlDedupHdfsMapper.class);
>>> }
>>> Scan urldbScan=new Scan();
>>> urldbScan.setCaching(hbaseBlockCache);
>>> urldbScan.setCacheBlocks(false);
>>> urldbScan.addFamily(HbaseTools.CF_BT);
>>>
>>> job.setInputFormatClass(SequenceFileInputFormat.class);
>>> TableMapReduceUtil.initTableMapperJob(HbaseTools.TB_URL_DB_BT,
>>> urldbScan,
>>> UrlDedupHbaseMapper.class, NullWritable.class,
>>> ExtractResultWritable.class, job);
>>> MultipleInputs.addInputPath(job, new Path("/null/path"),
>>> TableInputFormat.class,
>>> UrlDedupHbaseMapper.class);
>>> job.setOutputFormatClass(SequenceFileOutputFormat.class);
>>> FileOutputFormat.setOutputPath(job, new Path(args[1]));
>>> job.setReducerClass(UrlDedupReducer.class);
>>>
>>> job.setOutputKeyClass(NullWritable.class);
>>> job.setOutputValueClass(ExtractResultWritable.class);
>>> job.setMapOutputKeyClass(Text.class);
>>> job.setMapOutputValueClass(ExtractResultWritable.class);
>>>
>>> On Wed, Jun 25, 2014 at 4:49 PM, Ted Yu <yu...@gmail.com> wrote:
>>>> Can you ping vc141 from this machine ?
>>>>
>>>> Cheers
>>>>
>>>> On Jun 25, 2014, at 1:29 AM, Li Li <fa...@gmail.com> wrote:
>>>>
>>>>> I have a map reduce job using hbase table as input. when the job
>>>>> starts, it says:
>>>>>
>>>>> ERROR main org.apache.hadoop.hbase.mapreduce.TableInputFormatBase
>>>>> Cannot resolve the host name for vc141/172.16.10.141 because of
>>>>> javax.naming.CommunicationException: DNS error [Root exception is
>>>>> java.net.PortUnreachableException: ICMP Port Unreachable]; remaining
>>>>> name '141.10.16.172.in-addr.arpa'
>>>>>
>>>>> the /etc/hosts file:
>>>>> 172.16.10.137 vc137
>>>>> 172.16.10.138 vc138
>>>>> 172.16.10.139 vc139
>>>>> 172.16.10.140 vc140
>>>>> 172.16.10.141 vc141
>>>>> 172.16.10.142 vc142
>>>>> 172.16.10.143 vc143
>>>>> 172.16.10.144 vc144
>>>>> 172.16.10.145 vc145
>>>>> 172.16.10.146 vc146
>>>>> 172.16.10.147 vc147
>>>>> 172.16.10.148 vc148
>>>>> 172.16.10.149 vc149
>>>
Re: TableInputFormatBase Cannot resolve the host name
Posted by Samir Ahmic <ah...@gmail.com>.
Yes. From IP address to name that is reverse DNS lookup. Yes mapreduce will
work but you will see that error in logs.
Regards
On Jun 25, 2014 11:12 AM, "Li Li" <fa...@gmail.com> wrote:
> there is not any DNS server for me . you mean find name by ip? if no
> DNS server, will it work correctly?
>
> On Wed, Jun 25, 2014 at 5:08 PM, Samir Ahmic <ah...@gmail.com>
> wrote:
> > Do you use DNS server for name resolution ? Did you setup reverse DNS
> zone
> > in your cluster ? I have seen this errors before when there is no reverse
> > DNS setup. I believe that TableInputFormatBase class requires revers DNS
> > name resolution.
> >
> > Regards
> > Samir
> >
> >
> > On Wed, Jun 25, 2014 at 10:57 AM, Li Li <fa...@gmail.com> wrote:
> >
> >> I have many map reduce jobs using hbase table as input. Others are all
> >> correct.
> >> This one is a little bit difference because it use both hdfs and hbase
> >> as input source.
> >> btw, even there are errors, the job can run successfully.
> >>
> >> My codes:
> >>
> >> 1. Hbase Table Mapper, mapper output key is Text and value is my custom
> >> writable
> >> public class UrlDedupHbaseMapper extends TableMapper<Text,
> >> ExtractResultWritable> {
> >>
> >> @Override
> >> public void map(NullWritable key, ExtractResultWritable value, Context
> >> context)
> >> throws IOException, InterruptedException {
> >>
> >> }
> >>
> >> }
> >>
> >> 2. Hdfs Mapper, which read from sequence files, input key is
> >> NullWritable, value is my custom writable. Mapper output key is Text
> >> and value is my custome writable
> >>
> >> public class UrlDedupHbaseMapper extends TableMapper<Text,
> >> ExtractResultWritable> {
> >> public void map(ImmutableBytesWritable key, Result value, Context
> >> context)
> >> throws IOException, InterruptedException {
> >> }
> >> }
> >>
> >> 3. Reducer
> >> public class UrlDedupReducer extends
> >> Reducer<Text, ExtractResultWritable, NullWritable,
> ExtractResultWritable> {
> >>
> >> @Override
> >> public void reduce(Text key, Iterable<ExtractResultWritable>
> >> values, Context context)
> >> throws IOException, InterruptedException {
> >> }
> >> }
> >>
> >> 4. driver class
> >>
> >> Configuration conf=getConf();
> >> Configuration myConf = HBaseConfiguration.create(conf);
> >> myConf.set("hbase.zookeeper.quorum", zkQuorum);
> >> myConf.set("hbase.zookeeper.property.clientPort", zkPort);
> >> myConf.set("mapred.child.java.opts", "-Xmx1g");
> >> myConf.set("mapred.output.compress", "true");
> >> Job job = new Job(myConf);
> >> job.setJobName(UrlDedup.class.getSimpleName());
> >>
> >> job.setJarByClass(UrlDedup.class);
> >> ArrayList<Path> fileList=new ArrayList<Path>();
> >> FileSystem fs = FileSystem.get(conf);
> >> Path inFile = new Path(args[0]);
> >> FileStatus[] status = fs.listStatus(inFile);
> >> for(FileStatus file:status){
> >> fileList.add(file.getPath());
> >> }
> >>
> >> for(Path path:fileList){
> >> MultipleInputs.addInputPath(job, path,
> >>
> SequenceFileInputFormat.class,UrlDedupHdfsMapper.class);
> >> }
> >> Scan urldbScan=new Scan();
> >> urldbScan.setCaching(hbaseBlockCache);
> >> urldbScan.setCacheBlocks(false);
> >> urldbScan.addFamily(HbaseTools.CF_BT);
> >>
> >> job.setInputFormatClass(SequenceFileInputFormat.class);
> >> TableMapReduceUtil.initTableMapperJob(HbaseTools.TB_URL_DB_BT,
> >> urldbScan,
> >> UrlDedupHbaseMapper.class, NullWritable.class,
> >> ExtractResultWritable.class, job);
> >> MultipleInputs.addInputPath(job, new Path("/null/path"),
> >> TableInputFormat.class,
> >> UrlDedupHbaseMapper.class);
> >> job.setOutputFormatClass(SequenceFileOutputFormat.class);
> >> FileOutputFormat.setOutputPath(job, new Path(args[1]));
> >> job.setReducerClass(UrlDedupReducer.class);
> >>
> >> job.setOutputKeyClass(NullWritable.class);
> >> job.setOutputValueClass(ExtractResultWritable.class);
> >> job.setMapOutputKeyClass(Text.class);
> >> job.setMapOutputValueClass(ExtractResultWritable.class);
> >>
> >> On Wed, Jun 25, 2014 at 4:49 PM, Ted Yu <yu...@gmail.com> wrote:
> >> > Can you ping vc141 from this machine ?
> >> >
> >> > Cheers
> >> >
> >> > On Jun 25, 2014, at 1:29 AM, Li Li <fa...@gmail.com> wrote:
> >> >
> >> >> I have a map reduce job using hbase table as input. when the job
> >> >> starts, it says:
> >> >>
> >> >> ERROR main org.apache.hadoop.hbase.mapreduce.TableInputFormatBase
> >> >> Cannot resolve the host name for vc141/172.16.10.141 because of
> >> >> javax.naming.CommunicationException: DNS error [Root exception is
> >> >> java.net.PortUnreachableException: ICMP Port Unreachable]; remaining
> >> >> name '141.10.16.172.in-addr.arpa'
> >> >>
> >> >> the /etc/hosts file:
> >> >> 172.16.10.137 vc137
> >> >> 172.16.10.138 vc138
> >> >> 172.16.10.139 vc139
> >> >> 172.16.10.140 vc140
> >> >> 172.16.10.141 vc141
> >> >> 172.16.10.142 vc142
> >> >> 172.16.10.143 vc143
> >> >> 172.16.10.144 vc144
> >> >> 172.16.10.145 vc145
> >> >> 172.16.10.146 vc146
> >> >> 172.16.10.147 vc147
> >> >> 172.16.10.148 vc148
> >> >> 172.16.10.149 vc149
> >>
>
Re: TableInputFormatBase Cannot resolve the host name
Posted by Li Li <fa...@gmail.com>.
there is not any DNS server for me . you mean find name by ip? if no
DNS server, will it work correctly?
On Wed, Jun 25, 2014 at 5:08 PM, Samir Ahmic <ah...@gmail.com> wrote:
> Do you use DNS server for name resolution ? Did you setup reverse DNS zone
> in your cluster ? I have seen this errors before when there is no reverse
> DNS setup. I believe that TableInputFormatBase class requires revers DNS
> name resolution.
>
> Regards
> Samir
>
>
> On Wed, Jun 25, 2014 at 10:57 AM, Li Li <fa...@gmail.com> wrote:
>
>> I have many map reduce jobs using hbase table as input. Others are all
>> correct.
>> This one is a little bit difference because it use both hdfs and hbase
>> as input source.
>> btw, even there are errors, the job can run successfully.
>>
>> My codes:
>>
>> 1. Hbase Table Mapper, mapper output key is Text and value is my custom
>> writable
>> public class UrlDedupHbaseMapper extends TableMapper<Text,
>> ExtractResultWritable> {
>>
>> @Override
>> public void map(NullWritable key, ExtractResultWritable value, Context
>> context)
>> throws IOException, InterruptedException {
>>
>> }
>>
>> }
>>
>> 2. Hdfs Mapper, which read from sequence files, input key is
>> NullWritable, value is my custom writable. Mapper output key is Text
>> and value is my custome writable
>>
>> public class UrlDedupHbaseMapper extends TableMapper<Text,
>> ExtractResultWritable> {
>> public void map(ImmutableBytesWritable key, Result value, Context
>> context)
>> throws IOException, InterruptedException {
>> }
>> }
>>
>> 3. Reducer
>> public class UrlDedupReducer extends
>> Reducer<Text, ExtractResultWritable, NullWritable, ExtractResultWritable> {
>>
>> @Override
>> public void reduce(Text key, Iterable<ExtractResultWritable>
>> values, Context context)
>> throws IOException, InterruptedException {
>> }
>> }
>>
>> 4. driver class
>>
>> Configuration conf=getConf();
>> Configuration myConf = HBaseConfiguration.create(conf);
>> myConf.set("hbase.zookeeper.quorum", zkQuorum);
>> myConf.set("hbase.zookeeper.property.clientPort", zkPort);
>> myConf.set("mapred.child.java.opts", "-Xmx1g");
>> myConf.set("mapred.output.compress", "true");
>> Job job = new Job(myConf);
>> job.setJobName(UrlDedup.class.getSimpleName());
>>
>> job.setJarByClass(UrlDedup.class);
>> ArrayList<Path> fileList=new ArrayList<Path>();
>> FileSystem fs = FileSystem.get(conf);
>> Path inFile = new Path(args[0]);
>> FileStatus[] status = fs.listStatus(inFile);
>> for(FileStatus file:status){
>> fileList.add(file.getPath());
>> }
>>
>> for(Path path:fileList){
>> MultipleInputs.addInputPath(job, path,
>> SequenceFileInputFormat.class,UrlDedupHdfsMapper.class);
>> }
>> Scan urldbScan=new Scan();
>> urldbScan.setCaching(hbaseBlockCache);
>> urldbScan.setCacheBlocks(false);
>> urldbScan.addFamily(HbaseTools.CF_BT);
>>
>> job.setInputFormatClass(SequenceFileInputFormat.class);
>> TableMapReduceUtil.initTableMapperJob(HbaseTools.TB_URL_DB_BT,
>> urldbScan,
>> UrlDedupHbaseMapper.class, NullWritable.class,
>> ExtractResultWritable.class, job);
>> MultipleInputs.addInputPath(job, new Path("/null/path"),
>> TableInputFormat.class,
>> UrlDedupHbaseMapper.class);
>> job.setOutputFormatClass(SequenceFileOutputFormat.class);
>> FileOutputFormat.setOutputPath(job, new Path(args[1]));
>> job.setReducerClass(UrlDedupReducer.class);
>>
>> job.setOutputKeyClass(NullWritable.class);
>> job.setOutputValueClass(ExtractResultWritable.class);
>> job.setMapOutputKeyClass(Text.class);
>> job.setMapOutputValueClass(ExtractResultWritable.class);
>>
>> On Wed, Jun 25, 2014 at 4:49 PM, Ted Yu <yu...@gmail.com> wrote:
>> > Can you ping vc141 from this machine ?
>> >
>> > Cheers
>> >
>> > On Jun 25, 2014, at 1:29 AM, Li Li <fa...@gmail.com> wrote:
>> >
>> >> I have a map reduce job using hbase table as input. when the job
>> >> starts, it says:
>> >>
>> >> ERROR main org.apache.hadoop.hbase.mapreduce.TableInputFormatBase
>> >> Cannot resolve the host name for vc141/172.16.10.141 because of
>> >> javax.naming.CommunicationException: DNS error [Root exception is
>> >> java.net.PortUnreachableException: ICMP Port Unreachable]; remaining
>> >> name '141.10.16.172.in-addr.arpa'
>> >>
>> >> the /etc/hosts file:
>> >> 172.16.10.137 vc137
>> >> 172.16.10.138 vc138
>> >> 172.16.10.139 vc139
>> >> 172.16.10.140 vc140
>> >> 172.16.10.141 vc141
>> >> 172.16.10.142 vc142
>> >> 172.16.10.143 vc143
>> >> 172.16.10.144 vc144
>> >> 172.16.10.145 vc145
>> >> 172.16.10.146 vc146
>> >> 172.16.10.147 vc147
>> >> 172.16.10.148 vc148
>> >> 172.16.10.149 vc149
>>
Re: TableInputFormatBase Cannot resolve the host name
Posted by Samir Ahmic <ah...@gmail.com>.
Do you use DNS server for name resolution ? Did you setup reverse DNS zone
in your cluster ? I have seen this errors before when there is no reverse
DNS setup. I believe that TableInputFormatBase class requires revers DNS
name resolution.
Regards
Samir
On Wed, Jun 25, 2014 at 10:57 AM, Li Li <fa...@gmail.com> wrote:
> I have many map reduce jobs using hbase table as input. Others are all
> correct.
> This one is a little bit difference because it use both hdfs and hbase
> as input source.
> btw, even there are errors, the job can run successfully.
>
> My codes:
>
> 1. Hbase Table Mapper, mapper output key is Text and value is my custom
> writable
> public class UrlDedupHbaseMapper extends TableMapper<Text,
> ExtractResultWritable> {
>
> @Override
> public void map(NullWritable key, ExtractResultWritable value, Context
> context)
> throws IOException, InterruptedException {
>
> }
>
> }
>
> 2. Hdfs Mapper, which read from sequence files, input key is
> NullWritable, value is my custom writable. Mapper output key is Text
> and value is my custome writable
>
> public class UrlDedupHbaseMapper extends TableMapper<Text,
> ExtractResultWritable> {
> public void map(ImmutableBytesWritable key, Result value, Context
> context)
> throws IOException, InterruptedException {
> }
> }
>
> 3. Reducer
> public class UrlDedupReducer extends
> Reducer<Text, ExtractResultWritable, NullWritable, ExtractResultWritable> {
>
> @Override
> public void reduce(Text key, Iterable<ExtractResultWritable>
> values, Context context)
> throws IOException, InterruptedException {
> }
> }
>
> 4. driver class
>
> Configuration conf=getConf();
> Configuration myConf = HBaseConfiguration.create(conf);
> myConf.set("hbase.zookeeper.quorum", zkQuorum);
> myConf.set("hbase.zookeeper.property.clientPort", zkPort);
> myConf.set("mapred.child.java.opts", "-Xmx1g");
> myConf.set("mapred.output.compress", "true");
> Job job = new Job(myConf);
> job.setJobName(UrlDedup.class.getSimpleName());
>
> job.setJarByClass(UrlDedup.class);
> ArrayList<Path> fileList=new ArrayList<Path>();
> FileSystem fs = FileSystem.get(conf);
> Path inFile = new Path(args[0]);
> FileStatus[] status = fs.listStatus(inFile);
> for(FileStatus file:status){
> fileList.add(file.getPath());
> }
>
> for(Path path:fileList){
> MultipleInputs.addInputPath(job, path,
> SequenceFileInputFormat.class,UrlDedupHdfsMapper.class);
> }
> Scan urldbScan=new Scan();
> urldbScan.setCaching(hbaseBlockCache);
> urldbScan.setCacheBlocks(false);
> urldbScan.addFamily(HbaseTools.CF_BT);
>
> job.setInputFormatClass(SequenceFileInputFormat.class);
> TableMapReduceUtil.initTableMapperJob(HbaseTools.TB_URL_DB_BT,
> urldbScan,
> UrlDedupHbaseMapper.class, NullWritable.class,
> ExtractResultWritable.class, job);
> MultipleInputs.addInputPath(job, new Path("/null/path"),
> TableInputFormat.class,
> UrlDedupHbaseMapper.class);
> job.setOutputFormatClass(SequenceFileOutputFormat.class);
> FileOutputFormat.setOutputPath(job, new Path(args[1]));
> job.setReducerClass(UrlDedupReducer.class);
>
> job.setOutputKeyClass(NullWritable.class);
> job.setOutputValueClass(ExtractResultWritable.class);
> job.setMapOutputKeyClass(Text.class);
> job.setMapOutputValueClass(ExtractResultWritable.class);
>
> On Wed, Jun 25, 2014 at 4:49 PM, Ted Yu <yu...@gmail.com> wrote:
> > Can you ping vc141 from this machine ?
> >
> > Cheers
> >
> > On Jun 25, 2014, at 1:29 AM, Li Li <fa...@gmail.com> wrote:
> >
> >> I have a map reduce job using hbase table as input. when the job
> >> starts, it says:
> >>
> >> ERROR main org.apache.hadoop.hbase.mapreduce.TableInputFormatBase
> >> Cannot resolve the host name for vc141/172.16.10.141 because of
> >> javax.naming.CommunicationException: DNS error [Root exception is
> >> java.net.PortUnreachableException: ICMP Port Unreachable]; remaining
> >> name '141.10.16.172.in-addr.arpa'
> >>
> >> the /etc/hosts file:
> >> 172.16.10.137 vc137
> >> 172.16.10.138 vc138
> >> 172.16.10.139 vc139
> >> 172.16.10.140 vc140
> >> 172.16.10.141 vc141
> >> 172.16.10.142 vc142
> >> 172.16.10.143 vc143
> >> 172.16.10.144 vc144
> >> 172.16.10.145 vc145
> >> 172.16.10.146 vc146
> >> 172.16.10.147 vc147
> >> 172.16.10.148 vc148
> >> 172.16.10.149 vc149
>
Re: TableInputFormatBase Cannot resolve the host name
Posted by Li Li <fa...@gmail.com>.
I have many map reduce jobs using hbase table as input. Others are all correct.
This one is a little bit difference because it use both hdfs and hbase
as input source.
btw, even there are errors, the job can run successfully.
My codes:
1. Hbase Table Mapper, mapper output key is Text and value is my custom writable
public class UrlDedupHbaseMapper extends TableMapper<Text,
ExtractResultWritable> {
@Override
public void map(NullWritable key, ExtractResultWritable value, Context context)
throws IOException, InterruptedException {
}
}
2. Hdfs Mapper, which read from sequence files, input key is
NullWritable, value is my custom writable. Mapper output key is Text
and value is my custome writable
public class UrlDedupHbaseMapper extends TableMapper<Text,
ExtractResultWritable> {
public void map(ImmutableBytesWritable key, Result value, Context context)
throws IOException, InterruptedException {
}
}
3. Reducer
public class UrlDedupReducer extends
Reducer<Text, ExtractResultWritable, NullWritable, ExtractResultWritable> {
@Override
public void reduce(Text key, Iterable<ExtractResultWritable>
values, Context context)
throws IOException, InterruptedException {
}
}
4. driver class
Configuration conf=getConf();
Configuration myConf = HBaseConfiguration.create(conf);
myConf.set("hbase.zookeeper.quorum", zkQuorum);
myConf.set("hbase.zookeeper.property.clientPort", zkPort);
myConf.set("mapred.child.java.opts", "-Xmx1g");
myConf.set("mapred.output.compress", "true");
Job job = new Job(myConf);
job.setJobName(UrlDedup.class.getSimpleName());
job.setJarByClass(UrlDedup.class);
ArrayList<Path> fileList=new ArrayList<Path>();
FileSystem fs = FileSystem.get(conf);
Path inFile = new Path(args[0]);
FileStatus[] status = fs.listStatus(inFile);
for(FileStatus file:status){
fileList.add(file.getPath());
}
for(Path path:fileList){
MultipleInputs.addInputPath(job, path,
SequenceFileInputFormat.class,UrlDedupHdfsMapper.class);
}
Scan urldbScan=new Scan();
urldbScan.setCaching(hbaseBlockCache);
urldbScan.setCacheBlocks(false);
urldbScan.addFamily(HbaseTools.CF_BT);
job.setInputFormatClass(SequenceFileInputFormat.class);
TableMapReduceUtil.initTableMapperJob(HbaseTools.TB_URL_DB_BT, urldbScan,
UrlDedupHbaseMapper.class, NullWritable.class,
ExtractResultWritable.class, job);
MultipleInputs.addInputPath(job, new Path("/null/path"),
TableInputFormat.class,
UrlDedupHbaseMapper.class);
job.setOutputFormatClass(SequenceFileOutputFormat.class);
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.setReducerClass(UrlDedupReducer.class);
job.setOutputKeyClass(NullWritable.class);
job.setOutputValueClass(ExtractResultWritable.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(ExtractResultWritable.class);
On Wed, Jun 25, 2014 at 4:49 PM, Ted Yu <yu...@gmail.com> wrote:
> Can you ping vc141 from this machine ?
>
> Cheers
>
> On Jun 25, 2014, at 1:29 AM, Li Li <fa...@gmail.com> wrote:
>
>> I have a map reduce job using hbase table as input. when the job
>> starts, it says:
>>
>> ERROR main org.apache.hadoop.hbase.mapreduce.TableInputFormatBase
>> Cannot resolve the host name for vc141/172.16.10.141 because of
>> javax.naming.CommunicationException: DNS error [Root exception is
>> java.net.PortUnreachableException: ICMP Port Unreachable]; remaining
>> name '141.10.16.172.in-addr.arpa'
>>
>> the /etc/hosts file:
>> 172.16.10.137 vc137
>> 172.16.10.138 vc138
>> 172.16.10.139 vc139
>> 172.16.10.140 vc140
>> 172.16.10.141 vc141
>> 172.16.10.142 vc142
>> 172.16.10.143 vc143
>> 172.16.10.144 vc144
>> 172.16.10.145 vc145
>> 172.16.10.146 vc146
>> 172.16.10.147 vc147
>> 172.16.10.148 vc148
>> 172.16.10.149 vc149
Re: TableInputFormatBase Cannot resolve the host name
Posted by Li Li <fa...@gmail.com>.
yes
[hadoop@vc138 ~]$ ping vc141
PING vc141 (172.16.10.141) 56(84) bytes of data.
64 bytes from vc141 (172.16.10.141): icmp_seq=1 ttl=64 time=0.118 ms
On Wed, Jun 25, 2014 at 4:49 PM, Ted Yu <yu...@gmail.com> wrote:
> Can you ping vc141 from this machine ?
>
> Cheers
>
> On Jun 25, 2014, at 1:29 AM, Li Li <fa...@gmail.com> wrote:
>
>> I have a map reduce job using hbase table as input. when the job
>> starts, it says:
>>
>> ERROR main org.apache.hadoop.hbase.mapreduce.TableInputFormatBase
>> Cannot resolve the host name for vc141/172.16.10.141 because of
>> javax.naming.CommunicationException: DNS error [Root exception is
>> java.net.PortUnreachableException: ICMP Port Unreachable]; remaining
>> name '141.10.16.172.in-addr.arpa'
>>
>> the /etc/hosts file:
>> 172.16.10.137 vc137
>> 172.16.10.138 vc138
>> 172.16.10.139 vc139
>> 172.16.10.140 vc140
>> 172.16.10.141 vc141
>> 172.16.10.142 vc142
>> 172.16.10.143 vc143
>> 172.16.10.144 vc144
>> 172.16.10.145 vc145
>> 172.16.10.146 vc146
>> 172.16.10.147 vc147
>> 172.16.10.148 vc148
>> 172.16.10.149 vc149
Re: TableInputFormatBase Cannot resolve the host name
Posted by Ted Yu <yu...@gmail.com>.
Can you ping vc141 from this machine ?
Cheers
On Jun 25, 2014, at 1:29 AM, Li Li <fa...@gmail.com> wrote:
> I have a map reduce job using hbase table as input. when the job
> starts, it says:
>
> ERROR main org.apache.hadoop.hbase.mapreduce.TableInputFormatBase
> Cannot resolve the host name for vc141/172.16.10.141 because of
> javax.naming.CommunicationException: DNS error [Root exception is
> java.net.PortUnreachableException: ICMP Port Unreachable]; remaining
> name '141.10.16.172.in-addr.arpa'
>
> the /etc/hosts file:
> 172.16.10.137 vc137
> 172.16.10.138 vc138
> 172.16.10.139 vc139
> 172.16.10.140 vc140
> 172.16.10.141 vc141
> 172.16.10.142 vc142
> 172.16.10.143 vc143
> 172.16.10.144 vc144
> 172.16.10.145 vc145
> 172.16.10.146 vc146
> 172.16.10.147 vc147
> 172.16.10.148 vc148
> 172.16.10.149 vc149