You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Giuseppe Valente <gv...@arista.com.INVALID> on 2022/02/24 22:26:08 UTC
Using Hadoop Streaming with formats based on org.apache.hadoop.mapreduce.InputFormat
Hi,
I have my own input format based on the following hierarchy:
org.apache.hadoop.hbase.mapreduce.HFileInputFormat
-> org.apache.hadoop.mapreduce.lib.input.FileInputFormat
-> org.apache.hadoop.mapreduce.InputFormat
I’m running the following:
hadoop jar hadoop-streaming-3.1.4.jar -inputformat org.apache.hadoop.hbase.mapreduce.HFileInputFormat
Which gives me the following error:
Exception in thread "main" java.lang.RuntimeException: class org.apache.hadoop.hbase.mapreduce.HFileInputFormat not org.apache.hadoop.mapred.InputFormat
at org.apache.hadoop.conf.Configuration.setClass(Configuration.java:2712)
at org.apache.hadoop.mapred.JobConf.setInputFormat(JobConf.java:700)
at org.apache.hadoop.streaming.StreamJob.setJobConf(StreamJob.java:798)
at org.apache.hadoop.streaming.StreamJob.run(StreamJob.java:126)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:50)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
I see that StreamJob.java relies mainly on legacy org.apache.hadoop.mapred.* objects, is there a way to have it use the org.apache.hadoop.hbase.mapreduce.* equivalents instead?
Thanks,
Giuseppe
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
Re: Using Hadoop Streaming with formats based on org.apache.hadoop.mapreduce.InputFormat
Posted by Niketh Nikky <ni...@gmail.com>.
Unsubscribe!!!!!!!!!!!!!!!!!!!!!
Thanks
Niketh
> On Feb 24, 2022, at 4:26 PM, Giuseppe Valente <gv...@arista.com.invalid> wrote:
>
> Hi,
>
> I have my own input format based on the following hierarchy:
>
> org.apache.hadoop.hbase.mapreduce.HFileInputFormat
> -> org.apache.hadoop.mapreduce.lib.input.FileInputFormat
> -> org.apache.hadoop.mapreduce.InputFormat
>
> I’m running the following:
>
> hadoop jar hadoop-streaming-3.1.4.jar -inputformat org.apache.hadoop.hbase.mapreduce.HFileInputFormat
>
> Which gives me the following error:
>
> Exception in thread "main" java.lang.RuntimeException: class org.apache.hadoop.hbase.mapreduce.HFileInputFormat not org.apache.hadoop.mapred.InputFormat
> at org.apache.hadoop.conf.Configuration.setClass(Configuration.java:2712)
> at org.apache.hadoop.mapred.JobConf.setInputFormat(JobConf.java:700)
> at org.apache.hadoop.streaming.StreamJob.setJobConf(StreamJob.java:798)
> at org.apache.hadoop.streaming.StreamJob.run(StreamJob.java:126)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:50)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:564)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
>
> I see that StreamJob.java relies mainly on legacy org.apache.hadoop.mapred.* objects, is there a way to have it use the org.apache.hadoop.hbase.mapreduce.* equivalents instead?
>
> Thanks,
> Giuseppe
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org