You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by AnilKumar B <ak...@gmail.com> on 2014/01/07 04:12:18 UTC
Facing problem while using MultiTableOutputFormat
Hi,
In my MR job, I need to write output into multiple tables, So I am
using MultiTableOutputFormat as below. But I am getting
TableNotFoundException.
I am attaching code snippet below, Is this the correct way to use
MultiTableOutputFormat ?
Job class:
public int run(String[] args) throws Exception {
System.out.println( " Running with on tables "+args[1]+ " and "+args[2]+"
with zk "+args[3]);
Configuration hbaseConf = HBaseConfiguration.create(getConf());
// hbaseConf.set(Constants.HBASE_ZOOKEEPER_QUORUM_PROP,
Constants.HBASE_OS_CL1_QUORUM);
hbaseConf.set(Constants.HBASE_ZOOKEEPER_QUORUM_PROP, args[3]);
Job job = new Job(hbaseConf);
job.setJarByClass(MultiTableTestJob.class);
job.setInputFormatClass(TextInputFormat.class);
job.setMapperClass(MultiTableTestMapper.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(Text.class);
job.setReducerClass(MultiTableTestReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
FileInputFormat.setInputPaths(job, new Path(args[0]));
job.setOutputFormatClass(MultiTableOutputFormat.class);
TableMapReduceUtil.addDependencyJars(job);
TableMapReduceUtil.addDependencyJars(job.getConfiguration());
return job.waitForCompletion(true) == true ? 0 : -1;
}
public static void main(String[] args) throws Exception{
Configuration configuration = new Configuration();
configuration.set("HBASE_DEST_TABLE", args[1]);
configuration.set("HBASE_LOOKUP_TABLE", args[2]);
ToolRunner.run(configuration, new CISuperSessionJob(), args);
}
Reducer Class:
private ImmutableBytesWritable tbl1;
private ImmutableBytesWritable tbl2;
protected void setup(Context context) throws IOException
,InterruptedException {
Configuration c = context.getConfiguration();
tbl1 = new
ImmutableBytesWritable(Bytes.toBytes(context.getConfiguration().get("HBASE_DEST_TABLE")));
tbl2 = new
ImmutableBytesWritable(Bytes.toBytes(context.getConfiguration().get("HBASE_LOOKUP_TABLE")));
};
protected void reduce(Text key, java.lang.Iterable<Text>
values, Context context) throws IOException ,InterruptedException {
//
if (some condition) {
Put put = getSessionPut(key, vc);
if (put != null) {
context.write(tbl1, put);
}
} else {
//
Put put = getEventPut(key, vc);
context.write(tbl2, put);
}
}
}
Exception:
org.apache.hadoop.hbase.TableNotFoundException: mapred.reduce.tasks=100
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:999)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:864)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
at
org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
at
org.apache.hadoop.hbase.mapreduce.MultiTableOutputFormat$MultiTableRecordWriter.getTable(MultiTableOutputFormat.java:101)
at
org.apache.hadoop.hbase.mapreduce.MultiTableOutputFormat$MultiTableRecordWriter.write(MultiTableOutputFormat.java:127)
at
org.apache.hadoop.hbase.mapreduce.MultiTableOutputFormat$MultiTableRecordWriter.write(MultiTableOutputFormat.java:68)
at
org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:586)
at
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at
Thanks & Regards,
B Anil Kumar.
Re: Facing problem while using MultiTableOutputFormat
Posted by AnilKumar B <ak...@gmail.com>.
Hi Ted,
I modified the Job class as below. Then it's working.
Job class:
public int run(String[] args) throws Exception {
System.out.println( " Running with on tables "+args[1]+ " and "+args[2]+"
with zk "+args[3]);
Configuration hbaseConf = HBaseConfiguration.create(getConf());
// hbaseConf.set(Constants.HBASE_ZOOKEEPER_QUORUM_PROP,
Constants.HBASE_OS_CL1_QUORUM);
hbaseConf.set(Constants.HBASE_ZOOKEEPER_QUORUM_PROP, args[3]);
*hbaseConf.set("HBASE_DEST_CI_TABLE", args[1]);*
* hbaseConf.set("HBASE_CI_LOOKUP_TABLE", args[2]);*
Job job = new Job(hbaseConf);
job.setJarByClass(MultiTableTestJob.class);
job.setInputFormatClass(TextInputFormat.class);
job.setMapperClass(MultiTableTestMapper.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(Text.class);
job.setReducerClass(MultiTableTestReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
FileInputFormat.setInputPaths(job, new Path(args[0]));
job.setOutputFormatClass(MultiTableOutputFormat.class);
TableMapReduceUtil.addDependencyJars(job);
TableMapReduceUtil.addDependencyJars(job.getConfiguration());
return job.waitForCompletion(true) == true ? 0 : -1;
}
public static void main(String[] args) throws Exception{
Configuration configuration = new Configuration();
*// configuration.set("HBASE_DEST_TABLE", args[1]);*
* // configuration.set("HBASE_LOOKUP_TABLE", args[2]);*
ToolRunner.run(configuration, new CISuperSessionJob(), args);
}
*When I set the properties in Configuration object, and when I try to get
from context in reducer, it's giving wrong values. But when set them to
hbaseConf, then it's working fine. Why is like that?*
Thanks & Regards,
B Anil Kumar.
On Tue, Jan 7, 2014 at 9:52 AM, AnilKumar B <ak...@gmail.com> wrote:
> Hi Ted,
>
> It seems some issue in my code itself, will retry and update you.
>
> Thanks,
> B Anil Kumar.
>
> Thanks & Regards,
> B Anil Kumar.
>
>
> On Tue, Jan 7, 2014 at 9:35 AM, AnilKumar B <ak...@gmail.com> wrote:
>
>> Hi Ted,
>>
>> System.out.println( " Running with on tables "+args[1]+ " and "+args[2]+"
>> with zk "+args[3]);
>>
>> What was the output from the above ?
>> >> Running with on tables ci_history and ci_lookup with zk 10.9.208.71
>>
>> Tables are exist on hbase:
>> hbase(main):001:0> list
>> TABLE
>> ci_history
>> ci_lookup
>>
>>
>>
>>
>> Thanks & Regards,
>> B Anil Kumar.
>>
>>
>> On Tue, Jan 7, 2014 at 9:07 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> System.out.println( " Running with on tables "+args[1]+ " and
>>> "+args[2]+"
>>> with zk "+args[3]);
>>>
>>> What was the output from the above ?
>>>
>>> I would expect a call similar to the following in your run() method -
>>> this
>>> comes from TestTableMapReduce.java:
>>>
>>> TableMapReduceUtil.initTableReducerJob(
>>> Bytes.toString(table.getTableName()),
>>> IdentityTableReducer.class, job);
>>>
>>>
>>> On Mon, Jan 6, 2014 at 7:12 PM, AnilKumar B <ak...@gmail.com>
>>> wrote:
>>>
>>> > Hi,
>>> >
>>> > In my MR job, I need to write output into multiple tables, So I am
>>> > using MultiTableOutputFormat as below. But I am getting
>>> > TableNotFoundException.
>>> >
>>> > I am attaching code snippet below, Is this the correct way to use
>>> > MultiTableOutputFormat ?
>>> >
>>> >
>>> > Job class:
>>> > public int run(String[] args) throws Exception {
>>> > System.out.println( " Running with on tables "+args[1]+ " and
>>> "+args[2]+"
>>> > with zk "+args[3]);
>>> > Configuration hbaseConf = HBaseConfiguration.create(getConf());
>>> > // hbaseConf.set(Constants.HBASE_ZOOKEEPER_QUORUM_PROP,
>>> > Constants.HBASE_OS_CL1_QUORUM);
>>> > hbaseConf.set(Constants.HBASE_ZOOKEEPER_QUORUM_PROP, args[3]);
>>> > Job job = new Job(hbaseConf);
>>> > job.setJarByClass(MultiTableTestJob.class);
>>> > job.setInputFormatClass(TextInputFormat.class);
>>> > job.setMapperClass(MultiTableTestMapper.class);
>>> > job.setMapOutputKeyClass(Text.class);
>>> > job.setMapOutputValueClass(Text.class);
>>> > job.setReducerClass(MultiTableTestReducer.class);
>>> > job.setOutputKeyClass(Text.class);
>>> > job.setOutputValueClass(Text.class);
>>> > FileInputFormat.setInputPaths(job, new Path(args[0]));
>>> > job.setOutputFormatClass(MultiTableOutputFormat.class);
>>> > TableMapReduceUtil.addDependencyJars(job);
>>> > TableMapReduceUtil.addDependencyJars(job.getConfiguration());
>>> > return job.waitForCompletion(true) == true ? 0 : -1;
>>> > }
>>> > public static void main(String[] args) throws Exception{
>>> > Configuration configuration = new Configuration();
>>> > configuration.set("HBASE_DEST_TABLE", args[1]);
>>> > configuration.set("HBASE_LOOKUP_TABLE", args[2]);
>>> > ToolRunner.run(configuration, new CISuperSessionJob(), args);
>>> > }
>>> >
>>> > Reducer Class:
>>> >
>>> > private ImmutableBytesWritable tbl1;
>>> > private ImmutableBytesWritable tbl2;
>>> >
>>> > protected void setup(Context context) throws IOException
>>> > ,InterruptedException {
>>> > Configuration c = context.getConfiguration();
>>> > tbl1 = new
>>> >
>>> >
>>> ImmutableBytesWritable(Bytes.toBytes(context.getConfiguration().get("HBASE_DEST_TABLE")));
>>> > tbl2 = new
>>> >
>>> >
>>> ImmutableBytesWritable(Bytes.toBytes(context.getConfiguration().get("HBASE_LOOKUP_TABLE")));
>>> > };
>>> >
>>> > protected void reduce(Text key, java.lang.Iterable<Text>
>>> > values, Context context) throws IOException ,InterruptedException {
>>> > //
>>> > if (some condition) {
>>> > Put put = getSessionPut(key, vc);
>>> > if (put != null) {
>>> > context.write(tbl1, put);
>>> > }
>>> > } else {
>>> > //
>>> > Put put = getEventPut(key, vc);
>>> > context.write(tbl2, put);
>>> > }
>>> > }
>>> > }
>>> >
>>> >
>>> > Exception:
>>> > org.apache.hadoop.hbase.TableNotFoundException: mapred.reduce.tasks=100
>>> > at
>>> >
>>> >
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:999)
>>> > at
>>> >
>>> >
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:864)
>>> > at
>>> >
>>> >
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>>> > at
>>> > org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>>> > at
>>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>>> > at
>>> >
>>> >
>>> org.apache.hadoop.hbase.mapreduce.MultiTableOutputFormat$MultiTableRecordWriter.getTable(MultiTableOutputFormat.java:101)
>>> > at
>>> >
>>> >
>>> org.apache.hadoop.hbase.mapreduce.MultiTableOutputFormat$MultiTableRecordWriter.write(MultiTableOutputFormat.java:127)
>>> > at
>>> >
>>> >
>>> org.apache.hadoop.hbase.mapreduce.MultiTableOutputFormat$MultiTableRecordWriter.write(MultiTableOutputFormat.java:68)
>>> > at
>>> >
>>> >
>>> org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:586)
>>> > at
>>> >
>>> >
>>> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
>>> > at
>>> >
>>> >
>>> >
>>> > Thanks & Regards,
>>> > B Anil Kumar.
>>> >
>>>
>>
>>
>
Re: Facing problem while using MultiTableOutputFormat
Posted by AnilKumar B <ak...@gmail.com>.
Hi Ted,
It seems some issue in my code itself, will retry and update you.
Thanks,
B Anil Kumar.
Thanks & Regards,
B Anil Kumar.
On Tue, Jan 7, 2014 at 9:35 AM, AnilKumar B <ak...@gmail.com> wrote:
> Hi Ted,
>
> System.out.println( " Running with on tables "+args[1]+ " and "+args[2]+"
> with zk "+args[3]);
>
> What was the output from the above ?
> >> Running with on tables ci_history and ci_lookup with zk 10.9.208.71
>
> Tables are exist on hbase:
> hbase(main):001:0> list
> TABLE
> ci_history
> ci_lookup
>
>
>
>
> Thanks & Regards,
> B Anil Kumar.
>
>
> On Tue, Jan 7, 2014 at 9:07 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> System.out.println( " Running with on tables "+args[1]+ " and "+args[2]+"
>> with zk "+args[3]);
>>
>> What was the output from the above ?
>>
>> I would expect a call similar to the following in your run() method - this
>> comes from TestTableMapReduce.java:
>>
>> TableMapReduceUtil.initTableReducerJob(
>> Bytes.toString(table.getTableName()),
>> IdentityTableReducer.class, job);
>>
>>
>> On Mon, Jan 6, 2014 at 7:12 PM, AnilKumar B <ak...@gmail.com>
>> wrote:
>>
>> > Hi,
>> >
>> > In my MR job, I need to write output into multiple tables, So I am
>> > using MultiTableOutputFormat as below. But I am getting
>> > TableNotFoundException.
>> >
>> > I am attaching code snippet below, Is this the correct way to use
>> > MultiTableOutputFormat ?
>> >
>> >
>> > Job class:
>> > public int run(String[] args) throws Exception {
>> > System.out.println( " Running with on tables "+args[1]+ " and
>> "+args[2]+"
>> > with zk "+args[3]);
>> > Configuration hbaseConf = HBaseConfiguration.create(getConf());
>> > // hbaseConf.set(Constants.HBASE_ZOOKEEPER_QUORUM_PROP,
>> > Constants.HBASE_OS_CL1_QUORUM);
>> > hbaseConf.set(Constants.HBASE_ZOOKEEPER_QUORUM_PROP, args[3]);
>> > Job job = new Job(hbaseConf);
>> > job.setJarByClass(MultiTableTestJob.class);
>> > job.setInputFormatClass(TextInputFormat.class);
>> > job.setMapperClass(MultiTableTestMapper.class);
>> > job.setMapOutputKeyClass(Text.class);
>> > job.setMapOutputValueClass(Text.class);
>> > job.setReducerClass(MultiTableTestReducer.class);
>> > job.setOutputKeyClass(Text.class);
>> > job.setOutputValueClass(Text.class);
>> > FileInputFormat.setInputPaths(job, new Path(args[0]));
>> > job.setOutputFormatClass(MultiTableOutputFormat.class);
>> > TableMapReduceUtil.addDependencyJars(job);
>> > TableMapReduceUtil.addDependencyJars(job.getConfiguration());
>> > return job.waitForCompletion(true) == true ? 0 : -1;
>> > }
>> > public static void main(String[] args) throws Exception{
>> > Configuration configuration = new Configuration();
>> > configuration.set("HBASE_DEST_TABLE", args[1]);
>> > configuration.set("HBASE_LOOKUP_TABLE", args[2]);
>> > ToolRunner.run(configuration, new CISuperSessionJob(), args);
>> > }
>> >
>> > Reducer Class:
>> >
>> > private ImmutableBytesWritable tbl1;
>> > private ImmutableBytesWritable tbl2;
>> >
>> > protected void setup(Context context) throws IOException
>> > ,InterruptedException {
>> > Configuration c = context.getConfiguration();
>> > tbl1 = new
>> >
>> >
>> ImmutableBytesWritable(Bytes.toBytes(context.getConfiguration().get("HBASE_DEST_TABLE")));
>> > tbl2 = new
>> >
>> >
>> ImmutableBytesWritable(Bytes.toBytes(context.getConfiguration().get("HBASE_LOOKUP_TABLE")));
>> > };
>> >
>> > protected void reduce(Text key, java.lang.Iterable<Text>
>> > values, Context context) throws IOException ,InterruptedException {
>> > //
>> > if (some condition) {
>> > Put put = getSessionPut(key, vc);
>> > if (put != null) {
>> > context.write(tbl1, put);
>> > }
>> > } else {
>> > //
>> > Put put = getEventPut(key, vc);
>> > context.write(tbl2, put);
>> > }
>> > }
>> > }
>> >
>> >
>> > Exception:
>> > org.apache.hadoop.hbase.TableNotFoundException: mapred.reduce.tasks=100
>> > at
>> >
>> >
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:999)
>> > at
>> >
>> >
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:864)
>> > at
>> >
>> >
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>> > at
>> > org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>> > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>> > at
>> >
>> >
>> org.apache.hadoop.hbase.mapreduce.MultiTableOutputFormat$MultiTableRecordWriter.getTable(MultiTableOutputFormat.java:101)
>> > at
>> >
>> >
>> org.apache.hadoop.hbase.mapreduce.MultiTableOutputFormat$MultiTableRecordWriter.write(MultiTableOutputFormat.java:127)
>> > at
>> >
>> >
>> org.apache.hadoop.hbase.mapreduce.MultiTableOutputFormat$MultiTableRecordWriter.write(MultiTableOutputFormat.java:68)
>> > at
>> >
>> >
>> org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:586)
>> > at
>> >
>> >
>> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
>> > at
>> >
>> >
>> >
>> > Thanks & Regards,
>> > B Anil Kumar.
>> >
>>
>
>
Re: Facing problem while using MultiTableOutputFormat
Posted by AnilKumar B <ak...@gmail.com>.
Hi Ted,
System.out.println( " Running with on tables "+args[1]+ " and "+args[2]+"
with zk "+args[3]);
What was the output from the above ?
>> Running with on tables ci_history and ci_lookup with zk 10.9.208.71
Tables are exist on hbase:
hbase(main):001:0> list
TABLE
ci_history
ci_lookup
Thanks & Regards,
B Anil Kumar.
On Tue, Jan 7, 2014 at 9:07 AM, Ted Yu <yu...@gmail.com> wrote:
> System.out.println( " Running with on tables "+args[1]+ " and "+args[2]+"
> with zk "+args[3]);
>
> What was the output from the above ?
>
> I would expect a call similar to the following in your run() method - this
> comes from TestTableMapReduce.java:
>
> TableMapReduceUtil.initTableReducerJob(
> Bytes.toString(table.getTableName()),
> IdentityTableReducer.class, job);
>
>
> On Mon, Jan 6, 2014 at 7:12 PM, AnilKumar B <ak...@gmail.com> wrote:
>
> > Hi,
> >
> > In my MR job, I need to write output into multiple tables, So I am
> > using MultiTableOutputFormat as below. But I am getting
> > TableNotFoundException.
> >
> > I am attaching code snippet below, Is this the correct way to use
> > MultiTableOutputFormat ?
> >
> >
> > Job class:
> > public int run(String[] args) throws Exception {
> > System.out.println( " Running with on tables "+args[1]+ " and
> "+args[2]+"
> > with zk "+args[3]);
> > Configuration hbaseConf = HBaseConfiguration.create(getConf());
> > // hbaseConf.set(Constants.HBASE_ZOOKEEPER_QUORUM_PROP,
> > Constants.HBASE_OS_CL1_QUORUM);
> > hbaseConf.set(Constants.HBASE_ZOOKEEPER_QUORUM_PROP, args[3]);
> > Job job = new Job(hbaseConf);
> > job.setJarByClass(MultiTableTestJob.class);
> > job.setInputFormatClass(TextInputFormat.class);
> > job.setMapperClass(MultiTableTestMapper.class);
> > job.setMapOutputKeyClass(Text.class);
> > job.setMapOutputValueClass(Text.class);
> > job.setReducerClass(MultiTableTestReducer.class);
> > job.setOutputKeyClass(Text.class);
> > job.setOutputValueClass(Text.class);
> > FileInputFormat.setInputPaths(job, new Path(args[0]));
> > job.setOutputFormatClass(MultiTableOutputFormat.class);
> > TableMapReduceUtil.addDependencyJars(job);
> > TableMapReduceUtil.addDependencyJars(job.getConfiguration());
> > return job.waitForCompletion(true) == true ? 0 : -1;
> > }
> > public static void main(String[] args) throws Exception{
> > Configuration configuration = new Configuration();
> > configuration.set("HBASE_DEST_TABLE", args[1]);
> > configuration.set("HBASE_LOOKUP_TABLE", args[2]);
> > ToolRunner.run(configuration, new CISuperSessionJob(), args);
> > }
> >
> > Reducer Class:
> >
> > private ImmutableBytesWritable tbl1;
> > private ImmutableBytesWritable tbl2;
> >
> > protected void setup(Context context) throws IOException
> > ,InterruptedException {
> > Configuration c = context.getConfiguration();
> > tbl1 = new
> >
> >
> ImmutableBytesWritable(Bytes.toBytes(context.getConfiguration().get("HBASE_DEST_TABLE")));
> > tbl2 = new
> >
> >
> ImmutableBytesWritable(Bytes.toBytes(context.getConfiguration().get("HBASE_LOOKUP_TABLE")));
> > };
> >
> > protected void reduce(Text key, java.lang.Iterable<Text>
> > values, Context context) throws IOException ,InterruptedException {
> > //
> > if (some condition) {
> > Put put = getSessionPut(key, vc);
> > if (put != null) {
> > context.write(tbl1, put);
> > }
> > } else {
> > //
> > Put put = getEventPut(key, vc);
> > context.write(tbl2, put);
> > }
> > }
> > }
> >
> >
> > Exception:
> > org.apache.hadoop.hbase.TableNotFoundException: mapred.reduce.tasks=100
> > at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:999)
> > at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:864)
> > at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
> > at
> > org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
> > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
> > at
> >
> >
> org.apache.hadoop.hbase.mapreduce.MultiTableOutputFormat$MultiTableRecordWriter.getTable(MultiTableOutputFormat.java:101)
> > at
> >
> >
> org.apache.hadoop.hbase.mapreduce.MultiTableOutputFormat$MultiTableRecordWriter.write(MultiTableOutputFormat.java:127)
> > at
> >
> >
> org.apache.hadoop.hbase.mapreduce.MultiTableOutputFormat$MultiTableRecordWriter.write(MultiTableOutputFormat.java:68)
> > at
> >
> >
> org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:586)
> > at
> >
> >
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> > at
> >
> >
> >
> > Thanks & Regards,
> > B Anil Kumar.
> >
>
Re: Facing problem while using MultiTableOutputFormat
Posted by Ted Yu <yu...@gmail.com>.
System.out.println( " Running with on tables "+args[1]+ " and "+args[2]+"
with zk "+args[3]);
What was the output from the above ?
I would expect a call similar to the following in your run() method - this
comes from TestTableMapReduce.java:
TableMapReduceUtil.initTableReducerJob(
Bytes.toString(table.getTableName()),
IdentityTableReducer.class, job);
On Mon, Jan 6, 2014 at 7:12 PM, AnilKumar B <ak...@gmail.com> wrote:
> Hi,
>
> In my MR job, I need to write output into multiple tables, So I am
> using MultiTableOutputFormat as below. But I am getting
> TableNotFoundException.
>
> I am attaching code snippet below, Is this the correct way to use
> MultiTableOutputFormat ?
>
>
> Job class:
> public int run(String[] args) throws Exception {
> System.out.println( " Running with on tables "+args[1]+ " and "+args[2]+"
> with zk "+args[3]);
> Configuration hbaseConf = HBaseConfiguration.create(getConf());
> // hbaseConf.set(Constants.HBASE_ZOOKEEPER_QUORUM_PROP,
> Constants.HBASE_OS_CL1_QUORUM);
> hbaseConf.set(Constants.HBASE_ZOOKEEPER_QUORUM_PROP, args[3]);
> Job job = new Job(hbaseConf);
> job.setJarByClass(MultiTableTestJob.class);
> job.setInputFormatClass(TextInputFormat.class);
> job.setMapperClass(MultiTableTestMapper.class);
> job.setMapOutputKeyClass(Text.class);
> job.setMapOutputValueClass(Text.class);
> job.setReducerClass(MultiTableTestReducer.class);
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(Text.class);
> FileInputFormat.setInputPaths(job, new Path(args[0]));
> job.setOutputFormatClass(MultiTableOutputFormat.class);
> TableMapReduceUtil.addDependencyJars(job);
> TableMapReduceUtil.addDependencyJars(job.getConfiguration());
> return job.waitForCompletion(true) == true ? 0 : -1;
> }
> public static void main(String[] args) throws Exception{
> Configuration configuration = new Configuration();
> configuration.set("HBASE_DEST_TABLE", args[1]);
> configuration.set("HBASE_LOOKUP_TABLE", args[2]);
> ToolRunner.run(configuration, new CISuperSessionJob(), args);
> }
>
> Reducer Class:
>
> private ImmutableBytesWritable tbl1;
> private ImmutableBytesWritable tbl2;
>
> protected void setup(Context context) throws IOException
> ,InterruptedException {
> Configuration c = context.getConfiguration();
> tbl1 = new
>
> ImmutableBytesWritable(Bytes.toBytes(context.getConfiguration().get("HBASE_DEST_TABLE")));
> tbl2 = new
>
> ImmutableBytesWritable(Bytes.toBytes(context.getConfiguration().get("HBASE_LOOKUP_TABLE")));
> };
>
> protected void reduce(Text key, java.lang.Iterable<Text>
> values, Context context) throws IOException ,InterruptedException {
> //
> if (some condition) {
> Put put = getSessionPut(key, vc);
> if (put != null) {
> context.write(tbl1, put);
> }
> } else {
> //
> Put put = getEventPut(key, vc);
> context.write(tbl2, put);
> }
> }
> }
>
>
> Exception:
> org.apache.hadoop.hbase.TableNotFoundException: mapred.reduce.tasks=100
> at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:999)
> at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:864)
> at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
> at
> org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
> at
>
> org.apache.hadoop.hbase.mapreduce.MultiTableOutputFormat$MultiTableRecordWriter.getTable(MultiTableOutputFormat.java:101)
> at
>
> org.apache.hadoop.hbase.mapreduce.MultiTableOutputFormat$MultiTableRecordWriter.write(MultiTableOutputFormat.java:127)
> at
>
> org.apache.hadoop.hbase.mapreduce.MultiTableOutputFormat$MultiTableRecordWriter.write(MultiTableOutputFormat.java:68)
> at
>
> org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:586)
> at
>
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> at
>
>
>
> Thanks & Regards,
> B Anil Kumar.
>