You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Stuart Scott <St...@e-mis.com> on 2011/03/30 17:34:48 UTC
Changing Zookeeper address programmatically for reduces
Hi,
I have a map/reduce job that reads from a Hbase Table and writes to
another.
Does anyone know how to programmatically set the Zoo Keeper address for
a Reducer?
I can create a job as below, set the IP address using .set... it works
fine for the Maper. The reducer defaults to localhost. We have a couple
of clusters and wanted to be able to connect to them at different times.
HBaseConfiguration conf = new HBaseConfiguration();
conf.set("hbase.master", "44.128.161.86:60000");
conf.set("hbase.zookeeper.quorum","44.128.161.86");
...
Job job = new Job(conf, "MyJob_Job");
job.setJarByClass(MyClass.class);
Scan scan = new Scan();
String columns = "A:B";
scan.addColumns(columns);
scan.setStartRow(Bytes.toBytes(GetPaddedLongKey(1)));
scan.setStopRow(Bytes.toBytes(GetPaddedLongKey(100+1)));
TableMapReduceUtil.initTableMapperJob("MPI", scan,
Mapper2.class, ImmutableBytesWritable.class, IntWritable.class, job);
TableMapReduceUtil.initTableReducerJob("MyClass",
Reducer1.class, job);
job.waitForCompletion(true);
Runs the mapper without problem, then fails..
11/03/30 16:19:50 INFO mapred.LocalJobRunner:
11/03/30 16:19:50 INFO zookeeper.ZooKeeperWrapper: Reconnecting to
zookeeper
11/03/30 16:19:50 INFO zookeeper.ZooKeeper: Initiating client
connection, connectString=localhost:21810 sessionTimeout=60000
watcher=org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper@ae97c4
11/03/30 16:19:50 INFO zookeeper.ClientCnxn: Opening socket connection
to server localhost/127.0.0.1:21810
Closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)
2011-03-30 16:30:14,637 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server localhost/127.0.0.1:21810
2011-03-30 16:30:14,638 WARN org.apache.zookeeper.ClientCnxn: Session
0x0 for server null, unexpected error, closing socket connection and
attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
Any help would be really appreciated.
Stuart
Re: Changing Zookeeper address programmatically for reduces
Posted by Jean-Daniel Cryans <jd...@apache.org>.
So you tried to write to another cluster instead? Because it says you
didn't specify the other cluster correctly, CopyTable contains a help
that describes how that value should be constructed.
J-D
On Thu, Mar 31, 2011 at 2:23 AM, Stuart Scott <St...@e-mis.com> wrote:
> Hi J-D,
>
> Thanks for the info.
> I tried this but ended up with the following error. Any ideas?
>
> Exception in thread "main" java.io.IOException: Please specify the peer cluster as hbase.zookeeper.quorum:zookeeper.znode.parent
> at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableReducerJob(TableMapReduceUtil.java:172)
>
> Regards
>
> Stuart
>
> -----Original Message-----
> From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-Daniel Cryans
> Sent: 30 March 2011 17:34
> To: user@hbase.apache.org
> Subject: Re: Changing Zookeeper address programmatically for reduces
>
> That's basically what CopyTable does if I understand your need properly:
>
> https://github.com/apache/hbase/blob/0.90/src/main/java/org/apache/hadoop/hbase/mapreduce/CopyTable.java
>
> J-D
>
> On Wed, Mar 30, 2011 at 8:34 AM, Stuart Scott <St...@e-mis.com> wrote:
>> Hi,
>>
>>
>>
>> I have a map/reduce job that reads from a Hbase Table and writes to
>> another.
>>
>> Does anyone know how to programmatically set the Zoo Keeper address
>> for a Reducer?
>>
>> I can create a job as below, set the IP address using .set... it works
>> fine for the Maper. The reducer defaults to localhost. We have a
>> couple of clusters and wanted to be able to connect to them at different times.
>>
>>
>>
>> HBaseConfiguration conf = new HBaseConfiguration();
>>
>> conf.set("hbase.master", "44.128.161.86:60000");
>>
>> conf.set("hbase.zookeeper.quorum","44.128.161.86");
>>
>>
>>
>> ...
>>
>>
>>
>> Job job = new Job(conf, "MyJob_Job");
>>
>> job.setJarByClass(MyClass.class);
>>
>> Scan scan = new Scan();
>>
>> String columns = "A:B";
>>
>>
>> scan.addColumns(columns);
>>
>> scan.setStartRow(Bytes.toBytes(GetPaddedLongKey(1)));
>>
>> scan.setStopRow(Bytes.toBytes(GetPaddedLongKey(100+1)));
>>
>> TableMapReduceUtil.initTableMapperJob("MPI", scan,
>> Mapper2.class, ImmutableBytesWritable.class, IntWritable.class, job);
>>
>>
>>
>> TableMapReduceUtil.initTableReducerJob("MyClass",
>> Reducer1.class, job);
>>
>>
>>
>> job.waitForCompletion(true);
>>
>>
>>
>> Runs the mapper without problem, then fails..
>>
>>
>>
>> 11/03/30 16:19:50 INFO mapred.LocalJobRunner:
>>
>> 11/03/30 16:19:50 INFO zookeeper.ZooKeeperWrapper: Reconnecting to
>> zookeeper
>>
>> 11/03/30 16:19:50 INFO zookeeper.ZooKeeper: Initiating client
>> connection, connectString=localhost:21810 sessionTimeout=60000
>> watcher=org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper@ae97c4
>>
>> 11/03/30 16:19:50 INFO zookeeper.ClientCnxn: Opening socket connection
>> to server localhost/127.0.0.1:21810
>>
>>
>>
>> Closing socket connection and attempting reconnect
>>
>> java.net.ConnectException: Connection refused
>>
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>
>> at
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>>
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)
>>
>> 2011-03-30 16:30:14,637 INFO org.apache.zookeeper.ClientCnxn: Opening
>> socket connection to server localhost/127.0.0.1:21810
>>
>> 2011-03-30 16:30:14,638 WARN org.apache.zookeeper.ClientCnxn: Session
>> 0x0 for server null, unexpected error, closing socket connection and
>> attempting reconnect
>>
>> java.net.ConnectException: Connection refused
>>
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Any help would be really appreciated.
>>
>>
>>
>> Stuart
>>
>>
>>
>>
>>
>>
>
RE: Changing Zookeeper address programmatically for reduces
Posted by Stuart Scott <St...@e-mis.com>.
Hi J-D,
Thanks for the info.
I tried this but ended up with the following error. Any ideas?
Exception in thread "main" java.io.IOException: Please specify the peer cluster as hbase.zookeeper.quorum:zookeeper.znode.parent
at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableReducerJob(TableMapReduceUtil.java:172)
Regards
Stuart
-----Original Message-----
From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-Daniel Cryans
Sent: 30 March 2011 17:34
To: user@hbase.apache.org
Subject: Re: Changing Zookeeper address programmatically for reduces
That's basically what CopyTable does if I understand your need properly:
https://github.com/apache/hbase/blob/0.90/src/main/java/org/apache/hadoop/hbase/mapreduce/CopyTable.java
J-D
On Wed, Mar 30, 2011 at 8:34 AM, Stuart Scott <St...@e-mis.com> wrote:
> Hi,
>
>
>
> I have a map/reduce job that reads from a Hbase Table and writes to
> another.
>
> Does anyone know how to programmatically set the Zoo Keeper address
> for a Reducer?
>
> I can create a job as below, set the IP address using .set... it works
> fine for the Maper. The reducer defaults to localhost. We have a
> couple of clusters and wanted to be able to connect to them at different times.
>
>
>
> HBaseConfiguration conf = new HBaseConfiguration();
>
> conf.set("hbase.master", "44.128.161.86:60000");
>
> conf.set("hbase.zookeeper.quorum","44.128.161.86");
>
>
>
> ...
>
>
>
> Job job = new Job(conf, "MyJob_Job");
>
> job.setJarByClass(MyClass.class);
>
> Scan scan = new Scan();
>
> String columns = "A:B";
>
>
> scan.addColumns(columns);
>
> scan.setStartRow(Bytes.toBytes(GetPaddedLongKey(1)));
>
> scan.setStopRow(Bytes.toBytes(GetPaddedLongKey(100+1)));
>
> TableMapReduceUtil.initTableMapperJob("MPI", scan,
> Mapper2.class, ImmutableBytesWritable.class, IntWritable.class, job);
>
>
>
> TableMapReduceUtil.initTableReducerJob("MyClass",
> Reducer1.class, job);
>
>
>
> job.waitForCompletion(true);
>
>
>
> Runs the mapper without problem, then fails..
>
>
>
> 11/03/30 16:19:50 INFO mapred.LocalJobRunner:
>
> 11/03/30 16:19:50 INFO zookeeper.ZooKeeperWrapper: Reconnecting to
> zookeeper
>
> 11/03/30 16:19:50 INFO zookeeper.ZooKeeper: Initiating client
> connection, connectString=localhost:21810 sessionTimeout=60000
> watcher=org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper@ae97c4
>
> 11/03/30 16:19:50 INFO zookeeper.ClientCnxn: Opening socket connection
> to server localhost/127.0.0.1:21810
>
>
>
> Closing socket connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)
>
> 2011-03-30 16:30:14,637 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server localhost/127.0.0.1:21810
>
> 2011-03-30 16:30:14,638 WARN org.apache.zookeeper.ClientCnxn: Session
> 0x0 for server null, unexpected error, closing socket connection and
> attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
>
>
>
>
>
>
>
>
> Any help would be really appreciated.
>
>
>
> Stuart
>
>
>
>
>
>
Re: Changing Zookeeper address programmatically for reduces
Posted by Jean-Daniel Cryans <jd...@apache.org>.
That's basically what CopyTable does if I understand your need properly:
https://github.com/apache/hbase/blob/0.90/src/main/java/org/apache/hadoop/hbase/mapreduce/CopyTable.java
J-D
On Wed, Mar 30, 2011 at 8:34 AM, Stuart Scott <St...@e-mis.com> wrote:
> Hi,
>
>
>
> I have a map/reduce job that reads from a Hbase Table and writes to
> another.
>
> Does anyone know how to programmatically set the Zoo Keeper address for
> a Reducer?
>
> I can create a job as below, set the IP address using .set... it works
> fine for the Maper. The reducer defaults to localhost. We have a couple
> of clusters and wanted to be able to connect to them at different times.
>
>
>
> HBaseConfiguration conf = new HBaseConfiguration();
>
> conf.set("hbase.master", "44.128.161.86:60000");
>
> conf.set("hbase.zookeeper.quorum","44.128.161.86");
>
>
>
> ...
>
>
>
> Job job = new Job(conf, "MyJob_Job");
>
> job.setJarByClass(MyClass.class);
>
> Scan scan = new Scan();
>
> String columns = "A:B";
>
>
> scan.addColumns(columns);
>
> scan.setStartRow(Bytes.toBytes(GetPaddedLongKey(1)));
>
> scan.setStopRow(Bytes.toBytes(GetPaddedLongKey(100+1)));
>
> TableMapReduceUtil.initTableMapperJob("MPI", scan,
> Mapper2.class, ImmutableBytesWritable.class, IntWritable.class, job);
>
>
>
> TableMapReduceUtil.initTableReducerJob("MyClass",
> Reducer1.class, job);
>
>
>
> job.waitForCompletion(true);
>
>
>
> Runs the mapper without problem, then fails..
>
>
>
> 11/03/30 16:19:50 INFO mapred.LocalJobRunner:
>
> 11/03/30 16:19:50 INFO zookeeper.ZooKeeperWrapper: Reconnecting to
> zookeeper
>
> 11/03/30 16:19:50 INFO zookeeper.ZooKeeper: Initiating client
> connection, connectString=localhost:21810 sessionTimeout=60000
> watcher=org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper@ae97c4
>
> 11/03/30 16:19:50 INFO zookeeper.ClientCnxn: Opening socket connection
> to server localhost/127.0.0.1:21810
>
>
>
> Closing socket connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)
>
> 2011-03-30 16:30:14,637 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server localhost/127.0.0.1:21810
>
> 2011-03-30 16:30:14,638 WARN org.apache.zookeeper.ClientCnxn: Session
> 0x0 for server null, unexpected error, closing socket connection and
> attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
>
>
>
>
>
>
>
>
> Any help would be really appreciated.
>
>
>
> Stuart
>
>
>
>
>
>
Re: Changing Zookeeper address programmatically for reduces
Posted by Stack <st...@duboce.net>.
Hey Stuart:
Would the Mapper and Reducer talk to different clusters?
Do you need to go to a reduce? Can you write to the output from the
mapper? (See http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/package-summary.html#sink).
Can you use http://hadoop.apache.org/mapreduce/docs/r0.21.0/api/org/apache/hadoop/mapreduce/Reducer.html#setup(org.apache.hadoop.mapreduce.Reducer.Context)
in TableReducer to set output?
You want to do this programmatically rather than via configuration
files, adding hbase-site.xml to your job?
St.Ack
On Wed, Mar 30, 2011 at 8:34 AM, Stuart Scott <St...@e-mis.com> wrote:
> Hi,
>
>
>
> I have a map/reduce job that reads from a Hbase Table and writes to
> another.
>
> Does anyone know how to programmatically set the Zoo Keeper address for
> a Reducer?
>
> I can create a job as below, set the IP address using .set... it works
> fine for the Maper. The reducer defaults to localhost. We have a couple
> of clusters and wanted to be able to connect to them at different times.
>
>
>
> HBaseConfiguration conf = new HBaseConfiguration();
>
> conf.set("hbase.master", "44.128.161.86:60000");
>
> conf.set("hbase.zookeeper.quorum","44.128.161.86");
>
>
>
> ...
>
>
>
> Job job = new Job(conf, "MyJob_Job");
>
> job.setJarByClass(MyClass.class);
>
> Scan scan = new Scan();
>
> String columns = "A:B";
>
>
> scan.addColumns(columns);
>
> scan.setStartRow(Bytes.toBytes(GetPaddedLongKey(1)));
>
> scan.setStopRow(Bytes.toBytes(GetPaddedLongKey(100+1)));
>
> TableMapReduceUtil.initTableMapperJob("MPI", scan,
> Mapper2.class, ImmutableBytesWritable.class, IntWritable.class, job);
>
>
>
> TableMapReduceUtil.initTableReducerJob("MyClass",
> Reducer1.class, job);
>
>
>
> job.waitForCompletion(true);
>
>
>
> Runs the mapper without problem, then fails..
>
>
>
> 11/03/30 16:19:50 INFO mapred.LocalJobRunner:
>
> 11/03/30 16:19:50 INFO zookeeper.ZooKeeperWrapper: Reconnecting to
> zookeeper
>
> 11/03/30 16:19:50 INFO zookeeper.ZooKeeper: Initiating client
> connection, connectString=localhost:21810 sessionTimeout=60000
> watcher=org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper@ae97c4
>
> 11/03/30 16:19:50 INFO zookeeper.ClientCnxn: Opening socket connection
> to server localhost/127.0.0.1:21810
>
>
>
> Closing socket connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)
>
> 2011-03-30 16:30:14,637 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server localhost/127.0.0.1:21810
>
> 2011-03-30 16:30:14,638 WARN org.apache.zookeeper.ClientCnxn: Session
> 0x0 for server null, unexpected error, closing socket connection and
> attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
>
>
>
>
>
>
>
>
> Any help would be really appreciated.
>
>
>
> Stuart
>
>
>
>
>
>