You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by "Liu, Ming (HPIT-GADSC)" <mi...@hp.com> on 2014/11/24 09:33:17 UTC
What is proper way to make a hbase connection? using HTable
(conf,tbl) or createConnection? Zookeeper session run out.
Hello,
I am using HBase 0.98.5. In example hbase client programs, some use createConnection() and some use HTable() directly, I found they behave different, I wrote two simple tests program using these two different methods, each program will start two threads and do simple put. and I found:
One program will start only 1 zookeeper sessions shared in two threads while another will start 2 zookeeper sessions. So I don't know why the program using createConnection will do more zookeeper requests than simply use HTable. Is it possible to use createConnection in two threads but share the same zookeeper session?
Here is details:
Demo1 will make two zookeeper sessions, seems two HBase connections; but Demo2 will only make one zookeeper session. My real program is using createConnection in multiple threads as in demo1, since I have a very small zookeeper , it only allows 60 concurrent sessions, so my program always fail when there are hundreds of threads started. But I saw if using HTable directly, it will only consume 1 zookeeper session. But it will change a lot in my current program, so I wish there is a way to use createConnection and behave same as using HTable, is it possible?
Source code:
Demo1.java
class ClientThread extends Thread
{
public static Configuration configuration;
static {
configuration = HBaseConfiguration.create();
}
public void run()
{
try {
System.out.println("start insert data ......");
HConnection con=HConnectionManager.createConnection(configuration);
HTable table = (HTable)con.getTable("hbase_table1");
Put put = new Put("1".getBytes());
put.add("c1".getBytes(), null, "baidu".getBytes());
put.add("c2".getBytes(), null, "http://www.baidu.com1".getBytes());
try {
table.put(put);
} catch (IOException e) {
e.printStackTrace();
}
System.out.println("end insert data ......");
}
catch (Exception e) {
}
}
}
public class demo1 {
public static void main(String[] args) throws Exception {
Thread t1=new ClientThread();
Thread t2=new ClientThread();
t1.start();
t2.start();
}
}
Demo2.java
class ClientThread1 extends Thread
{
public static Configuration configuration;
static {
configuration = HBaseConfiguration.create();
}
public void run()
{
System.out.println("start insert data ......");
try {
HTableInterface table = new HTable(configuration, "hbase_table1");
Put put = new Put("1".getBytes());
put.add("c1".getBytes(), null, "baidu".getBytes());
put.add("c2".getBytes(), null, "http://www.baidu.com1".getBytes());
table.put(put);
} catch (Exception e) {
e.printStackTrace();
}
System.out.println("end insert data ......");
}
}
public class demo2 {
public static void main(String[] args) throws Exception {
Thread t1=new ClientThread1();
Thread t2=new ClientThread1();
t1.start();
t2.start();
}
}
This should be a very basic question, sorry, I really did some search but cannot find any good explaination. Any help will be very appreciated.
Thanks,
Ming
Re: What is proper way to make a hbase connection? using HTable
(conf,tbl) or createConnection? Zookeeper session run out.
Posted by Bharath Vissapragada <bh...@cloudera.com>.
Yes thats correct. HTable is not thread-safe.
On Mon, Nov 24, 2014 at 2:55 PM, Liu, Ming (HPIT-GADSC) <mi...@hp.com>
wrote:
> Thank you Bharath,
>
> This is a very helpful reply! I will share the connection between two
> threads. Simply put, HTable is not safe for multi-thread, is this true? In
> multi-threads, one must use HConnectionManager.
>
> Thanks,
> Ming
> -----Original Message-----
> From: Bharath Vissapragada [mailto:bharathv@cloudera.com]
> Sent: Monday, November 24, 2014 4:52 PM
> To: hbase-user
> Subject: Re: What is proper way to make a hbase connection? using HTable
> (conf,tbl) or createConnection? Zookeeper session run out.
>
> Hi Ming,
>
> HConnection connection = HConnectionManager.createConnection(conf);
> HTableInterface table = connection.getTable("mytable"); table.get(...); /
> table.put(...);
>
> Is the correct way to use. However
> HConnectionManager.createConnection(conf) gives you a "shared"
> HConnection which you can reuse across multiple threads that get their own
> conn.getTable() and do their puts/gets.
>
> Thanks,
> Bharath
>
> On Mon, Nov 24, 2014 at 2:03 PM, Liu, Ming (HPIT-GADSC) <mi...@hp.com>
> wrote:
>
> > Hello,
> >
> > I am using HBase 0.98.5. In example hbase client programs, some use
> > createConnection() and some use HTable() directly, I found they behave
> > different, I wrote two simple tests program using these two different
> > methods, each program will start two threads and do simple put. and I
> > found:
> > One program will start only 1 zookeeper sessions shared in two threads
> > while another will start 2 zookeeper sessions. So I don't know why the
> > program using createConnection will do more zookeeper requests than
> > simply use HTable. Is it possible to use createConnection in two
> > threads but share the same zookeeper session?
> >
> > Here is details:
> > Demo1 will make two zookeeper sessions, seems two HBase connections;
> > but
> > Demo2 will only make one zookeeper session. My real program is using
> > createConnection in multiple threads as in demo1, since I have a very
> > small zookeeper , it only allows 60 concurrent sessions, so my program
> > always fail when there are hundreds of threads started. But I saw if
> > using HTable directly, it will only consume 1 zookeeper session. But
> > it will change a lot in my current program, so I wish there is a way
> > to use createConnection and behave same as using HTable, is it possible?
> >
> > Source code:
> >
> > Demo1.java
> > class ClientThread extends Thread
> > {
> > public static Configuration configuration;
> > static {
> > configuration = HBaseConfiguration.create();
> > }
> > public void run()
> > {
> > try {
> > System.out.println("start insert data ......");
> > HConnection
> > con=HConnectionManager.createConnection(configuration);
> > HTable table = (HTable)con.getTable("hbase_table1");
> > Put put = new Put("1".getBytes());
> > put.add("c1".getBytes(), null, "baidu".getBytes());
> > put.add("c2".getBytes(), null, "http://www.baidu.com1
> > ".getBytes());
> > try {
> > table.put(put);
> > } catch (IOException e) {
> > e.printStackTrace();
> > }
> > System.out.println("end insert data ......");
> > }
> > catch (Exception e) {
> > }
> >
> > }
> > }
> > public class demo1 {
> >
> > public static void main(String[] args) throws Exception {
> > Thread t1=new ClientThread();
> > Thread t2=new ClientThread();
> > t1.start();
> > t2.start();
> > }
> >
> > }
> >
> >
> > Demo2.java
> > class ClientThread1 extends Thread
> > {
> > public static Configuration configuration;
> > static {
> > configuration = HBaseConfiguration.create();
> > }
> > public void run()
> > {
> > System.out.println("start insert data ......");
> > try {
> > HTableInterface table = new
> > HTable(configuration, "hbase_table1");
> > Put put = new Put("1".getBytes());
> > put.add("c1".getBytes(), null,
> "baidu".getBytes());
> > put.add("c2".getBytes(), null, "
> > http://www.baidu.com1".getBytes());
> > table.put(put);
> > } catch (Exception e) {
> > e.printStackTrace();
> > }
> > System.out.println("end insert data ......");
> >
> > }
> >
> >
> > }
> > public class demo2 {
> >
> > public static void main(String[] args) throws Exception {
> > Thread t1=new ClientThread1();
> > Thread t2=new ClientThread1();
> > t1.start();
> > t2.start();
> > }
> > }
> >
> >
> >
> > This should be a very basic question, sorry, I really did some search
> > but cannot find any good explaination. Any help will be very appreciated.
> >
> > Thanks,
> > Ming
> >
>
>
>
> --
> Bharath Vissapragada
> <http://www.cloudera.com>
>
--
Bharath Vissapragada
<http://www.cloudera.com>
RE: What is proper way to make a hbase connection? using HTable
(conf,tbl) or createConnection? Zookeeper session run out.
Posted by "Liu, Ming (HPIT-GADSC)" <mi...@hp.com>.
Thank you Bharath,
This is a very helpful reply! I will share the connection between two threads. Simply put, HTable is not safe for multi-thread, is this true? In multi-threads, one must use HConnectionManager.
Thanks,
Ming
-----Original Message-----
From: Bharath Vissapragada [mailto:bharathv@cloudera.com]
Sent: Monday, November 24, 2014 4:52 PM
To: hbase-user
Subject: Re: What is proper way to make a hbase connection? using HTable (conf,tbl) or createConnection? Zookeeper session run out.
Hi Ming,
HConnection connection = HConnectionManager.createConnection(conf);
HTableInterface table = connection.getTable("mytable"); table.get(...); / table.put(...);
Is the correct way to use. However
HConnectionManager.createConnection(conf) gives you a "shared" HConnection which you can reuse across multiple threads that get their own
conn.getTable() and do their puts/gets.
Thanks,
Bharath
On Mon, Nov 24, 2014 at 2:03 PM, Liu, Ming (HPIT-GADSC) <mi...@hp.com>
wrote:
> Hello,
>
> I am using HBase 0.98.5. In example hbase client programs, some use
> createConnection() and some use HTable() directly, I found they behave
> different, I wrote two simple tests program using these two different
> methods, each program will start two threads and do simple put. and I
> found:
> One program will start only 1 zookeeper sessions shared in two threads
> while another will start 2 zookeeper sessions. So I don't know why the
> program using createConnection will do more zookeeper requests than
> simply use HTable. Is it possible to use createConnection in two
> threads but share the same zookeeper session?
>
> Here is details:
> Demo1 will make two zookeeper sessions, seems two HBase connections;
> but
> Demo2 will only make one zookeeper session. My real program is using
> createConnection in multiple threads as in demo1, since I have a very
> small zookeeper , it only allows 60 concurrent sessions, so my program
> always fail when there are hundreds of threads started. But I saw if
> using HTable directly, it will only consume 1 zookeeper session. But
> it will change a lot in my current program, so I wish there is a way
> to use createConnection and behave same as using HTable, is it possible?
>
> Source code:
>
> Demo1.java
> class ClientThread extends Thread
> {
> public static Configuration configuration;
> static {
> configuration = HBaseConfiguration.create();
> }
> public void run()
> {
> try {
> System.out.println("start insert data ......");
> HConnection
> con=HConnectionManager.createConnection(configuration);
> HTable table = (HTable)con.getTable("hbase_table1");
> Put put = new Put("1".getBytes());
> put.add("c1".getBytes(), null, "baidu".getBytes());
> put.add("c2".getBytes(), null, "http://www.baidu.com1
> ".getBytes());
> try {
> table.put(put);
> } catch (IOException e) {
> e.printStackTrace();
> }
> System.out.println("end insert data ......");
> }
> catch (Exception e) {
> }
>
> }
> }
> public class demo1 {
>
> public static void main(String[] args) throws Exception {
> Thread t1=new ClientThread();
> Thread t2=new ClientThread();
> t1.start();
> t2.start();
> }
>
> }
>
>
> Demo2.java
> class ClientThread1 extends Thread
> {
> public static Configuration configuration;
> static {
> configuration = HBaseConfiguration.create();
> }
> public void run()
> {
> System.out.println("start insert data ......");
> try {
> HTableInterface table = new
> HTable(configuration, "hbase_table1");
> Put put = new Put("1".getBytes());
> put.add("c1".getBytes(), null, "baidu".getBytes());
> put.add("c2".getBytes(), null, "
> http://www.baidu.com1".getBytes());
> table.put(put);
> } catch (Exception e) {
> e.printStackTrace();
> }
> System.out.println("end insert data ......");
>
> }
>
>
> }
> public class demo2 {
>
> public static void main(String[] args) throws Exception {
> Thread t1=new ClientThread1();
> Thread t2=new ClientThread1();
> t1.start();
> t2.start();
> }
> }
>
>
>
> This should be a very basic question, sorry, I really did some search
> but cannot find any good explaination. Any help will be very appreciated.
>
> Thanks,
> Ming
>
--
Bharath Vissapragada
<http://www.cloudera.com>
Re: What is proper way to make a hbase connection? using HTable
(conf,tbl) or createConnection? Zookeeper session run out.
Posted by Bharath Vissapragada <bh...@cloudera.com>.
Hi Ming,
HConnection connection = HConnectionManager.createConnection(conf);
HTableInterface table = connection.getTable("mytable");
table.get(...); / table.put(...);
Is the correct way to use. However
HConnectionManager.createConnection(conf) gives you a "shared" HConnection
which you can reuse across multiple threads that get their own
conn.getTable() and do their puts/gets.
Thanks,
Bharath
On Mon, Nov 24, 2014 at 2:03 PM, Liu, Ming (HPIT-GADSC) <mi...@hp.com>
wrote:
> Hello,
>
> I am using HBase 0.98.5. In example hbase client programs, some use
> createConnection() and some use HTable() directly, I found they behave
> different, I wrote two simple tests program using these two different
> methods, each program will start two threads and do simple put. and I
> found:
> One program will start only 1 zookeeper sessions shared in two threads
> while another will start 2 zookeeper sessions. So I don't know why the
> program using createConnection will do more zookeeper requests than simply
> use HTable. Is it possible to use createConnection in two threads but share
> the same zookeeper session?
>
> Here is details:
> Demo1 will make two zookeeper sessions, seems two HBase connections; but
> Demo2 will only make one zookeeper session. My real program is using
> createConnection in multiple threads as in demo1, since I have a very small
> zookeeper , it only allows 60 concurrent sessions, so my program always
> fail when there are hundreds of threads started. But I saw if using HTable
> directly, it will only consume 1 zookeeper session. But it will change a
> lot in my current program, so I wish there is a way to use createConnection
> and behave same as using HTable, is it possible?
>
> Source code:
>
> Demo1.java
> class ClientThread extends Thread
> {
> public static Configuration configuration;
> static {
> configuration = HBaseConfiguration.create();
> }
> public void run()
> {
> try {
> System.out.println("start insert data ......");
> HConnection
> con=HConnectionManager.createConnection(configuration);
> HTable table = (HTable)con.getTable("hbase_table1");
> Put put = new Put("1".getBytes());
> put.add("c1".getBytes(), null, "baidu".getBytes());
> put.add("c2".getBytes(), null, "http://www.baidu.com1
> ".getBytes());
> try {
> table.put(put);
> } catch (IOException e) {
> e.printStackTrace();
> }
> System.out.println("end insert data ......");
> }
> catch (Exception e) {
> }
>
> }
> }
> public class demo1 {
>
> public static void main(String[] args) throws Exception {
> Thread t1=new ClientThread();
> Thread t2=new ClientThread();
> t1.start();
> t2.start();
> }
>
> }
>
>
> Demo2.java
> class ClientThread1 extends Thread
> {
> public static Configuration configuration;
> static {
> configuration = HBaseConfiguration.create();
> }
> public void run()
> {
> System.out.println("start insert data ......");
> try {
> HTableInterface table = new HTable(configuration,
> "hbase_table1");
> Put put = new Put("1".getBytes());
> put.add("c1".getBytes(), null, "baidu".getBytes());
> put.add("c2".getBytes(), null, "
> http://www.baidu.com1".getBytes());
> table.put(put);
> } catch (Exception e) {
> e.printStackTrace();
> }
> System.out.println("end insert data ......");
>
> }
>
>
> }
> public class demo2 {
>
> public static void main(String[] args) throws Exception {
> Thread t1=new ClientThread1();
> Thread t2=new ClientThread1();
> t1.start();
> t2.start();
> }
> }
>
>
>
> This should be a very basic question, sorry, I really did some search but
> cannot find any good explaination. Any help will be very appreciated.
>
> Thanks,
> Ming
>
--
Bharath Vissapragada
<http://www.cloudera.com>