You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Andrzej <bo...@wp.pl> on 2017/09/18 17:42:18 UTC
How efficient add row with columns to table?
I simple-client.cc is:
for (uint64_t i = 0; i < num_puts; i++)
table->Put(*MakePut(Row(FLAGS_row, i)));
where
std::unique_ptr<Put> MakePut(const std::string &row) {
auto put = std::make_unique<Put>(row);
put->AddColumn("f", "q", row);
return std::move(put);
}
But there are two problems:
1. In above example column "q" is row_key, but if I want add other
column than row, I don't know on which row will be added.
2.How efficient add many columns?
In general case , how add :
row_key, column1, column2...column 10
Re: Is possible store BINARY data in row: string?
Posted by Esteban Gutierrez <es...@cloudera.com>.
Hello Andrzej,
HBase doesn't really care what encoding you use for the key and the value.
This issue is most likely happing on the C++ client or in your application.
If you can reproduce this issue systematically with the C++ client please
feel free to open an upstream JIRA for the HBase project:
https://issues.apache.org/jira/HBASE
Thanks,
esteban.
--
Cloudera, Inc.
On Wed, Sep 27, 2017 at 11:49 AM, Andrzej <bo...@wp.pl> wrote:
> I have big problem:
> I have row: address 36 bytes + hasz20 bytes (binary with zeros!) + 4 bytes
> index + 1 byte type.
> I write:
> 304d494e454478434f494e42415345303030303030303030303030303030
> 303030300000982051fd1e4ba744bbbe680e1fee14677ba1a3c3540bf7b1
> cdb606e857233e0e0000000000
>
> and I read:
> 304d494e454478434f494e42415345303030303030303030303030303030
> 303030300020203c636f310000000000000000000000000000004f494e42
> 41534530303030303030303030
>
> address=0MINEDxCOINBASE0000000000000000000 hash=0c00680000000000000000000
> 00000000159a670000000000000005109227972 xio=32767, type=3
>
>
> only address is correct!
> address end with zero byte. Strings with zeros are not correct stored?
> I store with:
>
> auto put = std::make_unique<Put>(row);
> for (int i = 0; i<qualifiers.size(); i++)
> put->AddColumn(families[i], qualifiers[i], values[i]);
> getTable(hTable)->Put(*std::move(put));
>
> and read with:
> return string: result->Row()
>
Re: Is possible store BINARY data in row: string?
Posted by Ted Yu <yu...@gmail.com>.
In simple-client, we have this code:
void ValidateResult(const Result &result, const std::string &row) {
CHECK(!result.IsEmpty());
CHECK_EQ(result.Row(), row);
Are you able to reproduce what you described by modifying simple-client
(which results in assertion failure) ?
On Wed, Sep 27, 2017 at 11:49 AM, Andrzej <bo...@wp.pl> wrote:
> I have big problem:
> I have row: address 36 bytes + hasz20 bytes (binary with zeros!) + 4 bytes
> index + 1 byte type.
> I write:
> 304d494e454478434f494e42415345303030303030303030303030303030
> 303030300000982051fd1e4ba744bbbe680e1fee14677ba1a3c3540bf7b1
> cdb606e857233e0e0000000000
>
> and I read:
> 304d494e454478434f494e42415345303030303030303030303030303030
> 303030300020203c636f310000000000000000000000000000004f494e42
> 41534530303030303030303030
>
> address=0MINEDxCOINBASE0000000000000000000 hash=0c00680000000000000000000
> 00000000159a670000000000000005109227972 xio=32767, type=3
>
>
> only address is correct!
> address end with zero byte. Strings with zeros are not correct stored?
> I store with:
>
> auto put = std::make_unique<Put>(row);
> for (int i = 0; i<qualifiers.size(); i++)
> put->AddColumn(families[i], qualifiers[i], values[i]);
> getTable(hTable)->Put(*std::move(put));
>
> and read with:
> return string: result->Row()
>
Is possible store BINARY data in row: string?
Posted by Andrzej <bo...@wp.pl>.
I have big problem:
I have row: address 36 bytes + hasz20 bytes (binary with zeros!) + 4
bytes index + 1 byte type.
I write:
304d494e454478434f494e42415345303030303030303030303030303030303030300000982051fd1e4ba744bbbe680e1fee14677ba1a3c3540bf7b1cdb606e857233e0e0000000000
and I read:
304d494e454478434f494e42415345303030303030303030303030303030303030300020203c636f310000000000000000000000000000004f494e4241534530303030303030303030
address=0MINEDxCOINBASE0000000000000000000
hash=0c0068000000000000000000000000000159a670000000000000005109227972
xio=32767, type=3
only address is correct!
address end with zero byte. Strings with zeros are not correct stored?
I store with:
auto put = std::make_unique<Put>(row);
for (int i = 0; i<qualifiers.size(); i++)
put->AddColumn(families[i], qualifiers[i], values[i]);
getTable(hTable)->Put(*std::move(put));
and read with:
return string: result->Row()
Re: Unable to connect http://localhost:16010/
Posted by Ted Yu <yu...@gmail.com>.
Java 9 support is not fully in place in hbase.
Suggest using Java 8 to unblock you.
On Sat, Sep 23, 2017 at 8:13 AM, Andrzej <bo...@wp.pl> wrote:
> W dniu 23.09.2017 o 17:00, Ted Yu pisze:
>
>> Please inspect hbase-root-master-andrzej-VirtualBox.out (h
>> base-root-master-andrzej-VirtualBox.log) to see if there was any fatal
>> error.
>>
>
> Problem can be because I use new 9-th version of Java?
> I use hadoop-2.8.1.tar.gz and hbase-1.3.1-bin.tar.gz
>
> 2017-09-23 17:11:21,750 ERROR [main] master.HMasterCommandLine: Master
> exiting
> java.lang.RuntimeException: Failed construction of Master: class
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMasterUnexpected
> version format: 9-internal
> at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThre
> ad(JVMClusterUtil.java:143)
> at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBa
> seCluster.java:220)
> at org.apache.hadoop.hbase.LocalHBaseCluster.<init>(LocalHBaseC
> luster.java:155)
> at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaste
> r(HMasterCommandLine.java:222)
> at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMaste
> rCommandLine.java:137)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(Server
> CommandLine.java:126)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2610)
> Caused by: java.lang.ExceptionInInitializerError
> at org.apache.hadoop.hbase.ipc.IPCUtil.<init>(IPCUtil.java:75)
> at org.apache.hadoop.hbase.ipc.RpcServer.<init>(RpcServer.java:
> 2185)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.<init>(
> RSRpcServices.java:994)
> at org.apache.hadoop.hbase.master.MasterRpcServices.<init>(
> MasterRpcServices.java:233)
> at org.apache.hadoop.hbase.master.HMaster.createRpcServices(
> HMaster.java:587)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(
> HRegionServer.java:543)
> at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:427)
> at org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMast
> er.<init>(HMasterCommandLine.java:312)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(java.
> base@9-internal/Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(java.
> base@9-internal/NativeConstructorAccessorImpl.java:62)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> java.base@9-internal/DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(java.base@9-intern
> al/Constructor.java:453)
> at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThre
> ad(JVMClusterUtil.java:139)
> ... 7 more
> Caused by: java.lang.RuntimeException: Unexpected version format:
> 9-internal
> at org.apache.hadoop.hbase.util.ClassSize.<clinit>(ClassSize.ja
> va:119)
> ... 20 more
>
>
Re: Unable to connect http://localhost:16010/
Posted by Andrzej <bo...@wp.pl>.
W dniu 23.09.2017 o 17:00, Ted Yu pisze:
> Please inspect hbase-root-master-andrzej-VirtualBox.out (h
> base-root-master-andrzej-VirtualBox.log) to see if there was any fatal
> error.
Problem can be because I use new 9-th version of Java?
I use hadoop-2.8.1.tar.gz and hbase-1.3.1-bin.tar.gz
2017-09-23 17:11:21,750 ERROR [main] master.HMasterCommandLine: Master
exiting
java.lang.RuntimeException: Failed construction of Master: class
org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMasterUnexpected
version format: 9-internal
at
org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:143)
at
org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:220)
at
org.apache.hadoop.hbase.LocalHBaseCluster.<init>(LocalHBaseCluster.java:155)
at
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:222)
at
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:137)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2610)
Caused by: java.lang.ExceptionInInitializerError
at org.apache.hadoop.hbase.ipc.IPCUtil.<init>(IPCUtil.java:75)
at org.apache.hadoop.hbase.ipc.RpcServer.<init>(RpcServer.java:2185)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.<init>(RSRpcServices.java:994)
at
org.apache.hadoop.hbase.master.MasterRpcServices.<init>(MasterRpcServices.java:233)
at
org.apache.hadoop.hbase.master.HMaster.createRpcServices(HMaster.java:587)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:543)
at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:427)
at
org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.<init>(HMasterCommandLine.java:312)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(java.base@9-internal/Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(java.base@9-internal/NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(java.base@9-internal/DelegatingConstructorAccessorImpl.java:45)
at
java.lang.reflect.Constructor.newInstance(java.base@9-internal/Constructor.java:453)
at
org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139)
... 7 more
Caused by: java.lang.RuntimeException: Unexpected version format: 9-internal
at org.apache.hadoop.hbase.util.ClassSize.<clinit>(ClassSize.java:119)
... 20 more
Re: Unable to connect http://localhost:16010/
Posted by Ted Yu <yu...@gmail.com>.
You can use 'ps' command to see if master / region server is running.
Please inspect hbase-root-master-andrzej-VirtualBox.out (h
base-root-master-andrzej-VirtualBox.log) to see if there was any fatal
error.
On Sat, Sep 23, 2017 at 7:49 AM, Andrzej <bo...@wp.pl> wrote:
> In new Linux I install Hadoop and http://localhost:50070/ is ok,
> http://localhost:8088/cluster is ok.
> Nect I try install HBase by https://www.tutorialspoint.com
> /hbase/hbase_installation.htm
>
> I do
> sudo /usr/local/HBase/bin/start-hbase.sh
> next
> I check http://localhost:16010 or 60010
> but not connected
>
> sudo /usr/local/HBase/bin/start-hbase.sh gitve me no errors:
> $ sudo /usr/local/HBase/bin/start-hbase.sh
> starting master, logging to /usr/local/HBase/bin/../logs/h
> base-root-master-andrzej-VirtualBox.out
>
Unable to connect http://localhost:16010/
Posted by Andrzej <bo...@wp.pl>.
In new Linux I install Hadoop and http://localhost:50070/ is ok,
http://localhost:8088/cluster is ok.
Nect I try install HBase by
https://www.tutorialspoint.com/hbase/hbase_installation.htm
I do
sudo /usr/local/HBase/bin/start-hbase.sh
next
I check http://localhost:16010 or 60010
but not connected
sudo /usr/local/HBase/bin/start-hbase.sh gitve me no errors:
$ sudo /usr/local/HBase/bin/start-hbase.sh
starting master, logging to
/usr/local/HBase/bin/../logs/hbase-root-master-andrzej-VirtualBox.out
Re: PrefixFilter and Scanner
Posted by Ted Yu <yu...@gmail.com>.
Please take a look at ./hbase-native-client/src/hbase/client/filter-test.cc
:
get_one.SetFilter(FilterFactory::ColumnPrefixFilter("foo_"));
get_two.SetFilter(FilterFactory::ColumnPrefixFilter("column_"));
FYI
On Wed, Sep 20, 2017 at 3:33 AM, Andrzej <bo...@wp.pl> wrote:
> PrefixFilter - takes a single argument, a prefix of a row key. It returns
> only those key-values present in a row that start with the specified row
> prefix
>
> Previously I thought, that if we have rows:
> Addr1Hash1
> Addr1Hash2
> Addr1Hash3
> Addr2Hash1
> Addr2Hash2
> Addr2Hash3
>
> If we call
> shared_ptr<Scan> scan = make_shared<Scan>("Addr2");
> shared_ptr<ResultScanner> scanner = table->Scan(*scan);
>
> that Scanner go to "Addr2Hash1"
> and if we apply make_shared<Scan>(""), Scanner got to first row of table.
> But is needed PrefixFilter? How it apply with Scanner?
>
PrefixFilter and Scanner
Posted by Andrzej <bo...@wp.pl>.
PrefixFilter - takes a single argument, a prefix of a row key. It
returns only those key-values present in a row that start with the
specified row prefix
Previously I thought, that if we have rows:
Addr1Hash1
Addr1Hash2
Addr1Hash3
Addr2Hash1
Addr2Hash2
Addr2Hash3
If we call
shared_ptr<Scan> scan = make_shared<Scan>("Addr2");
shared_ptr<ResultScanner> scanner = table->Scan(*scan);
that Scanner go to "Addr2Hash1"
and if we apply make_shared<Scan>(""), Scanner got to first row of table.
But is needed PrefixFilter? How it apply with Scanner?
Re: How efficient add row with columns to table?
Posted by Ted Yu <yu...@gmail.com>.
Looks Okay to me.
On Tue, Sep 19, 2017 at 9:53 AM, Andrzej <bo...@wp.pl> wrote:
> W dniu 18.09.2017 o 20:25, Ted Yu pisze:
>
>> There is no such logic on C++ client side.
>> When server receives the request, the columns would be added.
>>
>
> void hbPutColumns(uint32_t hTable, string row, vector<string>& families,
> vector<string>& qualifiers, vector<string>& values)
> {
> auto put = std::make_unique<Put>(row);
> assert(families.size() == qualifiers.size() && qualifiers.size()
> == values.size());
> for (int i=0; i<qualifiers.size(); i++)
> put->AddColumn(families[i], qualifiers[i], values[i]);
> getTable(hTable)->Put(*std::move(put));
> }
>
> Above method efficient adds columns to table?
>
Re: How efficient add row with columns to table?
Posted by Andrzej <bo...@wp.pl>.
W dniu 18.09.2017 o 20:25, Ted Yu pisze:
> There is no such logic on C++ client side.
> When server receives the request, the columns would be added.
void hbPutColumns(uint32_t hTable, string row, vector<string>& families,
vector<string>& qualifiers, vector<string>& values)
{
auto put = std::make_unique<Put>(row);
assert(families.size() == qualifiers.size() && qualifiers.size() ==
values.size());
for (int i=0; i<qualifiers.size(); i++)
put->AddColumn(families[i], qualifiers[i], values[i]);
getTable(hTable)->Put(*std::move(put));
}
Above method efficient adds columns to table?
Re: How efficient add row with columns to table?
Posted by Ted Yu <yu...@gmail.com>.
bq. locate (for column addings)
There is no such logic on C++ client side.
When server receives the request, the columns would be added.
On Mon, Sep 18, 2017 at 11:23 AM, Andrzej <bo...@wp.pl> wrote:
> W dniu 18.09.2017 o 20:15, Ted Yu pisze:
>
>> Looks like the row is "row357" as std::make_unique<Put>("row357")
>> indicates.
>>
>
> If code
> "std::make_unique<Put>("row357")"
> make new row if "row357" not exists
> or locate (for column addings) row "row357" if this row exists?
>
Re: How efficient add row with columns to table?
Posted by Andrzej <bo...@wp.pl>.
W dniu 18.09.2017 o 20:15, Ted Yu pisze:
> Looks like the row is "row357" as std::make_unique<Put>("row357") indicates.
If code
"std::make_unique<Put>("row357")"
make new row if "row357" not exists
or locate (for column addings) row "row357" if this row exists?
Re: How efficient add row with columns to table?
Posted by Ted Yu <yu...@gmail.com>.
You have a column called row_key. It is a bit confusing.
What would be the actual row for row357 ?
Looks like the row is "row357" as std::make_unique<Put>("row357") indicates.
On Mon, Sep 18, 2017 at 11:07 AM, Andrzej <bo...@wp.pl> wrote:
> W dniu 18.09.2017 o 19:47, Ted Yu pisze:
>
>> bq. column "q" is row_key,
>>
>> I don't know what you meant.
>> row key should be row, the parameter.
>>
>> If you want to add other columns, just call AddColumn() for those columns.
>>
>
> For example, table has columns: row_key, col1, col2, col3 (family f)
> I want add "row357","col11","col12","col13".
> code
> auto put = std::make_unique<Put>("row357");
> put->AddColumn("f", "col1", "col11");
> put->AddColumn("f", "col2", "col12");
> put->AddColumn("f", "col3", "col13");
> table->Put(std::move(put));
>
> I must additionally add :
> put->AddColumn("f", "row_key", "row357")
> or it is added in "std::make_unique<Put>("row357")"?
>
Re: How efficient add row with columns to table?
Posted by Andrzej <bo...@wp.pl>.
W dniu 18.09.2017 o 19:47, Ted Yu pisze:
> bq. column "q" is row_key,
>
> I don't know what you meant.
> row key should be row, the parameter.
>
> If you want to add other columns, just call AddColumn() for those columns.
For example, table has columns: row_key, col1, col2, col3 (family f)
I want add "row357","col11","col12","col13".
code
auto put = std::make_unique<Put>("row357");
put->AddColumn("f", "col1", "col11");
put->AddColumn("f", "col2", "col12");
put->AddColumn("f", "col3", "col13");
table->Put(std::move(put));
I must additionally add :
put->AddColumn("f", "row_key", "row357")
or it is added in "std::make_unique<Put>("row357")"?
Re: How efficient add row with columns to table?
Posted by Ted Yu <yu...@gmail.com>.
bq. column "q" is row_key,
I don't know what you meant.
row key should be row, the parameter.
If you want to add other columns, just call AddColumn() for those columns.
On Mon, Sep 18, 2017 at 10:42 AM, Andrzej <bo...@wp.pl> wrote:
> I simple-client.cc is:
> for (uint64_t i = 0; i < num_puts; i++)
> table->Put(*MakePut(Row(FLAGS_row, i)));
> where
> std::unique_ptr<Put> MakePut(const std::string &row) {
> auto put = std::make_unique<Put>(row);
> put->AddColumn("f", "q", row);
> return std::move(put);
> }
>
> But there are two problems:
> 1. In above example column "q" is row_key, but if I want add other column
> than row, I don't know on which row will be added.
> 2.How efficient add many columns?
>
> In general case , how add :
> row_key, column1, column2...column 10
>
>