You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Andrzej <bo...@wp.pl> on 2017/08/26 13:52:31 UTC
Questions about my bitcoin table
I need database for Bitcoin blockchain data. Previously I have two main
tables : general and detail in SQL database. These tables were tables of
transactions.
Now I want table of address, because main search would be search by
address. (unless is possible more than one index)
First approach is:
adr = address
txh = transaction hash
bh = block_height
bt = block_time
xtx = indx_tx = index transaction in block
cin = count_in
cout = count_out
xio = indx_io = index input/output in transaction
family general family detail
adr|| txh | bh | bt | xtx| fee | size| cin | cout || type | xio |amount
-----------------------------------------------------------------------
aaa|| x | 17 | t17| 5 | 200 | 450 | 5 | 2 || 0 | 2 | 10
|| | | | | | | | || 0 | 3 | 1
|| | | | | | | | || 1 | 0 | 5
||-----------------------------------------------------------------
|| y | 20 | t20| 1 | 100 | 250 | 1 | 1 || 0 | 0 | 10
||-----------------------------------------------------------------
|| z | 22 | t22| 100| | 50 | 10 | 20 || 1 | 5 | 10
|| | | | | | | | || 1 | 6 | 10
----------------------------------------------------------------------
abb|| x | 17 | t17| 6 | 210 | 400 | 2 | 2 || 1 | 0 | 5
|| | | | | | | | || 1 | 1 | 5
||-----------------------------------------------------------------
|| x2 | 39 | t39| 0| | 100 | 0 | 1 || 1 | 0 | 12
this nested approach has disadvantages:
adr is unique and if I add next row, old row will be replaced.
I am training after
https://github.com/borneq/thrift-exercises/blob/master/cpp/Hbase/DemoClient.cpp
there are issues:
- only families are created at start, columns are created when mutateRow:
mutations.push_back(Mutation());
mutations.back().column = "entry:num";
mutations.back().value = boost::lexical_cast<std::string>(i);
cells are filled by string, not bytes, this has disadvantages:
- hashes have 32 raw bytes or 64 hex strings
- numbers must be saved as strings like 0000000004, this disable
advantages numbers as integers.
My questions:
- how put data as bytes?
- is possible make table as transaction (not address) table and
efficient search address?
- unique key can be more than one column?
- better are two tables (general, detail) or one table with two families?
- in DemoClient.cpp were problems with mutation to empty value - how
save space for rows like:
|| | | | | | | | || 0 | 3 | 1
where are empty value because its value are redundant with first row.
Re: Questions about my bitcoin table
Posted by Andrzej <bo...@wp.pl>.
W dniu 26.08.2017 o 16:02, Ted Yu pisze:
> auto int_val = hbase::BytesUtil::ToInt64(*(result->Value(family, > incrPrefix + col)));
There is in load-client.cc
In method BytesUtil::ToInt64 is:
int64_t BytesUtil::ToInt64(std::string str) {
if (str.length() < 8) {
throw std::runtime_error("There are not enough bytes. Expected: 8,
actual: " + str.length());
}
const unsigned char *bytes = reinterpret_cast<unsigned char
*>(const_cast<char *>(str.c_str()));
int64_t l = 0;
for (int i = 0; i < 8; i++) {
l <<= 8;
l ^= bytes[i];
}
return l;
}
This means that is possible retrieve any binary data from string,
including not utf-8 chars and \0 char ?
Similar, put->AddColumn(family, qualifier, value) can add any binary
data as value casting it to char?
In example simple-test.cc was error when trying put empty string.
Re: Fast search by any column
Posted by Josh Elser <el...@apache.org>.
Well put, Dave! And yes, same for Phoenix.
HBase provides exactly one "index" (on rowkey). Thus it's the
application's responsibility to build other indexes to support different
kind of lookups. This is exactly what projects like Trafodian and
Phoenix (not to mention, others) do.
On 8/30/17 2:09 PM, Dave Birdsall wrote:
> Trafodion (and I think Phoenix also) both support secondary indexes. So you can create indexes on any attribute that you wish to search upon.
>
> Trafodion in addition allows one to pack multiple columns (logically speaking) into the HBase key. It also has a feature that allows intelligent use of multiple-column indexes. For example, if I have an index on STATE, CITY, and I have a predicate of the form CITY = 'St. Louis' (without a predicate on STATE), Trafodion can implicitly materialize the distinct STATE values efficiently and access CITY = 'St. Louis' directly using that index. So one does not have to create as many indexes as one might otherwise. This feature is useful when the table continues to grow or is updated; fewer indexes requires less overhead for index maintenance. But if the data set is static, you may as well just create indexes until your heart is content (space permitting of course).
>
> -----Original Message-----
> From: Andrzej [mailto:borucki_andrzej@wp.pl]
> Sent: Wednesday, August 30, 2017 11:03 AM
> To: user@hbase.apache.org
> Subject: Re: Fast search by any column
>
> W dniu 30.08.2017 o 19:54, Dave Birdsall pisze:
>> As Josh Elser mentioned, you might try Apache Phoenix.
>> You could try any SQL-on-HBase solution, actually. Apache Trafodion (incubating) is another example.
>
> As I understand, Apache Phoenix and Apache Trafodion are highest layer than HBase and both uses HBase. How they can fast seach, since HBase does not allow this?
>
RE: Fast search by any column
Posted by Dave Birdsall <da...@esgyn.com>.
Trafodion (and I think Phoenix also) both support secondary indexes. So you can create indexes on any attribute that you wish to search upon.
Trafodion in addition allows one to pack multiple columns (logically speaking) into the HBase key. It also has a feature that allows intelligent use of multiple-column indexes. For example, if I have an index on STATE, CITY, and I have a predicate of the form CITY = 'St. Louis' (without a predicate on STATE), Trafodion can implicitly materialize the distinct STATE values efficiently and access CITY = 'St. Louis' directly using that index. So one does not have to create as many indexes as one might otherwise. This feature is useful when the table continues to grow or is updated; fewer indexes requires less overhead for index maintenance. But if the data set is static, you may as well just create indexes until your heart is content (space permitting of course).
-----Original Message-----
From: Andrzej [mailto:borucki_andrzej@wp.pl]
Sent: Wednesday, August 30, 2017 11:03 AM
To: user@hbase.apache.org
Subject: Re: Fast search by any column
W dniu 30.08.2017 o 19:54, Dave Birdsall pisze:
> As Josh Elser mentioned, you might try Apache Phoenix.
> You could try any SQL-on-HBase solution, actually. Apache Trafodion (incubating) is another example.
As I understand, Apache Phoenix and Apache Trafodion are highest layer than HBase and both uses HBase. How they can fast seach, since HBase does not allow this?
Re: Fast search by any column
Posted by Andrzej <bo...@wp.pl>.
W dniu 30.08.2017 o 19:54, Dave Birdsall pisze:
> As Josh Elser mentioned, you might try Apache Phoenix.
> You could try any SQL-on-HBase solution, actually. Apache Trafodion (incubating) is another example.
As I understand, Apache Phoenix and Apache Trafodion are highest layer
than HBase and both uses HBase. How they can fast seach, since HBase
does not allow this?
RE: Fast search by any column
Posted by Dave Birdsall <da...@esgyn.com>.
As Josh Elser mentioned, you might try Apache Phoenix.
You could try any SQL-on-HBase solution, actually. Apache Trafodion (incubating) is another example.
-----Original Message-----
From: Andrzej [mailto:borucki_andrzej@wp.pl]
Sent: Wednesday, August 30, 2017 10:32 AM
To: user@hbase.apache.org
Subject: Re: Fast search by any column
I am unfortunately unfamiliar with SOLR; is any other solution based on Scan,Get and Query, fast search with secondary indexes ?
> Index all your data into SOLR, make it return you the row key for each > lookup?
Re: Fast search by any column
Posted by Andrzej <bo...@wp.pl>.
I am unfortunately unfamiliar with SOLR;
is any other solution based on Scan,Get and Query, fast search with
secondary indexes ?
> Index all your data into SOLR, make it return you the row key for each
> lookup?
Re: Fast search by any column
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Andrzej,
Index all your data into SOLR, make it return you the row key for each
lookup?
JMS
2017-08-28 14:51 GMT-04:00 Andrzej <bo...@wp.pl>:
> How add index to any column?
>
Fast search by any column
Posted by Andrzej <bo...@wp.pl>.
How add index to any column?
Re: Folly not installed
Posted by Ted Yu <yu...@gmail.com>.
I assume you used the following to start docker:
hbase-native-client/bin/start-docker.sh
Did you use this command to build simple-client ?
buck build //core:simple-client
Running buck-out/gen/core/simple-client , I see this output:
https://pastebin.com/n0SPm7Bm
There was no complaint about libsasl2.so.3
On Tue, Aug 29, 2017 at 5:27 AM, Andrzej <bo...@wp.pl> wrote:
> I use docker. Next I use buck inside docker.
> Buck creates me buck-out directory - 3.1 GB
> Where is output native client library?
> How use native client?
> I found simple_client at buck-out/gen/core
> no libsasl2 sharted library
>
> andrzej@andrzej-VirtualBox ~/code/hbase/hbase-native-client/buck-out/gen/core
> $ ./simple-client
> ./simple-client: error while loading shared libraries: libsasl2.so.3:
> cannot open shared object file: No such file or directory
>
>
> W dniu 28.08.2017 o 17:53, Ted Yu pisze:
>
> Did you try to build using docker ?
>>
>> If not, what error(s) did you encounter building folly ?
>> Please use 2017.06.19.00 version as shown
>> in hbase-native-client/docker-files/Dockerfile
>>
>
Re: Folly not installed
Posted by Ted Yu <yu...@gmail.com>.
load-client is a native client which exercises puts / appends / increments
operations using multiple threads.
It also has logic to verify that data is written correctly using multi-get
/ get / scan operations.
The HBASE-14850 branch was made off of master branch. So the load-client
works with master branch.
As I said previously, it works with 1.1.2 cluster (and 1.2, 1.3 since RPC
is compatible).
w.r.t. general consumption, it is not fully ready yet.
On Tue, Aug 29, 2017 at 4:34 PM, Sean Busbey <bu...@apache.org> wrote:
> +dev@hbase
> -user@hbase to bcc
>
> AFAIK the native client isn't ready for downstream consumption yet.
> can we please move this discussion to the dev@ list?
>
> On Tue, Aug 29, 2017 at 11:57 AM, Stack <st...@duboce.net> wrote:
> > On Tue, Aug 29, 2017 at 8:54 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> >> I have successfully run load-client against an OpenStack cluster running
> >> hbase 1.1.2 .
> >>
> >>
> > What does load-client do?
> >
> > Does it not work w/ master, 1.2 or 1.3?
> >
> >
> >> There is still some gap between the buck build and the Makefile build
> (e.g.
> >> load-client and simple-client cannot be built with Makefile) -
> according to
> >> Enis, separate tickets would be opened.
> >>
> >> Other pending work includes multiput (HBASE-18507).
> >>
> >>
> > Is it ready for general consumption by general users or does it have too
> > many holes still? Should they wait?
> >
> > St.Ack
> >
> >
> >
> >
> >> Cheers
> >>
> >> On Tue, Aug 29, 2017 at 8:44 AM, Stack <st...@duboce.net> wrote:
> >>
> >> > On Tue, Aug 29, 2017 at 7:01 AM, Ted Yu <yu...@gmail.com> wrote:
> >> >
> >> > > You can follow the structure of simple-client to create your own
> >> program.
> >> > >
> >> > > bq. it is possible compiling my own program without docker and buck?
> >> > >
> >> > > You can use make.
> >> > >
> >> > > bq. it is problem with includes <folly/...>
> >> > >
> >> > > Folly is used heavily by the native client source code. Ultimately
> your
> >> > > native program would include Folly.
> >> > >
> >> > >
> >> > Does the native client work? Is it complete? If not, what is TODO
> still?
> >> > Does it have limitations? If so, what are they?
> >> >
> >> > Thanks,
> >> > S
> >> >
> >> >
> >> >
> >> >
> >> > > Cheers
> >> > >
> >> > > On Tue, Aug 29, 2017 at 6:36 AM, Andrzej <bo...@wp.pl>
> >> wrote:
> >> > >
> >> > > > W dniu 29.08.2017 o 14:27, Andrzej pisze:
> >> > > >
> >> > > >> I found simple_client at buck-out/gen/core
> >> > > >> no libsasl2 sharted library
> >> > > >>
> >> > > >
> >> > > > libsasl2.so.3 I add symbolic link to libsasl2.so.2 and is ok,
> >> > > > buck build number executable like simple-client but I am
> interested
> >> > > > libraries *.a to build my own program using Hbase.
> >> > > > There are :
> >> > > > libfolly.a
> >> > > > libprotobuf.a
> >> > > > libif.a
> >> > > > libwangle.a
> >> > > > libcore.a
> >> > > > .....
> >> > > > Which I must use in my own program? it is possible compiling my
> own
> >> > > > program without docker and buck?
> >> > > > When I compiling , it is problem with includes <folly/...> which
> >> > includes
> >> > > > many other files.
> >> > > > Is possible crate program which crate/delete tables and its source
> >> not
> >> > > > included <folly/..> but only link libfolly.a ?
> >> > > >
> >> > >
> >> >
> >>
>
Re: Folly not installed
Posted by Ted Yu <yu...@gmail.com>.
load-client is a native client which exercises puts / appends / increments
operations using multiple threads.
It also has logic to verify that data is written correctly using multi-get
/ get / scan operations.
The HBASE-14850 branch was made off of master branch. So the load-client
works with master branch.
As I said previously, it works with 1.1.2 cluster (and 1.2, 1.3 since RPC
is compatible).
w.r.t. general consumption, it is not fully ready yet.
On Tue, Aug 29, 2017 at 4:34 PM, Sean Busbey <bu...@apache.org> wrote:
> +dev@hbase
> -user@hbase to bcc
>
> AFAIK the native client isn't ready for downstream consumption yet.
> can we please move this discussion to the dev@ list?
>
> On Tue, Aug 29, 2017 at 11:57 AM, Stack <st...@duboce.net> wrote:
> > On Tue, Aug 29, 2017 at 8:54 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> >> I have successfully run load-client against an OpenStack cluster running
> >> hbase 1.1.2 .
> >>
> >>
> > What does load-client do?
> >
> > Does it not work w/ master, 1.2 or 1.3?
> >
> >
> >> There is still some gap between the buck build and the Makefile build
> (e.g.
> >> load-client and simple-client cannot be built with Makefile) -
> according to
> >> Enis, separate tickets would be opened.
> >>
> >> Other pending work includes multiput (HBASE-18507).
> >>
> >>
> > Is it ready for general consumption by general users or does it have too
> > many holes still? Should they wait?
> >
> > St.Ack
> >
> >
> >
> >
> >> Cheers
> >>
> >> On Tue, Aug 29, 2017 at 8:44 AM, Stack <st...@duboce.net> wrote:
> >>
> >> > On Tue, Aug 29, 2017 at 7:01 AM, Ted Yu <yu...@gmail.com> wrote:
> >> >
> >> > > You can follow the structure of simple-client to create your own
> >> program.
> >> > >
> >> > > bq. it is possible compiling my own program without docker and buck?
> >> > >
> >> > > You can use make.
> >> > >
> >> > > bq. it is problem with includes <folly/...>
> >> > >
> >> > > Folly is used heavily by the native client source code. Ultimately
> your
> >> > > native program would include Folly.
> >> > >
> >> > >
> >> > Does the native client work? Is it complete? If not, what is TODO
> still?
> >> > Does it have limitations? If so, what are they?
> >> >
> >> > Thanks,
> >> > S
> >> >
> >> >
> >> >
> >> >
> >> > > Cheers
> >> > >
> >> > > On Tue, Aug 29, 2017 at 6:36 AM, Andrzej <bo...@wp.pl>
> >> wrote:
> >> > >
> >> > > > W dniu 29.08.2017 o 14:27, Andrzej pisze:
> >> > > >
> >> > > >> I found simple_client at buck-out/gen/core
> >> > > >> no libsasl2 sharted library
> >> > > >>
> >> > > >
> >> > > > libsasl2.so.3 I add symbolic link to libsasl2.so.2 and is ok,
> >> > > > buck build number executable like simple-client but I am
> interested
> >> > > > libraries *.a to build my own program using Hbase.
> >> > > > There are :
> >> > > > libfolly.a
> >> > > > libprotobuf.a
> >> > > > libif.a
> >> > > > libwangle.a
> >> > > > libcore.a
> >> > > > .....
> >> > > > Which I must use in my own program? it is possible compiling my
> own
> >> > > > program without docker and buck?
> >> > > > When I compiling , it is problem with includes <folly/...> which
> >> > includes
> >> > > > many other files.
> >> > > > Is possible crate program which crate/delete tables and its source
> >> not
> >> > > > included <folly/..> but only link libfolly.a ?
> >> > > >
> >> > >
> >> >
> >>
>
Re: Folly not installed
Posted by Sean Busbey <bu...@apache.org>.
+dev@hbase
-user@hbase to bcc
AFAIK the native client isn't ready for downstream consumption yet.
can we please move this discussion to the dev@ list?
On Tue, Aug 29, 2017 at 11:57 AM, Stack <st...@duboce.net> wrote:
> On Tue, Aug 29, 2017 at 8:54 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> I have successfully run load-client against an OpenStack cluster running
>> hbase 1.1.2 .
>>
>>
> What does load-client do?
>
> Does it not work w/ master, 1.2 or 1.3?
>
>
>> There is still some gap between the buck build and the Makefile build (e.g.
>> load-client and simple-client cannot be built with Makefile) - according to
>> Enis, separate tickets would be opened.
>>
>> Other pending work includes multiput (HBASE-18507).
>>
>>
> Is it ready for general consumption by general users or does it have too
> many holes still? Should they wait?
>
> St.Ack
>
>
>
>
>> Cheers
>>
>> On Tue, Aug 29, 2017 at 8:44 AM, Stack <st...@duboce.net> wrote:
>>
>> > On Tue, Aug 29, 2017 at 7:01 AM, Ted Yu <yu...@gmail.com> wrote:
>> >
>> > > You can follow the structure of simple-client to create your own
>> program.
>> > >
>> > > bq. it is possible compiling my own program without docker and buck?
>> > >
>> > > You can use make.
>> > >
>> > > bq. it is problem with includes <folly/...>
>> > >
>> > > Folly is used heavily by the native client source code. Ultimately your
>> > > native program would include Folly.
>> > >
>> > >
>> > Does the native client work? Is it complete? If not, what is TODO still?
>> > Does it have limitations? If so, what are they?
>> >
>> > Thanks,
>> > S
>> >
>> >
>> >
>> >
>> > > Cheers
>> > >
>> > > On Tue, Aug 29, 2017 at 6:36 AM, Andrzej <bo...@wp.pl>
>> wrote:
>> > >
>> > > > W dniu 29.08.2017 o 14:27, Andrzej pisze:
>> > > >
>> > > >> I found simple_client at buck-out/gen/core
>> > > >> no libsasl2 sharted library
>> > > >>
>> > > >
>> > > > libsasl2.so.3 I add symbolic link to libsasl2.so.2 and is ok,
>> > > > buck build number executable like simple-client but I am interested
>> > > > libraries *.a to build my own program using Hbase.
>> > > > There are :
>> > > > libfolly.a
>> > > > libprotobuf.a
>> > > > libif.a
>> > > > libwangle.a
>> > > > libcore.a
>> > > > .....
>> > > > Which I must use in my own program? it is possible compiling my own
>> > > > program without docker and buck?
>> > > > When I compiling , it is problem with includes <folly/...> which
>> > includes
>> > > > many other files.
>> > > > Is possible crate program which crate/delete tables and its source
>> not
>> > > > included <folly/..> but only link libfolly.a ?
>> > > >
>> > >
>> >
>>
Re: Folly not installed
Posted by Sean Busbey <bu...@apache.org>.
+dev@hbase
-user@hbase to bcc
AFAIK the native client isn't ready for downstream consumption yet.
can we please move this discussion to the dev@ list?
On Tue, Aug 29, 2017 at 11:57 AM, Stack <st...@duboce.net> wrote:
> On Tue, Aug 29, 2017 at 8:54 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> I have successfully run load-client against an OpenStack cluster running
>> hbase 1.1.2 .
>>
>>
> What does load-client do?
>
> Does it not work w/ master, 1.2 or 1.3?
>
>
>> There is still some gap between the buck build and the Makefile build (e.g.
>> load-client and simple-client cannot be built with Makefile) - according to
>> Enis, separate tickets would be opened.
>>
>> Other pending work includes multiput (HBASE-18507).
>>
>>
> Is it ready for general consumption by general users or does it have too
> many holes still? Should they wait?
>
> St.Ack
>
>
>
>
>> Cheers
>>
>> On Tue, Aug 29, 2017 at 8:44 AM, Stack <st...@duboce.net> wrote:
>>
>> > On Tue, Aug 29, 2017 at 7:01 AM, Ted Yu <yu...@gmail.com> wrote:
>> >
>> > > You can follow the structure of simple-client to create your own
>> program.
>> > >
>> > > bq. it is possible compiling my own program without docker and buck?
>> > >
>> > > You can use make.
>> > >
>> > > bq. it is problem with includes <folly/...>
>> > >
>> > > Folly is used heavily by the native client source code. Ultimately your
>> > > native program would include Folly.
>> > >
>> > >
>> > Does the native client work? Is it complete? If not, what is TODO still?
>> > Does it have limitations? If so, what are they?
>> >
>> > Thanks,
>> > S
>> >
>> >
>> >
>> >
>> > > Cheers
>> > >
>> > > On Tue, Aug 29, 2017 at 6:36 AM, Andrzej <bo...@wp.pl>
>> wrote:
>> > >
>> > > > W dniu 29.08.2017 o 14:27, Andrzej pisze:
>> > > >
>> > > >> I found simple_client at buck-out/gen/core
>> > > >> no libsasl2 sharted library
>> > > >>
>> > > >
>> > > > libsasl2.so.3 I add symbolic link to libsasl2.so.2 and is ok,
>> > > > buck build number executable like simple-client but I am interested
>> > > > libraries *.a to build my own program using Hbase.
>> > > > There are :
>> > > > libfolly.a
>> > > > libprotobuf.a
>> > > > libif.a
>> > > > libwangle.a
>> > > > libcore.a
>> > > > .....
>> > > > Which I must use in my own program? it is possible compiling my own
>> > > > program without docker and buck?
>> > > > When I compiling , it is problem with includes <folly/...> which
>> > includes
>> > > > many other files.
>> > > > Is possible crate program which crate/delete tables and its source
>> not
>> > > > included <folly/..> but only link libfolly.a ?
>> > > >
>> > >
>> >
>>
Re: Folly not installed
Posted by Stack <st...@duboce.net>.
On Tue, Aug 29, 2017 at 8:54 AM, Ted Yu <yu...@gmail.com> wrote:
> I have successfully run load-client against an OpenStack cluster running
> hbase 1.1.2 .
>
>
What does load-client do?
Does it not work w/ master, 1.2 or 1.3?
> There is still some gap between the buck build and the Makefile build (e.g.
> load-client and simple-client cannot be built with Makefile) - according to
> Enis, separate tickets would be opened.
>
> Other pending work includes multiput (HBASE-18507).
>
>
Is it ready for general consumption by general users or does it have too
many holes still? Should they wait?
St.Ack
> Cheers
>
> On Tue, Aug 29, 2017 at 8:44 AM, Stack <st...@duboce.net> wrote:
>
> > On Tue, Aug 29, 2017 at 7:01 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > You can follow the structure of simple-client to create your own
> program.
> > >
> > > bq. it is possible compiling my own program without docker and buck?
> > >
> > > You can use make.
> > >
> > > bq. it is problem with includes <folly/...>
> > >
> > > Folly is used heavily by the native client source code. Ultimately your
> > > native program would include Folly.
> > >
> > >
> > Does the native client work? Is it complete? If not, what is TODO still?
> > Does it have limitations? If so, what are they?
> >
> > Thanks,
> > S
> >
> >
> >
> >
> > > Cheers
> > >
> > > On Tue, Aug 29, 2017 at 6:36 AM, Andrzej <bo...@wp.pl>
> wrote:
> > >
> > > > W dniu 29.08.2017 o 14:27, Andrzej pisze:
> > > >
> > > >> I found simple_client at buck-out/gen/core
> > > >> no libsasl2 sharted library
> > > >>
> > > >
> > > > libsasl2.so.3 I add symbolic link to libsasl2.so.2 and is ok,
> > > > buck build number executable like simple-client but I am interested
> > > > libraries *.a to build my own program using Hbase.
> > > > There are :
> > > > libfolly.a
> > > > libprotobuf.a
> > > > libif.a
> > > > libwangle.a
> > > > libcore.a
> > > > .....
> > > > Which I must use in my own program? it is possible compiling my own
> > > > program without docker and buck?
> > > > When I compiling , it is problem with includes <folly/...> which
> > includes
> > > > many other files.
> > > > Is possible crate program which crate/delete tables and its source
> not
> > > > included <folly/..> but only link libfolly.a ?
> > > >
> > >
> >
>
Re: How send/receive (if need) native client messages?
Posted by Sean Busbey <bu...@apache.org>.
+dev@hbase
-user@hbase
please move this discussion to the dev list.
On Tue, Aug 29, 2017 at 11:22 AM, Ted Yu <yu...@gmail.com> wrote:
> GetTableNames API hasn't been implemented yet.
>
> If you have bandwidth, you're welcome to contribute to HBASE-14850 (as
> subtask).
>
> On Tue, Aug 29, 2017 at 9:11 AM, Andrzej <bo...@wp.pl> wrote:
>
>> I try code:
>> hbase::pb::GetTableNamesRequest getTableNamesRequest;
>> //<---- how send request?
>> hbase::pb::GetTableNamesResponse getTableNamesResponse;
>> //<---- how receive response?
>> int n = getTableNamesResponse.table_names_size();
>> std::cout << n << std::endl; // <------ is 0
>> for (int i=0; i<n; i++)
>> {
>> ::hbase::pb::TableName const name =
>> getTableNamesResponse.table_names(i);
>> bool has = name.has_namespace_();
>> std::cout << has << std::endl;
>> }
>>
>> or is another way to list table names?
>>
Re: How send/receive (if need) native client messages?
Posted by Sean Busbey <bu...@apache.org>.
+dev@hbase
-user@hbase
please move this discussion to the dev list.
On Tue, Aug 29, 2017 at 11:22 AM, Ted Yu <yu...@gmail.com> wrote:
> GetTableNames API hasn't been implemented yet.
>
> If you have bandwidth, you're welcome to contribute to HBASE-14850 (as
> subtask).
>
> On Tue, Aug 29, 2017 at 9:11 AM, Andrzej <bo...@wp.pl> wrote:
>
>> I try code:
>> hbase::pb::GetTableNamesRequest getTableNamesRequest;
>> //<---- how send request?
>> hbase::pb::GetTableNamesResponse getTableNamesResponse;
>> //<---- how receive response?
>> int n = getTableNamesResponse.table_names_size();
>> std::cout << n << std::endl; // <------ is 0
>> for (int i=0; i<n; i++)
>> {
>> ::hbase::pb::TableName const name =
>> getTableNamesResponse.table_names(i);
>> bool has = name.has_namespace_();
>> std::cout << has << std::endl;
>> }
>>
>> or is another way to list table names?
>>
Re: How send/receive (if need) native client messages?
Posted by Ted Yu <yu...@gmail.com>.
GetTableNames API hasn't been implemented yet.
If you have bandwidth, you're welcome to contribute to HBASE-14850 (as
subtask).
On Tue, Aug 29, 2017 at 9:11 AM, Andrzej <bo...@wp.pl> wrote:
> I try code:
> hbase::pb::GetTableNamesRequest getTableNamesRequest;
> //<---- how send request?
> hbase::pb::GetTableNamesResponse getTableNamesResponse;
> //<---- how receive response?
> int n = getTableNamesResponse.table_names_size();
> std::cout << n << std::endl; // <------ is 0
> for (int i=0; i<n; i++)
> {
> ::hbase::pb::TableName const name =
> getTableNamesResponse.table_names(i);
> bool has = name.has_namespace_();
> std::cout << has << std::endl;
> }
>
> or is another way to list table names?
>
How send/receive (if need) native client messages?
Posted by Andrzej <bo...@wp.pl>.
I try code:
hbase::pb::GetTableNamesRequest getTableNamesRequest;
//<---- how send request?
hbase::pb::GetTableNamesResponse getTableNamesResponse;
//<---- how receive response?
int n = getTableNamesResponse.table_names_size();
std::cout << n << std::endl; // <------ is 0
for (int i=0; i<n; i++)
{
::hbase::pb::TableName const name =
getTableNamesResponse.table_names(i);
bool has = name.has_namespace_();
std::cout << has << std::endl;
}
or is another way to list table names?
Re: Folly not installed
Posted by Ted Yu <yu...@gmail.com>.
I have successfully run load-client against an OpenStack cluster running
hbase 1.1.2 .
There is still some gap between the buck build and the Makefile build (e.g.
load-client and simple-client cannot be built with Makefile) - according to
Enis, separate tickets would be opened.
Other pending work includes multiput (HBASE-18507).
Cheers
On Tue, Aug 29, 2017 at 8:44 AM, Stack <st...@duboce.net> wrote:
> On Tue, Aug 29, 2017 at 7:01 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > You can follow the structure of simple-client to create your own program.
> >
> > bq. it is possible compiling my own program without docker and buck?
> >
> > You can use make.
> >
> > bq. it is problem with includes <folly/...>
> >
> > Folly is used heavily by the native client source code. Ultimately your
> > native program would include Folly.
> >
> >
> Does the native client work? Is it complete? If not, what is TODO still?
> Does it have limitations? If so, what are they?
>
> Thanks,
> S
>
>
>
>
> > Cheers
> >
> > On Tue, Aug 29, 2017 at 6:36 AM, Andrzej <bo...@wp.pl> wrote:
> >
> > > W dniu 29.08.2017 o 14:27, Andrzej pisze:
> > >
> > >> I found simple_client at buck-out/gen/core
> > >> no libsasl2 sharted library
> > >>
> > >
> > > libsasl2.so.3 I add symbolic link to libsasl2.so.2 and is ok,
> > > buck build number executable like simple-client but I am interested
> > > libraries *.a to build my own program using Hbase.
> > > There are :
> > > libfolly.a
> > > libprotobuf.a
> > > libif.a
> > > libwangle.a
> > > libcore.a
> > > .....
> > > Which I must use in my own program? it is possible compiling my own
> > > program without docker and buck?
> > > When I compiling , it is problem with includes <folly/...> which
> includes
> > > many other files.
> > > Is possible crate program which crate/delete tables and its source not
> > > included <folly/..> but only link libfolly.a ?
> > >
> >
>
Re: Folly not installed
Posted by Stack <st...@duboce.net>.
On Tue, Aug 29, 2017 at 7:01 AM, Ted Yu <yu...@gmail.com> wrote:
> You can follow the structure of simple-client to create your own program.
>
> bq. it is possible compiling my own program without docker and buck?
>
> You can use make.
>
> bq. it is problem with includes <folly/...>
>
> Folly is used heavily by the native client source code. Ultimately your
> native program would include Folly.
>
>
Does the native client work? Is it complete? If not, what is TODO still?
Does it have limitations? If so, what are they?
Thanks,
S
> Cheers
>
> On Tue, Aug 29, 2017 at 6:36 AM, Andrzej <bo...@wp.pl> wrote:
>
> > W dniu 29.08.2017 o 14:27, Andrzej pisze:
> >
> >> I found simple_client at buck-out/gen/core
> >> no libsasl2 sharted library
> >>
> >
> > libsasl2.so.3 I add symbolic link to libsasl2.so.2 and is ok,
> > buck build number executable like simple-client but I am interested
> > libraries *.a to build my own program using Hbase.
> > There are :
> > libfolly.a
> > libprotobuf.a
> > libif.a
> > libwangle.a
> > libcore.a
> > .....
> > Which I must use in my own program? it is possible compiling my own
> > program without docker and buck?
> > When I compiling , it is problem with includes <folly/...> which includes
> > many other files.
> > Is possible crate program which crate/delete tables and its source not
> > included <folly/..> but only link libfolly.a ?
> >
>
Re: Folly not installed
Posted by Ted Yu <yu...@gmail.com>.
You can follow the structure of simple-client to create your own program.
bq. it is possible compiling my own program without docker and buck?
You can use make.
bq. it is problem with includes <folly/...>
Folly is used heavily by the native client source code. Ultimately your
native program would include Folly.
Cheers
On Tue, Aug 29, 2017 at 6:36 AM, Andrzej <bo...@wp.pl> wrote:
> W dniu 29.08.2017 o 14:27, Andrzej pisze:
>
>> I found simple_client at buck-out/gen/core
>> no libsasl2 sharted library
>>
>
> libsasl2.so.3 I add symbolic link to libsasl2.so.2 and is ok,
> buck build number executable like simple-client but I am interested
> libraries *.a to build my own program using Hbase.
> There are :
> libfolly.a
> libprotobuf.a
> libif.a
> libwangle.a
> libcore.a
> .....
> Which I must use in my own program? it is possible compiling my own
> program without docker and buck?
> When I compiling , it is problem with includes <folly/...> which includes
> many other files.
> Is possible crate program which crate/delete tables and its source not
> included <folly/..> but only link libfolly.a ?
>
Re: Folly not installed
Posted by Andrzej <bo...@wp.pl>.
W dniu 29.08.2017 o 14:27, Andrzej pisze:
> I found simple_client at buck-out/gen/core
> no libsasl2 sharted library
libsasl2.so.3 I add symbolic link to libsasl2.so.2 and is ok,
buck build number executable like simple-client but I am interested
libraries *.a to build my own program using Hbase.
There are :
libfolly.a
libprotobuf.a
libif.a
libwangle.a
libcore.a
.....
Which I must use in my own program? it is possible compiling my own
program without docker and buck?
When I compiling , it is problem with includes <folly/...> which
includes many other files.
Is possible crate program which crate/delete tables and its source not
included <folly/..> but only link libfolly.a ?
Re: Folly not installed
Posted by Andrzej <bo...@wp.pl>.
I use docker. Next I use buck inside docker.
Buck creates me buck-out directory - 3.1 GB
Where is output native client library?
How use native client?
I found simple_client at buck-out/gen/core
no libsasl2 sharted library
andrzej@andrzej-VirtualBox
~/code/hbase/hbase-native-client/buck-out/gen/core $ ./simple-client
./simple-client: error while loading shared libraries: libsasl2.so.3:
cannot open shared object file: No such file or directory
W dniu 28.08.2017 o 17:53, Ted Yu pisze:
> Did you try to build using docker ?
>
> If not, what error(s) did you encounter building folly ?
> Please use 2017.06.19.00 version as shown
> in hbase-native-client/docker-files/Dockerfile
Re: Folly not installed
Posted by Ted Yu <yu...@gmail.com>.
Did you try to build using docker ?
If not, what error(s) did you encounter building folly ?
Please use 2017.06.19.00 version as shown
in hbase-native-client/docker-files/Dockerfile
Cheers
On Mon, Aug 28, 2017 at 8:50 AM, Andrzej <bo...@wp.pl> wrote:
> I try build hbase-native-client, I am writing buck build //core:core
> but
> fatal error: folly/futures/Future.h: no such file or directory
> Problem is build folly, because I have Linux Mint, and folly must be
> compiled on Windows 64 bit.
>
>
Folly not installed
Posted by Andrzej <bo...@wp.pl>.
I try build hbase-native-client, I am writing buck build //core:core
but
fatal error: folly/futures/Future.h: no such file or directory
Problem is build folly, because I have Linux Mint, and folly must be
compiled on Windows 64 bit.
Re: Questions about my bitcoin table
Posted by Ted Yu <yu...@gmail.com>.
Take a look at hbase-native-client/BUILDING.md in the HBASE-14850 branch.
hbase-native-client/core/load-client.cc shows you how to do append / put /
increment / scan / get.
Cheers
On Mon, Aug 28, 2017 at 4:24 AM, Andrzej <bo...@wp.pl> wrote:
> W dniu 26.08.2017 o 16:02, Ted Yu pisze:
>
>> auto int_val = hbase::BytesUtil::ToInt64(*(result->Value(family,
>> incrPrefix + col)));
>>
>
> Is in https://github.com/apache/hbase.git
> in hbase-native-client/core/load-client.cc
> It means, is native C++ client and no need thrift? How use native client?
>
Re: Questions about my bitcoin table
Posted by Andrzej <bo...@wp.pl>.
W dniu 26.08.2017 o 16:02, Ted Yu pisze:
> auto int_val = hbase::BytesUtil::ToInt64(*(result->Value(family,
> incrPrefix + col)));
Is in https://github.com/apache/hbase.git
in hbase-native-client/core/load-client.cc
It means, is native C++ client and no need thrift? How use native client?
Re: Questions about my bitcoin table
Posted by Ted Yu <yu...@gmail.com>.
For #1, take a look at load-client in HBASE-14850 branch where you can find
example for converting bytes to int:
auto int_val = hbase::BytesUtil::ToInt64(*(result->Value(family,
incrPrefix + col)));
you can use std::to_string(k) to convert integer to string.
For #3, adr is unique and if I add next row, old row will be replaced.
You can either make the row for each unique address wide by putting the
columns (for the address) in one row or, adding prefix / suffix to the
address so that the rows have unique row keys.
Cheers
On Sat, Aug 26, 2017 at 6:52 AM, Andrzej <bo...@wp.pl> wrote:
> I need database for Bitcoin blockchain data. Previously I have two main
> tables : general and detail in SQL database. These tables were tables of
> transactions.
> Now I want table of address, because main search would be search by
> address. (unless is possible more than one index)
> First approach is:
> adr = address
> txh = transaction hash
> bh = block_height
> bt = block_time
> xtx = indx_tx = index transaction in block
> cin = count_in
> cout = count_out
> xio = indx_io = index input/output in transaction
>
> family general family detail
> adr|| txh | bh | bt | xtx| fee | size| cin | cout || type | xio |amount
> -----------------------------------------------------------------------
> aaa|| x | 17 | t17| 5 | 200 | 450 | 5 | 2 || 0 | 2 | 10
> || | | | | | | | || 0 | 3 | 1
> || | | | | | | | || 1 | 0 | 5
> ||-----------------------------------------------------------------
> || y | 20 | t20| 1 | 100 | 250 | 1 | 1 || 0 | 0 | 10
> ||-----------------------------------------------------------------
> || z | 22 | t22| 100| | 50 | 10 | 20 || 1 | 5 | 10
> || | | | | | | | || 1 | 6 | 10
> ----------------------------------------------------------------------
> abb|| x | 17 | t17| 6 | 210 | 400 | 2 | 2 || 1 | 0 | 5
> || | | | | | | | || 1 | 1 | 5
> ||-----------------------------------------------------------------
> || x2 | 39 | t39| 0| | 100 | 0 | 1 || 1 | 0 | 12
>
> this nested approach has disadvantages:
> adr is unique and if I add next row, old row will be replaced.
>
> I am training after https://github.com/borneq/thri
> ft-exercises/blob/master/cpp/Hbase/DemoClient.cpp
> there are issues:
> - only families are created at start, columns are created when mutateRow:
> mutations.push_back(Mutation());
> mutations.back().column = "entry:num";
> mutations.back().value = boost::lexical_cast<std::string>(i);
> cells are filled by string, not bytes, this has disadvantages:
> - hashes have 32 raw bytes or 64 hex strings
> - numbers must be saved as strings like 0000000004, this disable
> advantages numbers as integers.
>
> My questions:
> - how put data as bytes?
> - is possible make table as transaction (not address) table and efficient
> search address?
> - unique key can be more than one column?
> - better are two tables (general, detail) or one table with two families?
> - in DemoClient.cpp were problems with mutation to empty value - how save
> space for rows like:
> || | | | | | | | || 0 | 3 | 1
> where are empty value because its value are redundant with first row.
>