You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Carol McDonald <cm...@maprtech.com> on 2015/08/01 04:59:04 UTC
Re: Convert Hbase Values from Bytes to Actual Strings
the point I was trying to make is :
The good news is, Drill does provide a nice simple way to abstract these
details away. You simply create a view on top of HBase [1]. The view can
contain the physical conversions. Then users can interact with the view
rather than the underlying table.
On Fri, Jul 31, 2015 at 11:36 AM, Jacques Nadeau <ja...@dremio.com> wrote:
> Carol, your statements are misleading. There are some situations where
> people have stored in HBase encoded as text. In other cases, they've used
> a straight binary encoding. CAST is used generally used when the data is
> already encoded as a UTF8 string. Otherwise, the user should use
> CONVERT_FROM to decode the binary value.
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Fri, Jul 31, 2015 at 7:42 AM, Carol McDonald <cm...@maprtech.com>
> wrote:
>
> > This blog shows an example using REST , drill , and HBase. In this
> example
> > a view converts the HBase byte values to strings
> >
> >
> >
> https://www.mapr.com/blog/how-use-sql-hadoop-drill-rest-json-nosql-and-hbase-simple-rest-client
> >
> > 0: jdbc:drill:> use dfs.mydata;
> >
> > create or replace view prodview as SELECT CAST(row_key AS INTEGER) AS
> > prod_id, CAST(t.details.category AS VARCHAR(40)) AS category,
> > CAST(t.details.name AS VARCHAR(129)) AS name FROM maprdb.products t;
> >
> >
> > On Tue, Jul 28, 2015 at 11:35 AM, Usman Ali <us...@platalytics.com>
> > wrote:
> >
> > > Hi,
> > > When we query an Hbase table like "select * from
> > > hbasePluginName.`tableName`" using Restful APIs, the json it returns
> > > contains values in the form of bytes. I know that all the values in my
> > > hbase table were originally of String form but I do not know exact
> schema
> > > (column families and column names) of my table. How can I convert those
> > > bytes stored in strings back to my original strings.
> > >
> > > Thanks.
> > > Usman Ali
> > >
> >
>
Re: Convert Hbase Values from Bytes to Actual Strings
Posted by Abdel Hakim Deneche <ad...@maprtech.com>.
>> Are
>> we suffering performance hits when we do queries on 60+ columns?
>
> No. Extra information referenced in the view is pruned away if it isn't
> directly in your query so you don't pay any penalty for excess complexity
> described in the view.
Unless it's a bug of course, like DRILL-2412
<https://issues.apache.org/jira/browse/DRILL-3412>. If you see any
performance hits, check the query's plan, if the scan is reading more
columns than needed it may be a bug.
On Thu, Aug 6, 2015 at 5:22 PM, Ted Dunning <te...@gmail.com> wrote:
> On Sat, Aug 1, 2015 at 5:53 PM, John Omernik <jo...@omernik.com> wrote:
>
> > I am working with this as well and I find the "abstraction" frustrating
> > from a user perspective as well.
>
>
> Yes. It can be a pain.
>
>
> > I understand we can create views, I've
> > done that, but it's a tedious process of ensuring my syntax is correct
> and
> > one that I did have the Schema on hand to easily do this. Could a
> feature
> > be added to "register" a view doing this for us based on the schema" ?
>
>
> It should be relatively easy to build a tool to convert schema information
> that you have into a view definition.
>
>
>
> > Are
> > we suffering performance hits when we do queries on 60+ columns?
>
>
> No. Extra information referenced in the view is pruned away if it isn't
> directly in your query so you don't pay any penalty for excess complexity
> described in the view.
>
>
> > This
> > would make this integration much more intuitive for users who know sql,
> > know their data is in an hbase table, and just want to go query. (Trying
> to
> > make this more self serve for users)
> >
>
> Yes. A GUI would be a nice-to-have here. It isn't clear if this will be
> high enough priority to happen right away.
>
>
>
>
> >
> > John
> >
> > On Sat, Aug 1, 2015 at 2:53 AM, Usman Ali <us...@platalytics.com>
> > wrote:
> >
> > > I understand creating views and querying on them. But my problem is, I
> > don
> > > not have schema of the table, therefore I can not create views on the
> > > table. The only thing I know about the table is that each column
> > contains a
> > > "utf-8" encoded string. Now what I want to do to is to query the table
> > (not
> > > the view) through Restful APIs and get column values in bytes in my
> > > response json. Then I want to convert those byte values to my original
> > > string. For eample my rresponse json is :
> > > {
> > > myFamily: "{ "name" : "name string in bytes" , "address" :
> > "address
> > > string in bytes"}'
> > >
> > > }
> > >
> > > Now , I want to convert byte values of name and address to original
> > string.
> > > I hope I am clear in stating my problem.
> > > Any other work around this problem is also welcome.
> > >
> > > Thanks,
> > > Usman Ali
> > >
> > >
> > > On Sat, Aug 1, 2015 at 8:12 AM, Jacques Nadeau <ja...@dremio.com>
> > wrote:
> > >
> > > > Got it. Was just concerned that the person might be confused by your
> > > > example using CAST when we were talking about CONVERT. I think we
> need
> > > to
> > > > come up with a simpler/clearer way to express the differences since
> > > people
> > > > keep tripping up on this.
> > > >
> > > > --
> > > > Jacques Nadeau
> > > > CTO and Co-Founder, Dremio
> > > >
> > > > On Fri, Jul 31, 2015 at 7:59 PM, Carol McDonald <
> > cmcdonald@maprtech.com>
> > > > wrote:
> > > >
> > > > > the point I was trying to make is :
> > > > > The good news is, Drill does provide a nice simple way to abstract
> > > these
> > > > > details away. You simply create a view on top of HBase [1]. The
> > view
> > > > can
> > > > > contain the physical conversions. Then users can interact with the
> > > view
> > > > > rather than the underlying table.
> > > > >
> > > > > On Fri, Jul 31, 2015 at 11:36 AM, Jacques Nadeau <
> jacques@dremio.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Carol, your statements are misleading. There are some situations
> > > where
> > > > > > people have stored in HBase encoded as text. In other cases,
> > they've
> > > > > used
> > > > > > a straight binary encoding. CAST is used generally used when the
> > > data
> > > > is
> > > > > > already encoded as a UTF8 string. Otherwise, the user should use
> > > > > > CONVERT_FROM to decode the binary value.
> > > > > >
> > > > > > --
> > > > > > Jacques Nadeau
> > > > > > CTO and Co-Founder, Dremio
> > > > > >
> > > > > > On Fri, Jul 31, 2015 at 7:42 AM, Carol McDonald <
> > > > cmcdonald@maprtech.com>
> > > > > > wrote:
> > > > > >
> > > > > > > This blog shows an example using REST , drill , and HBase. In
> > this
> > > > > > example
> > > > > > > a view converts the HBase byte values to strings
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.mapr.com/blog/how-use-sql-hadoop-drill-rest-json-nosql-and-hbase-simple-rest-client
> > > > > > >
> > > > > > > 0: jdbc:drill:> use dfs.mydata;
> > > > > > >
> > > > > > > create or replace view prodview as SELECT CAST(row_key AS
> > INTEGER)
> > > AS
> > > > > > > prod_id, CAST(t.details.category AS VARCHAR(40)) AS category,
> > > > > > > CAST(t.details.name AS VARCHAR(129)) AS name FROM
> > maprdb.products
> > > t;
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jul 28, 2015 at 11:35 AM, Usman Ali <
> > > > usman.ali@platalytics.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > > When we query an Hbase table like "select * from
> > > > > > > > hbasePluginName.`tableName`" using Restful APIs, the json it
> > > > returns
> > > > > > > > contains values in the form of bytes. I know that all the
> > values
> > > > in
> > > > > my
> > > > > > > > hbase table were originally of String form but I do not know
> > > exact
> > > > > > schema
> > > > > > > > (column families and column names) of my table. How can I
> > convert
> > > > > those
> > > > > > > > bytes stored in strings back to my original strings.
> > > > > > > >
> > > > > > > > Thanks.
> > > > > > > > Usman Ali
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
Re: Convert Hbase Values from Bytes to Actual Strings
Posted by Ted Dunning <te...@gmail.com>.
On Sat, Aug 1, 2015 at 5:53 PM, John Omernik <jo...@omernik.com> wrote:
> I am working with this as well and I find the "abstraction" frustrating
> from a user perspective as well.
Yes. It can be a pain.
> I understand we can create views, I've
> done that, but it's a tedious process of ensuring my syntax is correct and
> one that I did have the Schema on hand to easily do this. Could a feature
> be added to "register" a view doing this for us based on the schema" ?
It should be relatively easy to build a tool to convert schema information
that you have into a view definition.
> Are
> we suffering performance hits when we do queries on 60+ columns?
No. Extra information referenced in the view is pruned away if it isn't
directly in your query so you don't pay any penalty for excess complexity
described in the view.
> This
> would make this integration much more intuitive for users who know sql,
> know their data is in an hbase table, and just want to go query. (Trying to
> make this more self serve for users)
>
Yes. A GUI would be a nice-to-have here. It isn't clear if this will be
high enough priority to happen right away.
>
> John
>
> On Sat, Aug 1, 2015 at 2:53 AM, Usman Ali <us...@platalytics.com>
> wrote:
>
> > I understand creating views and querying on them. But my problem is, I
> don
> > not have schema of the table, therefore I can not create views on the
> > table. The only thing I know about the table is that each column
> contains a
> > "utf-8" encoded string. Now what I want to do to is to query the table
> (not
> > the view) through Restful APIs and get column values in bytes in my
> > response json. Then I want to convert those byte values to my original
> > string. For eample my rresponse json is :
> > {
> > myFamily: "{ "name" : "name string in bytes" , "address" :
> "address
> > string in bytes"}'
> >
> > }
> >
> > Now , I want to convert byte values of name and address to original
> string.
> > I hope I am clear in stating my problem.
> > Any other work around this problem is also welcome.
> >
> > Thanks,
> > Usman Ali
> >
> >
> > On Sat, Aug 1, 2015 at 8:12 AM, Jacques Nadeau <ja...@dremio.com>
> wrote:
> >
> > > Got it. Was just concerned that the person might be confused by your
> > > example using CAST when we were talking about CONVERT. I think we need
> > to
> > > come up with a simpler/clearer way to express the differences since
> > people
> > > keep tripping up on this.
> > >
> > > --
> > > Jacques Nadeau
> > > CTO and Co-Founder, Dremio
> > >
> > > On Fri, Jul 31, 2015 at 7:59 PM, Carol McDonald <
> cmcdonald@maprtech.com>
> > > wrote:
> > >
> > > > the point I was trying to make is :
> > > > The good news is, Drill does provide a nice simple way to abstract
> > these
> > > > details away. You simply create a view on top of HBase [1]. The
> view
> > > can
> > > > contain the physical conversions. Then users can interact with the
> > view
> > > > rather than the underlying table.
> > > >
> > > > On Fri, Jul 31, 2015 at 11:36 AM, Jacques Nadeau <jacques@dremio.com
> >
> > > > wrote:
> > > >
> > > > > Carol, your statements are misleading. There are some situations
> > where
> > > > > people have stored in HBase encoded as text. In other cases,
> they've
> > > > used
> > > > > a straight binary encoding. CAST is used generally used when the
> > data
> > > is
> > > > > already encoded as a UTF8 string. Otherwise, the user should use
> > > > > CONVERT_FROM to decode the binary value.
> > > > >
> > > > > --
> > > > > Jacques Nadeau
> > > > > CTO and Co-Founder, Dremio
> > > > >
> > > > > On Fri, Jul 31, 2015 at 7:42 AM, Carol McDonald <
> > > cmcdonald@maprtech.com>
> > > > > wrote:
> > > > >
> > > > > > This blog shows an example using REST , drill , and HBase. In
> this
> > > > > example
> > > > > > a view converts the HBase byte values to strings
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.mapr.com/blog/how-use-sql-hadoop-drill-rest-json-nosql-and-hbase-simple-rest-client
> > > > > >
> > > > > > 0: jdbc:drill:> use dfs.mydata;
> > > > > >
> > > > > > create or replace view prodview as SELECT CAST(row_key AS
> INTEGER)
> > AS
> > > > > > prod_id, CAST(t.details.category AS VARCHAR(40)) AS category,
> > > > > > CAST(t.details.name AS VARCHAR(129)) AS name FROM
> maprdb.products
> > t;
> > > > > >
> > > > > >
> > > > > > On Tue, Jul 28, 2015 at 11:35 AM, Usman Ali <
> > > usman.ali@platalytics.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > > When we query an Hbase table like "select * from
> > > > > > > hbasePluginName.`tableName`" using Restful APIs, the json it
> > > returns
> > > > > > > contains values in the form of bytes. I know that all the
> values
> > > in
> > > > my
> > > > > > > hbase table were originally of String form but I do not know
> > exact
> > > > > schema
> > > > > > > (column families and column names) of my table. How can I
> convert
> > > > those
> > > > > > > bytes stored in strings back to my original strings.
> > > > > > >
> > > > > > > Thanks.
> > > > > > > Usman Ali
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Convert Hbase Values from Bytes to Actual Strings
Posted by John Omernik <jo...@omernik.com>.
I am working with this as well and I find the "abstraction" frustrating
from a user perspective as well. I understand we can create views, I've
done that, but it's a tedious process of ensuring my syntax is correct and
one that I did have the Schema on hand to easily do this. Could a feature
be added to "register" a view doing this for us based on the schema" Are
we suffering performance hits when we do queries on 60+ columns? This
would make this integration much more intuitive for users who know sql,
know their data is in an hbase table, and just want to go query. (Trying to
make this more self serve for users)
John
On Sat, Aug 1, 2015 at 2:53 AM, Usman Ali <us...@platalytics.com> wrote:
> I understand creating views and querying on them. But my problem is, I don
> not have schema of the table, therefore I can not create views on the
> table. The only thing I know about the table is that each column contains a
> "utf-8" encoded string. Now what I want to do to is to query the table (not
> the view) through Restful APIs and get column values in bytes in my
> response json. Then I want to convert those byte values to my original
> string. For eample my rresponse json is :
> {
> myFamily: "{ "name" : "name string in bytes" , "address" : "address
> string in bytes"}'
>
> }
>
> Now , I want to convert byte values of name and address to original string.
> I hope I am clear in stating my problem.
> Any other work around this problem is also welcome.
>
> Thanks,
> Usman Ali
>
>
> On Sat, Aug 1, 2015 at 8:12 AM, Jacques Nadeau <ja...@dremio.com> wrote:
>
> > Got it. Was just concerned that the person might be confused by your
> > example using CAST when we were talking about CONVERT. I think we need
> to
> > come up with a simpler/clearer way to express the differences since
> people
> > keep tripping up on this.
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Fri, Jul 31, 2015 at 7:59 PM, Carol McDonald <cm...@maprtech.com>
> > wrote:
> >
> > > the point I was trying to make is :
> > > The good news is, Drill does provide a nice simple way to abstract
> these
> > > details away. You simply create a view on top of HBase [1]. The view
> > can
> > > contain the physical conversions. Then users can interact with the
> view
> > > rather than the underlying table.
> > >
> > > On Fri, Jul 31, 2015 at 11:36 AM, Jacques Nadeau <ja...@dremio.com>
> > > wrote:
> > >
> > > > Carol, your statements are misleading. There are some situations
> where
> > > > people have stored in HBase encoded as text. In other cases, they've
> > > used
> > > > a straight binary encoding. CAST is used generally used when the
> data
> > is
> > > > already encoded as a UTF8 string. Otherwise, the user should use
> > > > CONVERT_FROM to decode the binary value.
> > > >
> > > > --
> > > > Jacques Nadeau
> > > > CTO and Co-Founder, Dremio
> > > >
> > > > On Fri, Jul 31, 2015 at 7:42 AM, Carol McDonald <
> > cmcdonald@maprtech.com>
> > > > wrote:
> > > >
> > > > > This blog shows an example using REST , drill , and HBase. In this
> > > > example
> > > > > a view converts the HBase byte values to strings
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://www.mapr.com/blog/how-use-sql-hadoop-drill-rest-json-nosql-and-hbase-simple-rest-client
> > > > >
> > > > > 0: jdbc:drill:> use dfs.mydata;
> > > > >
> > > > > create or replace view prodview as SELECT CAST(row_key AS INTEGER)
> AS
> > > > > prod_id, CAST(t.details.category AS VARCHAR(40)) AS category,
> > > > > CAST(t.details.name AS VARCHAR(129)) AS name FROM maprdb.products
> t;
> > > > >
> > > > >
> > > > > On Tue, Jul 28, 2015 at 11:35 AM, Usman Ali <
> > usman.ali@platalytics.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > > When we query an Hbase table like "select * from
> > > > > > hbasePluginName.`tableName`" using Restful APIs, the json it
> > returns
> > > > > > contains values in the form of bytes. I know that all the values
> > in
> > > my
> > > > > > hbase table were originally of String form but I do not know
> exact
> > > > schema
> > > > > > (column families and column names) of my table. How can I convert
> > > those
> > > > > > bytes stored in strings back to my original strings.
> > > > > >
> > > > > > Thanks.
> > > > > > Usman Ali
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Convert Hbase Values from Bytes to Actual Strings
Posted by Usman Ali <us...@platalytics.com>.
I understand creating views and querying on them. But my problem is, I don
not have schema of the table, therefore I can not create views on the
table. The only thing I know about the table is that each column contains a
"utf-8" encoded string. Now what I want to do to is to query the table (not
the view) through Restful APIs and get column values in bytes in my
response json. Then I want to convert those byte values to my original
string. For eample my rresponse json is :
{
myFamily: "{ "name" : "name string in bytes" , "address" : "address
string in bytes"}'
}
Now , I want to convert byte values of name and address to original string.
I hope I am clear in stating my problem.
Any other work around this problem is also welcome.
Thanks,
Usman Ali
On Sat, Aug 1, 2015 at 8:12 AM, Jacques Nadeau <ja...@dremio.com> wrote:
> Got it. Was just concerned that the person might be confused by your
> example using CAST when we were talking about CONVERT. I think we need to
> come up with a simpler/clearer way to express the differences since people
> keep tripping up on this.
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Fri, Jul 31, 2015 at 7:59 PM, Carol McDonald <cm...@maprtech.com>
> wrote:
>
> > the point I was trying to make is :
> > The good news is, Drill does provide a nice simple way to abstract these
> > details away. You simply create a view on top of HBase [1]. The view
> can
> > contain the physical conversions. Then users can interact with the view
> > rather than the underlying table.
> >
> > On Fri, Jul 31, 2015 at 11:36 AM, Jacques Nadeau <ja...@dremio.com>
> > wrote:
> >
> > > Carol, your statements are misleading. There are some situations where
> > > people have stored in HBase encoded as text. In other cases, they've
> > used
> > > a straight binary encoding. CAST is used generally used when the data
> is
> > > already encoded as a UTF8 string. Otherwise, the user should use
> > > CONVERT_FROM to decode the binary value.
> > >
> > > --
> > > Jacques Nadeau
> > > CTO and Co-Founder, Dremio
> > >
> > > On Fri, Jul 31, 2015 at 7:42 AM, Carol McDonald <
> cmcdonald@maprtech.com>
> > > wrote:
> > >
> > > > This blog shows an example using REST , drill , and HBase. In this
> > > example
> > > > a view converts the HBase byte values to strings
> > > >
> > > >
> > > >
> > >
> >
> https://www.mapr.com/blog/how-use-sql-hadoop-drill-rest-json-nosql-and-hbase-simple-rest-client
> > > >
> > > > 0: jdbc:drill:> use dfs.mydata;
> > > >
> > > > create or replace view prodview as SELECT CAST(row_key AS INTEGER) AS
> > > > prod_id, CAST(t.details.category AS VARCHAR(40)) AS category,
> > > > CAST(t.details.name AS VARCHAR(129)) AS name FROM maprdb.products t;
> > > >
> > > >
> > > > On Tue, Jul 28, 2015 at 11:35 AM, Usman Ali <
> usman.ali@platalytics.com
> > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > > When we query an Hbase table like "select * from
> > > > > hbasePluginName.`tableName`" using Restful APIs, the json it
> returns
> > > > > contains values in the form of bytes. I know that all the values
> in
> > my
> > > > > hbase table were originally of String form but I do not know exact
> > > schema
> > > > > (column families and column names) of my table. How can I convert
> > those
> > > > > bytes stored in strings back to my original strings.
> > > > >
> > > > > Thanks.
> > > > > Usman Ali
> > > > >
> > > >
> > >
> >
>
Re: Convert Hbase Values from Bytes to Actual Strings
Posted by Jacques Nadeau <ja...@dremio.com>.
Got it. Was just concerned that the person might be confused by your
example using CAST when we were talking about CONVERT. I think we need to
come up with a simpler/clearer way to express the differences since people
keep tripping up on this.
--
Jacques Nadeau
CTO and Co-Founder, Dremio
On Fri, Jul 31, 2015 at 7:59 PM, Carol McDonald <cm...@maprtech.com>
wrote:
> the point I was trying to make is :
> The good news is, Drill does provide a nice simple way to abstract these
> details away. You simply create a view on top of HBase [1]. The view can
> contain the physical conversions. Then users can interact with the view
> rather than the underlying table.
>
> On Fri, Jul 31, 2015 at 11:36 AM, Jacques Nadeau <ja...@dremio.com>
> wrote:
>
> > Carol, your statements are misleading. There are some situations where
> > people have stored in HBase encoded as text. In other cases, they've
> used
> > a straight binary encoding. CAST is used generally used when the data is
> > already encoded as a UTF8 string. Otherwise, the user should use
> > CONVERT_FROM to decode the binary value.
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Fri, Jul 31, 2015 at 7:42 AM, Carol McDonald <cm...@maprtech.com>
> > wrote:
> >
> > > This blog shows an example using REST , drill , and HBase. In this
> > example
> > > a view converts the HBase byte values to strings
> > >
> > >
> > >
> >
> https://www.mapr.com/blog/how-use-sql-hadoop-drill-rest-json-nosql-and-hbase-simple-rest-client
> > >
> > > 0: jdbc:drill:> use dfs.mydata;
> > >
> > > create or replace view prodview as SELECT CAST(row_key AS INTEGER) AS
> > > prod_id, CAST(t.details.category AS VARCHAR(40)) AS category,
> > > CAST(t.details.name AS VARCHAR(129)) AS name FROM maprdb.products t;
> > >
> > >
> > > On Tue, Jul 28, 2015 at 11:35 AM, Usman Ali <usman.ali@platalytics.com
> >
> > > wrote:
> > >
> > > > Hi,
> > > > When we query an Hbase table like "select * from
> > > > hbasePluginName.`tableName`" using Restful APIs, the json it returns
> > > > contains values in the form of bytes. I know that all the values in
> my
> > > > hbase table were originally of String form but I do not know exact
> > schema
> > > > (column families and column names) of my table. How can I convert
> those
> > > > bytes stored in strings back to my original strings.
> > > >
> > > > Thanks.
> > > > Usman Ali
> > > >
> > >
> >
>