You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by anil gupta <an...@gmail.com> on 2015/08/05 09:51:48 UTC
Re: Disable Base64 encoding in Stargate request and Return as String
Hi Andrew,
Thanks for sharing your thoughts. Sorry for late reply as i recently came
back from vacation.
I understand that HBase stores byte arrays, so its hard for HBase to figure
out the data type.
What if, the client knows that all the columns in the Rest request are
Strings. In that case, can we give the option of setting a request header
"StringDecoding:True". By default, we can assume "StringDecoding: false".
Just some food for thought.
Also, if we can replicate the Encoding that we do in HBase Shell(where
string are shown in readable format and we hex encode all binary data).
That would be best. In this case, it would be really convenient use of Rest
service rather than invoking "hbase shell". Right now, IMO, due to lack of
readability its only good to fetch images.(we store images in HBase)
Provided my employer allows me to contribute, I am willing to work on this.
Would HBase accept a patch?
Thanks,
Anil Gupta
On Fri, Jul 17, 2015 at 4:57 PM, Andrew Purtell <ap...@apache.org> wrote:
>
>
> The closest you can get to just a string is have your client use an accept
> header of "Accept: application/octet-stream" with making a query. This will
> return zero or one value in the response. If a value is present in the
> table at the requested location, the response body will be the unencoded
> bytes. If you've stored a string, you'll get back a string. If you've
> stored an image, you'll get back the raw image bytes. Note that using an
> accept header of "application/octet-stream" implicitly limits you to
> queries that only return zero or one values. (Strictly speaking, per the
> package doc: "If binary encoding is requested, only one cell can be
> returned, the first to match the resource specification. The row, column,
> and timestamp associated with the cell will be transmitted in X headers:
> X-Row, X-Column, and X-Timestamp, respectively. Depending on the precision
> of the resource specification, some of the X-headers may be elided as
> redundant.")
>
> In general, the REST gateway supports
>
> several
> alternate encodings. See
>
> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/rest/package-summary.html
> for some examples.
>
> Note that HBase
> cell data
> is binary
> , not string.
> It
> does not
> make sense to turn off base64 encoding for the default response encoding,
> XML, because
> that
> would produce invalid XML
> if a value happens to include non XML safe bytes
> .
> HBase can't know that in advance. We need to encode keys and values in a
> safe manner to avoid blowing up your
> client's XML.
>
> The same is roughly true for JSON.
>
>
> If your client sends an accept header of "Accept: application/protobuf"
> you'll get back a protobuf encoded object. Your client will need to be
> prepared to handle that representation. This is probably not what you want.
>
> Why are we
> even
> talking about using XML
> , JSON,
> or
> protobuf to
> encode
> responses? Because for many types of REST queries, HBase
> must return
> a structured response.
> The client has asked for more than
> simply
> one value, simply one string
> . The response
> must include
> key
> s
> ,
> values
> ,
> timestamps
> ;
> maybe a whole row
> 's worth
> of
> keys, values, and timestamps
> ;
> maybe multiple rows. It depends on the query you issued.
> (See the '
> Cell or Row Query (Multiple Values)
> ' section in the package doc.)
>
>
>
>
> On Fri, Jul 17, 2015 at 2:20 PM, anil gupta <an...@gmail.com> wrote:
>
> > Hi All,
> >
> > We have a String Rowkey. We have String values of cells.
> > Still, Stargate returns the data with Base64 encoding due to which a user
> > cant read the data. Is there a way to disable Base64 encoding and then
> Rest
> > request would just return Strings.
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
--
Thanks & Regards,
Anil Gupta
Re: Disable Base64 encoding in Stargate request and Return as String
Posted by anil gupta <an...@gmail.com>.
Thanks Andrew. I didnt to change behavior of hbase shell. I intend to
provide an enhancement to HBase Rest while not impacting its default
behavior.
On Thu, Aug 6, 2015 at 5:29 PM, Andrew Purtell <ap...@apache.org> wrote:
> > returned from the shell
>
> Meant "returned from the REST gateway".
>
>
> On Thu, Aug 6, 2015 at 5:28 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > Unfortunately we can't change the current set of representations are
> > returned from the shell, that would be a backwards compatibility problem.
> > We can however add new representations (selectable by way of the Accept
> > header, e.g. Accept: text/plain). If you'd like to propose a patch we'd
> > certainly look at it.
> >
> > Thanks.
> >
> >
> > On Wed, Aug 5, 2015 at 12:51 AM, anil gupta <an...@gmail.com>
> wrote:
> >
> >> Hi Andrew,
> >>
> >> Thanks for sharing your thoughts. Sorry for late reply as i recently
> came
> >> back from vacation.
> >> I understand that HBase stores byte arrays, so its hard for HBase to
> >> figure
> >> out the data type.
> >> What if, the client knows that all the columns in the Rest request are
> >> Strings. In that case, can we give the option of setting a request
> header
> >> "StringDecoding:True". By default, we can assume "StringDecoding:
> false".
> >> Just some food for thought.
> >>
> >> Also, if we can replicate the Encoding that we do in HBase Shell(where
> >> string are shown in readable format and we hex encode all binary data).
> >> That would be best. In this case, it would be really convenient use of
> >> Rest
> >> service rather than invoking "hbase shell". Right now, IMO, due to lack
> of
> >> readability its only good to fetch images.(we store images in HBase)
> >>
> >> Provided my employer allows me to contribute, I am willing to work on
> >> this.
> >> Would HBase accept a patch?
> >>
> >> Thanks,
> >> Anil Gupta
> >>
> >> On Fri, Jul 17, 2015 at 4:57 PM, Andrew Purtell <ap...@apache.org>
> >> wrote:
> >>
> >> >
> >> >
> >> > The closest you can get to just a string is have your client use an
> >> accept
> >> > header of "Accept: application/octet-stream" with making a query. This
> >> will
> >> > return zero or one value in the response. If a value is present in the
> >> > table at the requested location, the response body will be the
> unencoded
> >> > bytes. If you've stored a string, you'll get back a string. If you've
> >> > stored an image, you'll get back the raw image bytes. Note that using
> an
> >> > accept header of "application/octet-stream" implicitly limits you to
> >> > queries that only return zero or one values. (Strictly speaking, per
> the
> >> > package doc: "If binary encoding is requested, only one cell can be
> >> > returned, the first to match the resource specification. The row,
> >> column,
> >> > and timestamp associated with the cell will be transmitted in X
> headers:
> >> > X-Row, X-Column, and X-Timestamp, respectively. Depending on the
> >> precision
> >> > of the resource specification, some of the X-headers may be elided as
> >> > redundant.")
> >> >
> >> > In general, the REST gateway supports
> >> >
> >> > several
> >> > alternate encodings. See
> >> >
> >> >
> >>
> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/rest/package-summary.html
> >> > for some examples.
> >> >
> >> > Note that HBase
> >> > cell data
> >> > is binary
> >> > , not string.
> >> > It
> >> > does not
> >> > make sense to turn off base64 encoding for the default response
> >> encoding,
> >> > XML, because
> >> > that
> >> > would produce invalid XML
> >> > if a value happens to include non XML safe bytes
> >> > .
> >> > HBase can't know that in advance. We need to encode keys and values
> in
> >> a
> >> > safe manner to avoid blowing up your
> >> > client's XML.
> >> >
> >> > The same is roughly true for JSON.
> >> >
> >> >
> >> > If your client sends an accept header of "Accept:
> application/protobuf"
> >> > you'll get back a protobuf encoded object. Your client will need to be
> >> > prepared to handle that representation. This is probably not what you
> >> want.
> >> >
> >> > Why are we
> >> > even
> >> > talking about using XML
> >> > , JSON,
> >> > or
> >> > protobuf to
> >> > encode
> >> > responses? Because for many types of REST queries, HBase
> >> > must return
> >> > a structured response.
> >> > The client has asked for more than
> >> > simply
> >> > one value, simply one string
> >> > . The response
> >> > must include
> >> > key
> >> > s
> >> > ,
> >> > values
> >> > ,
> >> > timestamps
> >> > ;
> >> > maybe a whole row
> >> > 's worth
> >> > of
> >> > keys, values, and timestamps
> >> > ;
> >> > maybe multiple rows. It depends on the query you issued.
> >> > (See the '
> >> > Cell or Row Query (Multiple Values)
> >> > ' section in the package doc.)
> >> >
> >> >
> >> >
> >> >
> >> > On Fri, Jul 17, 2015 at 2:20 PM, anil gupta <an...@gmail.com>
> >> wrote:
> >> >
> >> > > Hi All,
> >> > >
> >> > > We have a String Rowkey. We have String values of cells.
> >> > > Still, Stargate returns the data with Base64 encoding due to which a
> >> user
> >> > > cant read the data. Is there a way to disable Base64 encoding and
> then
> >> > Rest
> >> > > request would just return Strings.
> >> > >
> >> > > --
> >> > > Thanks & Regards,
> >> > > Anil Gupta
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Best regards,
> >> >
> >> > - Andy
> >> >
> >> > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> >> > (via Tom White)
> >> >
> >>
> >>
> >>
> >> --
> >> Thanks & Regards,
> >> Anil Gupta
> >>
> >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
--
Thanks & Regards,
Anil Gupta
Re: Disable Base64 encoding in Stargate request and Return as String
Posted by Andrew Purtell <ap...@apache.org>.
> returned from the shell
Meant "returned from the REST gateway".
On Thu, Aug 6, 2015 at 5:28 PM, Andrew Purtell <ap...@apache.org> wrote:
> Unfortunately we can't change the current set of representations are
> returned from the shell, that would be a backwards compatibility problem.
> We can however add new representations (selectable by way of the Accept
> header, e.g. Accept: text/plain). If you'd like to propose a patch we'd
> certainly look at it.
>
> Thanks.
>
>
> On Wed, Aug 5, 2015 at 12:51 AM, anil gupta <an...@gmail.com> wrote:
>
>> Hi Andrew,
>>
>> Thanks for sharing your thoughts. Sorry for late reply as i recently came
>> back from vacation.
>> I understand that HBase stores byte arrays, so its hard for HBase to
>> figure
>> out the data type.
>> What if, the client knows that all the columns in the Rest request are
>> Strings. In that case, can we give the option of setting a request header
>> "StringDecoding:True". By default, we can assume "StringDecoding: false".
>> Just some food for thought.
>>
>> Also, if we can replicate the Encoding that we do in HBase Shell(where
>> string are shown in readable format and we hex encode all binary data).
>> That would be best. In this case, it would be really convenient use of
>> Rest
>> service rather than invoking "hbase shell". Right now, IMO, due to lack of
>> readability its only good to fetch images.(we store images in HBase)
>>
>> Provided my employer allows me to contribute, I am willing to work on
>> this.
>> Would HBase accept a patch?
>>
>> Thanks,
>> Anil Gupta
>>
>> On Fri, Jul 17, 2015 at 4:57 PM, Andrew Purtell <ap...@apache.org>
>> wrote:
>>
>> >
>> >
>> > The closest you can get to just a string is have your client use an
>> accept
>> > header of "Accept: application/octet-stream" with making a query. This
>> will
>> > return zero or one value in the response. If a value is present in the
>> > table at the requested location, the response body will be the unencoded
>> > bytes. If you've stored a string, you'll get back a string. If you've
>> > stored an image, you'll get back the raw image bytes. Note that using an
>> > accept header of "application/octet-stream" implicitly limits you to
>> > queries that only return zero or one values. (Strictly speaking, per the
>> > package doc: "If binary encoding is requested, only one cell can be
>> > returned, the first to match the resource specification. The row,
>> column,
>> > and timestamp associated with the cell will be transmitted in X headers:
>> > X-Row, X-Column, and X-Timestamp, respectively. Depending on the
>> precision
>> > of the resource specification, some of the X-headers may be elided as
>> > redundant.")
>> >
>> > In general, the REST gateway supports
>> >
>> > several
>> > alternate encodings. See
>> >
>> >
>> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/rest/package-summary.html
>> > for some examples.
>> >
>> > Note that HBase
>> > cell data
>> > is binary
>> > , not string.
>> > It
>> > does not
>> > make sense to turn off base64 encoding for the default response
>> encoding,
>> > XML, because
>> > that
>> > would produce invalid XML
>> > if a value happens to include non XML safe bytes
>> > .
>> > HBase can't know that in advance. We need to encode keys and values in
>> a
>> > safe manner to avoid blowing up your
>> > client's XML.
>> >
>> > The same is roughly true for JSON.
>> >
>> >
>> > If your client sends an accept header of "Accept: application/protobuf"
>> > you'll get back a protobuf encoded object. Your client will need to be
>> > prepared to handle that representation. This is probably not what you
>> want.
>> >
>> > Why are we
>> > even
>> > talking about using XML
>> > , JSON,
>> > or
>> > protobuf to
>> > encode
>> > responses? Because for many types of REST queries, HBase
>> > must return
>> > a structured response.
>> > The client has asked for more than
>> > simply
>> > one value, simply one string
>> > . The response
>> > must include
>> > key
>> > s
>> > ,
>> > values
>> > ,
>> > timestamps
>> > ;
>> > maybe a whole row
>> > 's worth
>> > of
>> > keys, values, and timestamps
>> > ;
>> > maybe multiple rows. It depends on the query you issued.
>> > (See the '
>> > Cell or Row Query (Multiple Values)
>> > ' section in the package doc.)
>> >
>> >
>> >
>> >
>> > On Fri, Jul 17, 2015 at 2:20 PM, anil gupta <an...@gmail.com>
>> wrote:
>> >
>> > > Hi All,
>> > >
>> > > We have a String Rowkey. We have String values of cells.
>> > > Still, Stargate returns the data with Base64 encoding due to which a
>> user
>> > > cant read the data. Is there a way to disable Base64 encoding and then
>> > Rest
>> > > request would just return Strings.
>> > >
>> > > --
>> > > Thanks & Regards,
>> > > Anil Gupta
>> > >
>> >
>> >
>> >
>> > --
>> > Best regards,
>> >
>> > - Andy
>> >
>> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> > (via Tom White)
>> >
>>
>>
>>
>> --
>> Thanks & Regards,
>> Anil Gupta
>>
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Disable Base64 encoding in Stargate request and Return as String
Posted by Andrew Purtell <ap...@apache.org>.
Unfortunately we can't change the current set of representations are
returned from the shell, that would be a backwards compatibility problem.
We can however add new representations (selectable by way of the Accept
header, e.g. Accept: text/plain). If you'd like to propose a patch we'd
certainly look at it.
Thanks.
On Wed, Aug 5, 2015 at 12:51 AM, anil gupta <an...@gmail.com> wrote:
> Hi Andrew,
>
> Thanks for sharing your thoughts. Sorry for late reply as i recently came
> back from vacation.
> I understand that HBase stores byte arrays, so its hard for HBase to figure
> out the data type.
> What if, the client knows that all the columns in the Rest request are
> Strings. In that case, can we give the option of setting a request header
> "StringDecoding:True". By default, we can assume "StringDecoding: false".
> Just some food for thought.
>
> Also, if we can replicate the Encoding that we do in HBase Shell(where
> string are shown in readable format and we hex encode all binary data).
> That would be best. In this case, it would be really convenient use of Rest
> service rather than invoking "hbase shell". Right now, IMO, due to lack of
> readability its only good to fetch images.(we store images in HBase)
>
> Provided my employer allows me to contribute, I am willing to work on this.
> Would HBase accept a patch?
>
> Thanks,
> Anil Gupta
>
> On Fri, Jul 17, 2015 at 4:57 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> >
> >
> > The closest you can get to just a string is have your client use an
> accept
> > header of "Accept: application/octet-stream" with making a query. This
> will
> > return zero or one value in the response. If a value is present in the
> > table at the requested location, the response body will be the unencoded
> > bytes. If you've stored a string, you'll get back a string. If you've
> > stored an image, you'll get back the raw image bytes. Note that using an
> > accept header of "application/octet-stream" implicitly limits you to
> > queries that only return zero or one values. (Strictly speaking, per the
> > package doc: "If binary encoding is requested, only one cell can be
> > returned, the first to match the resource specification. The row, column,
> > and timestamp associated with the cell will be transmitted in X headers:
> > X-Row, X-Column, and X-Timestamp, respectively. Depending on the
> precision
> > of the resource specification, some of the X-headers may be elided as
> > redundant.")
> >
> > In general, the REST gateway supports
> >
> > several
> > alternate encodings. See
> >
> >
> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/rest/package-summary.html
> > for some examples.
> >
> > Note that HBase
> > cell data
> > is binary
> > , not string.
> > It
> > does not
> > make sense to turn off base64 encoding for the default response
> encoding,
> > XML, because
> > that
> > would produce invalid XML
> > if a value happens to include non XML safe bytes
> > .
> > HBase can't know that in advance. We need to encode keys and values in a
> > safe manner to avoid blowing up your
> > client's XML.
> >
> > The same is roughly true for JSON.
> >
> >
> > If your client sends an accept header of "Accept: application/protobuf"
> > you'll get back a protobuf encoded object. Your client will need to be
> > prepared to handle that representation. This is probably not what you
> want.
> >
> > Why are we
> > even
> > talking about using XML
> > , JSON,
> > or
> > protobuf to
> > encode
> > responses? Because for many types of REST queries, HBase
> > must return
> > a structured response.
> > The client has asked for more than
> > simply
> > one value, simply one string
> > . The response
> > must include
> > key
> > s
> > ,
> > values
> > ,
> > timestamps
> > ;
> > maybe a whole row
> > 's worth
> > of
> > keys, values, and timestamps
> > ;
> > maybe multiple rows. It depends on the query you issued.
> > (See the '
> > Cell or Row Query (Multiple Values)
> > ' section in the package doc.)
> >
> >
> >
> >
> > On Fri, Jul 17, 2015 at 2:20 PM, anil gupta <an...@gmail.com>
> wrote:
> >
> > > Hi All,
> > >
> > > We have a String Rowkey. We have String values of cells.
> > > Still, Stargate returns the data with Base64 encoding due to which a
> user
> > > cant read the data. Is there a way to disable Base64 encoding and then
> > Rest
> > > request would just return Strings.
> > >
> > > --
> > > Thanks & Regards,
> > > Anil Gupta
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)