You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Yubao Liu (Jira)" <ji...@apache.org> on 2020/12/04 04:27:00 UTC
[jira] [Updated] (HBASE-25357) allow specifying binary row key
range to pre-split regions
[ https://issues.apache.org/jira/browse/HBASE-25357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yubao Liu updated HBASE-25357:
------------------------------
Description:
Currently, spark hbase connector use `String` to specify regionStart and regionEnd, but we often have serialized binary row key, I made a little patch at [https://github.com/apache/hbase-connectors/pull/72/files] to always treat the `String` in ISO_8859_1, so we can put raw bytes into the String object and get it unchanged.
This has a drawback, if your row key is really UTF-8 strings, you should convert it to UTF-8 encoded bytes and then encapsulate it in ISO_8859_1 string. This is a limitation of Spark option interface which allows only string to string map.
{code:java}
import java.nio.charset.StandardCharsets;
df.write()
.format("org.apache.hadoop.hbase.spark")
.option(HBaseTableCatalog.tableCatalog(), catalog)
.option(HBaseTableCatalog.newTable(), 5)
.option(HBaseTableCatalog.regionStart(), new String("你好".getBytes(StandardCharsets.UTF_8), StandardCharsets.ISO_8859_1))
.option(HBaseTableCatalog.regionEnd(), new String("世界".getBytes(StandardCharsets.UTF_8), StandardCharsets.ISO_8859_1))
.mode(SaveMode.Append)
.save();
{code}
was:
Currently, spark hbase connector use `String` to specify regionStart and regionEnd, but we often have serialized binary row key, I made a little patch at [https://github.com/apache/hbase-connectors/pull/72/files] to always treat the `String` in ISO_8859_1, so we can put raw bytes into the String object and get it unchanged.
This has a drawback, if your row key is really UTF-8 strings, you should convert it to UTF-8 encoded bytes and then encapsulate it in ISO_8859_1 string. This is a limitation of Spark option interface which allows only string to string map.
> allow specifying binary row key range to pre-split regions
> ----------------------------------------------------------
>
> Key: HBASE-25357
> URL: https://issues.apache.org/jira/browse/HBASE-25357
> Project: HBase
> Issue Type: Improvement
> Components: spark
> Reporter: Yubao Liu
> Priority: Major
>
> Currently, spark hbase connector use `String` to specify regionStart and regionEnd, but we often have serialized binary row key, I made a little patch at [https://github.com/apache/hbase-connectors/pull/72/files] to always treat the `String` in ISO_8859_1, so we can put raw bytes into the String object and get it unchanged.
> This has a drawback, if your row key is really UTF-8 strings, you should convert it to UTF-8 encoded bytes and then encapsulate it in ISO_8859_1 string. This is a limitation of Spark option interface which allows only string to string map.
> {code:java}
> import java.nio.charset.StandardCharsets;
> df.write()
> .format("org.apache.hadoop.hbase.spark")
> .option(HBaseTableCatalog.tableCatalog(), catalog)
> .option(HBaseTableCatalog.newTable(), 5)
> .option(HBaseTableCatalog.regionStart(), new String("你好".getBytes(StandardCharsets.UTF_8), StandardCharsets.ISO_8859_1))
> .option(HBaseTableCatalog.regionEnd(), new String("世界".getBytes(StandardCharsets.UTF_8), StandardCharsets.ISO_8859_1))
> .mode(SaveMode.Append)
> .save();
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)