You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by "Lisen Mu (JIRA)" <ji...@apache.org> on 2013/08/13 08:52:49 UTC
[jira] [Updated] (DRILL-15) Build HBase storage engine
implementation
[ https://issues.apache.org/jira/browse/DRILL-15?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lisen Mu updated DRILL-15:
--------------------------
Attachment: DRILL-15.metastore.patch.txt
cut related part form our code base. this patch is pretty far from ready-to-merge state, but I guess it won't be merged anyway:) just show the idea and the problem.
We are using hive metastore to represent data schema information of HTables. Metastore has 3 kind of information:
* WHAT the logical fields look like. From user's perspective, a record may have different fields like integer, date, boolean or text etc. This also indicates how drill would process fields in memory.
* WHERE the logical fields is stored in HTable. there are many places in HTable in which information can be stored: rowkey; qualifier name of a particular CF; value under a particular CF:qualifier; or even version number of a particular cell. The value of a logical field can be stored as any of above. Further, rowkey may contain multiple logical fields. This highly depends on how user design their storage schema.
* HOW logical fields is stored. HBase basically provides a storage for byte[]. So HTable scanner needs to know how the fields like integer, date, boolean are serialized as byte[]. For example, 255 would be serialized to \xFF as BINARY:1byte, or [FF 00 00 00] as BINARY:4byte, or "255" as TEXT(with variable length), or "00000255" as TEXT(with fixed length:8). Another example would be logical DATE to (first integer 1375483564 then) [AC 36 FC 51] as BINARY:4byte, or "20130803" as TEXT(with fixed length:8).
The meta definition is in com.xingcloud.meta.HBaseFieldInfo.java
These information will be used in HBase scanner to generate most effective scan (mapping logical Filter to HBase's filter class, and deciding startKey and endKey to scan least data), and in conversion from LogicalPlan to PhysicalPlan, to generate the correct ReadEntry for HBase.
I do understand that strong schema is not drill's primary concern, however I think other approaches to HBase scanner also have to solve the problems above to work correctly.
> Build HBase storage engine implementation
> -----------------------------------------
>
> Key: DRILL-15
> URL: https://issues.apache.org/jira/browse/DRILL-15
> Project: Apache Drill
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: David Alves
> Attachments: DRILL-15-02.patch, DRILL-15-03.patch, DRILL-15.metastore.patch.txt, DRILL-15.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira