You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by prasadns14 <gi...@git.apache.org> on 2017/10/20 23:31:17 UTC
[GitHub] drill pull request #1005: DRILL-5896: Handle HBase columns vector creation i...
GitHub user prasadns14 opened a pull request:
https://github.com/apache/drill/pull/1005
DRILL-5896: Handle HBase columns vector creation in the HBaseRecordReader
Handling vector creation for projected hbase columns, by keeping track of column families and columns separately.
@paul-rogers please review
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/prasadns14/drill DRILL-5896
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/1005.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1005
----
commit 0f61a2dcd955d30d52103be0454fa45089eee064
Author: Prasad Nagaraj Subramanya <pr...@gmail.com>
Date: 2017-10-20T21:29:01Z
DRILL-5896: Handle HBase columns vector creation in the HBaseRecordReader
----
---
[GitHub] drill pull request #1005: DRILL-5896: Handle HBase columns vector creation i...
Posted by prasadns14 <gi...@git.apache.org>.
Github user prasadns14 commented on a diff in the pull request:
https://github.com/apache/drill/pull/1005#discussion_r147042924
--- Diff: contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseRecordReader.java ---
@@ -186,6 +192,10 @@ public void setup(OperatorContext context, OutputMutator output) throws Executio
}
}
}
+
+ for (String familyName : completeFamilies) {
+ getOrCreateFamilyVector(familyName, false);
+ }
--- End diff --
It creates only the map vector
---
[GitHub] drill pull request #1005: DRILL-5896: Handle HBase columns vector creation i...
Posted by prasadns14 <gi...@git.apache.org>.
Github user prasadns14 commented on a diff in the pull request:
https://github.com/apache/drill/pull/1005#discussion_r147042347
--- Diff: contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseRecordReader.java ---
@@ -75,6 +75,8 @@
private TableName hbaseTableName;
private Scan hbaseScan;
+ private Scan hbaseScan1;
+ Set<String> completeFamilies;
--- End diff --
Fixed
---
[GitHub] drill pull request #1005: DRILL-5896: Handle HBase columns vector creation i...
Posted by paul-rogers <gi...@git.apache.org>.
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/1005#discussion_r146659105
--- Diff: contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseRecordReader.java ---
@@ -121,16 +125,18 @@ public HBaseRecordReader(Connection connection, HBaseSubScan.HBaseSubScanSpec su
byte[] family = root.getPath().getBytes();
transformed.add(SchemaPath.getSimplePath(root.getPath()));
PathSegment child = root.getChild();
- if (!completeFamilies.contains(new String(family, StandardCharsets.UTF_8).toLowerCase())) {
- if (child != null && child.isNamed()) {
- byte[] qualifier = child.getNameSegment().getPath().getBytes();
+ if (child != null && child.isNamed()) {
+ byte[] qualifier = child.getNameSegment().getPath().getBytes();
+ hbaseScan1.addColumn(family, qualifier);
+ if (!completeFamilies.contains(new String(family, StandardCharsets.UTF_8))) {
--- End diff --
Redundant conversion of `family` to `String`, here and below.
---
[GitHub] drill pull request #1005: DRILL-5896: Handle HBase columns vector creation i...
Posted by paul-rogers <gi...@git.apache.org>.
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/1005#discussion_r146659717
--- Diff: contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseRecordReader.java ---
@@ -186,6 +192,10 @@ public void setup(OperatorContext context, OutputMutator output) throws Executio
}
}
}
+
+ for (String familyName : completeFamilies) {
+ getOrCreateFamilyVector(familyName, false);
+ }
--- End diff --
Does this create just the map, or also the vectors within the map? Maybe a comment to explain the goals?
---
[GitHub] drill pull request #1005: DRILL-5896: Handle HBase columns vector creation i...
Posted by paul-rogers <gi...@git.apache.org>.
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/1005#discussion_r146658667
--- Diff: contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseRecordReader.java ---
@@ -75,6 +75,8 @@
private TableName hbaseTableName;
private Scan hbaseScan;
+ private Scan hbaseScan1;
+ Set<String> completeFamilies;
--- End diff --
`private`?
---
[GitHub] drill pull request #1005: DRILL-5896: Handle HBase columns vector creation i...
Posted by prasadns14 <gi...@git.apache.org>.
Github user prasadns14 commented on a diff in the pull request:
https://github.com/apache/drill/pull/1005#discussion_r147042356
--- Diff: contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseRecordReader.java ---
@@ -87,6 +89,7 @@ public HBaseRecordReader(Connection connection, HBaseSubScan.HBaseSubScanSpec su
hbaseTableName = TableName.valueOf(
Preconditions.checkNotNull(subScanSpec, "HBase reader needs a sub-scan spec").getTableName());
hbaseScan = new Scan(subScanSpec.getStartRow(), subScanSpec.getStopRow());
+ hbaseScan1 = new Scan();
--- End diff --
Fixed
---
[GitHub] drill pull request #1005: DRILL-5896: Handle HBase columns vector creation i...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/drill/pull/1005
---
[GitHub] drill pull request #1005: DRILL-5896: Handle HBase columns vector creation i...
Posted by prasadns14 <gi...@git.apache.org>.
Github user prasadns14 commented on a diff in the pull request:
https://github.com/apache/drill/pull/1005#discussion_r147042366
--- Diff: contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseRecordReader.java ---
@@ -121,16 +125,18 @@ public HBaseRecordReader(Connection connection, HBaseSubScan.HBaseSubScanSpec su
byte[] family = root.getPath().getBytes();
transformed.add(SchemaPath.getSimplePath(root.getPath()));
PathSegment child = root.getChild();
- if (!completeFamilies.contains(new String(family, StandardCharsets.UTF_8).toLowerCase())) {
- if (child != null && child.isNamed()) {
- byte[] qualifier = child.getNameSegment().getPath().getBytes();
+ if (child != null && child.isNamed()) {
+ byte[] qualifier = child.getNameSegment().getPath().getBytes();
+ hbaseScan1.addColumn(family, qualifier);
+ if (!completeFamilies.contains(new String(family, StandardCharsets.UTF_8))) {
--- End diff --
Fixed
---
[GitHub] drill issue #1005: DRILL-5896: Handle HBase columns vector creation in the H...
Posted by prasadns14 <gi...@git.apache.org>.
Github user prasadns14 commented on the issue:
https://github.com/apache/drill/pull/1005
@paul-rogers please review the changes
---
[GitHub] drill pull request #1005: DRILL-5896: Handle HBase columns vector creation i...
Posted by paul-rogers <gi...@git.apache.org>.
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/1005#discussion_r146658771
--- Diff: contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseRecordReader.java ---
@@ -87,6 +89,7 @@ public HBaseRecordReader(Connection connection, HBaseSubScan.HBaseSubScanSpec su
hbaseTableName = TableName.valueOf(
Preconditions.checkNotNull(subScanSpec, "HBase reader needs a sub-scan spec").getTableName());
hbaseScan = new Scan(subScanSpec.getStartRow(), subScanSpec.getStopRow());
+ hbaseScan1 = new Scan();
--- End diff --
Better name or comment to explain.
---