You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by paul-rogers <gi...@git.apache.org> on 2017/09/30 21:17:51 UTC
[GitHub] drill pull request #968: DRILL-5830: Resolve regressions to MapR DB from DRI...
GitHub user paul-rogers opened a pull request:
https://github.com/apache/drill/pull/968
DRILL-5830: Resolve regressions to MapR DB from DRILL-5546
DRILL-5546 fixed a wide variety of "empty batch" problems. But, it introduced a regression in the HBase and MapR-DB binary storage plugins. This PR refines the fix to resolve those regressions.
Prior to DRILL-5546, HBase provided a project push-down rule to expand wildcard columns. However, a bug in the push-down rule prevented proper function. DRILL-5546 fixed that rule. But, DRILL-5546 also explicitly expanded wildcards for HBase, which turned out to be redundant, and so is backed out in this PR.
Wildcard expansion in the HBase storage plugin was meant to overcome the schema change conflict that occurs with empty regions. In such regions, we get the row key and the column family as an empty map. In regions with data, we get the row key and a non-empty map for the column family. Examples:
* Empty: (row_key, cf{})
* Non-empty: (row_key, cf{col1, col2})
Where cf is a column family and col1, col2 are columns.
It turns out that the receivers were getting confused. The `RecordBatchLoader` class treated empty and non-empty maps as an identical schema. This was mentioned in DRILL-5546:
> In HBase a column family always has map type, and a non-rowkey column always has nullable varbinary type, this ensures that HBaseRecordReader across different HBase regions will have the same top level schema, even if the region is empty or prune all the rows due to filter pushdown optimization. In other words, we will not see different top level schema from different HBaseRecordReader for the same table.
The problem is, a difference in map content really is a schema change, so we need to detect and report it. This PR makes that change.
Now, as it turns out, changes made by DRILL-5546 to the top-level project operator gracefully removes the empty batches (with empty maps), passing along just the non-empty batches (with non-empty maps.)
In short, this PR:
* Backs out the HBase-specific changes to DRILL-5546,
* Fixes the schema change issue in `RecordBatchLoader`,
* Adds unit tests for the fixes to `RecordBatchLoader`,
* Does a number of minor code cleanups.
The result is that the HBase problems that DRILL-5546 solved are still solved, but the regressions to MapR DB binary are also fixed.
We propose to modify the MapR DB binary storage plugin to do projection push-down the same way as is done in HBase, but that will be a separate PR.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/paul-rogers/drill DRILL-5830
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/968.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #968
----
commit 6cb5e98c6cccfc519cd7413248685ddc42af96b4
Author: Paul Rogers <pr...@maprtech.com>
Date: 2017-09-28T16:49:38Z
Back out HBase changes
commit 333bd1b36e72950d926c899b9050ddfae09fc817
Author: Paul Rogers <pr...@maprtech.com>
Date: 2017-09-30T01:14:55Z
Code cleanup
commit 8baca87708af2a38c5748a1a0435312e34e90903
Author: Paul Rogers <pr...@maprtech.com>
Date: 2017-09-30T01:18:16Z
Test utilities
commit 2ce7bf76dc37393b0326e23db99706f2abea7f5c
Author: Paul Rogers <pr...@maprtech.com>
Date: 2017-09-30T01:18:48Z
Fix for DRILL-5829
commit f660731df0168304456976e22687459b41d35546
Author: Paul Rogers <pr...@maprtech.com>
Date: 2017-09-30T20:58:22Z
Code cleanup
----
---
[GitHub] drill pull request #968: DRILL-5830: Resolve regressions to MapR DB from DRI...
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/968#discussion_r143714717
--- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatchLoader.java ---
@@ -150,6 +162,40 @@ public boolean load(RecordBatchDef def, DrillBuf buf) throws SchemaChangeExcepti
return schemaChanged;
}
+ private boolean checkMapSchema(Collection<MaterializedField> currentChildren,
--- End diff --
May be we can rename method? At least for me under `checkMapSchema` is not quite obvious that we will be comparing two schemas and return true if two maps are the same (comments inside of the method were helpful though).
---
[GitHub] drill pull request #968: DRILL-5830: Resolve regressions to MapR DB from DRI...
Posted by paul-rogers <gi...@git.apache.org>.
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/968#discussion_r144111856
--- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatchLoader.java ---
@@ -150,6 +162,40 @@ public boolean load(RecordBatchDef def, DrillBuf buf) throws SchemaChangeExcepti
return schemaChanged;
}
+ private boolean checkMapSchema(Collection<MaterializedField> currentChildren,
--- End diff --
Renamed and added Javadoc comments to better explain the purpose.
---
[GitHub] drill issue #968: DRILL-5830: Resolve regressions to MapR DB from DRILL-5546
Posted by jinfengni <gi...@git.apache.org>.
Github user jinfengni commented on the issue:
https://github.com/apache/drill/pull/968
I'm not convinced that it's a good idea to back out the change to HBase specific changes made in DRILL-5546.
You are right that project push-down in planner ideally should do its job and push the list of columns. However, until planner could claim there is no issues at all ( just like what happened prior to DRILL-5546), execution still may face a columns list with "*". That's why in HBaseGroupScan we have verifyColumnsAndConvertStar, just in case planner's project push-down did not work in the way we want. If as you suggested, such conversion in HBaseGroupScan is redundant, why would it cause regression (if planner's rule works as expected)? or is it your intention to still keep "*" in HBaseRecordReader? If that's your intention, I think we are going in the wrong direction. HBase has a unified schema at table level.
I agree that the analysis of empty map {}, vs {a:varbinary}. It's something we have to deal with. As a matter of fact, such scenarios does not have to come from empty batch. It could happen with two regions with >0 rows.
For instance, regrion 1 has 10 rows, with cf1.c1 appears in only first 5 rows, while region 2 has 20 rows with cf1.c1 appears in every rows. For the following query:
select CF1 FROM table where some_condition_on_row_key;
if "some_condition_on_row_key" is pushed to hbase and prunes the first 5 rows, region1 will return a batch with 5 rows, but with cf1 as an empty map, while region2 will have map with cf1 as {c1:varbinary}.
In that sense, DRILL-5546 exposes such issues, and force us to have a solution to handle empty map {} vs {a:varbinary}/
---
[GitHub] drill pull request #968: DRILL-5830: Resolve regressions to MapR DB from DRI...
Posted by paul-rogers <gi...@git.apache.org>.
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/968#discussion_r144114556
--- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/record/vector/TestLoad.java ---
@@ -119,7 +122,283 @@ public void testLoadValueVector() throws Exception {
}
}
assertEquals(100, recordCount);
+
+ // Free the original vectors
+
+ writableBatch.clear();
+
+ // Free the deserialized vectors
+
batchLoader.clear();
+
+ // The allocator will verify that the frees were done correctly.
+
+ allocator.close();
+ }
+
+ // TODO: Replace this low-level code with RowSet usage once
+ // DRILL-5657 is committed to master.
+
+ private static List<ValueVector> createVectors(BufferAllocator allocator, BatchSchema schema, int i) {
+ final List<ValueVector> vectors = new ArrayList<>();
+ for (MaterializedField field : schema) {
+ @SuppressWarnings("resource")
+ ValueVector v = TypeHelper.getNewVector(field, allocator);
+ AllocationHelper.allocate(v, 100, 50);
+ v.getMutator().generateTestData(100);
+ vectors.add(v);
+ }
+ return vectors;
+ }
+
+ private static DrillBuf serializeBatch(BufferAllocator allocator, WritableBatch writableBatch) {
+ final ByteBuf[] byteBufs = writableBatch.getBuffers();
+ int bytes = 0;
+ for (ByteBuf buf : byteBufs) {
+ bytes += buf.writerIndex();
+ }
+ final DrillBuf byteBuf = allocator.buffer(bytes);
+ int index = 0;
+ for (ByteBuf buf : byteBufs) {
+ buf.readBytes(byteBuf, index, buf.writerIndex());
+ index += buf.writerIndex();
+ }
+ byteBuf.writerIndex(bytes);
+ return byteBuf;
+ }
+
+ @SuppressWarnings("resource")
+ private boolean loadBatch(BufferAllocator allocator,
+ final RecordBatchLoader batchLoader,
+ BatchSchema schema) throws SchemaChangeException {
+ final List<ValueVector> vectors = createVectors(allocator, schema, 100);
+ final WritableBatch writableBatch = WritableBatch.getBatchNoHV(100, vectors, false);
+ final DrillBuf byteBuf = serializeBatch(allocator, writableBatch);
+ boolean result = batchLoader.load(writableBatch.getDef(), byteBuf);
+ byteBuf.release();
writableBatch.clear();
+ return result;
+ }
+
+ @Test
+ public void testSchemaChange() throws SchemaChangeException {
+ final BufferAllocator allocator = RootAllocatorFactory.newRoot(drillConfig);
+ final RecordBatchLoader batchLoader = new RecordBatchLoader(allocator);
+
+ // Initial schema: a: INT, b: VARCHAR
+ // Schema change: N/A
+
+ BatchSchema schema1 = new SchemaBuilder()
+ .add("a", MinorType.INT)
+ .add("b", MinorType.VARCHAR)
+ .build();
+ {
+ assertTrue(loadBatch(allocator, batchLoader, schema1));
+ assertTrue(schema1.isEquivalent(batchLoader.getSchema()));
+ batchLoader.getContainer().zeroVectors();
+ }
+
+ // Same schema
+ // Schema change: No
+
+ {
+ assertFalse(loadBatch(allocator, batchLoader, schema1));
+ assertTrue(schema1.isEquivalent(batchLoader.getSchema()));
+ batchLoader.getContainer().zeroVectors();
+ }
+
+ // Reverse columns: b: VARCHAR, a: INT
+ // Schema change: ?
+
+ {
+ BatchSchema schema = new SchemaBuilder()
+ .add("b", MinorType.VARCHAR)
+ .add("a", MinorType.INT)
+ .build();
+ assertFalse(loadBatch(allocator, batchLoader, schema));
+
+ // Potential bug: see DRILL-5828
+
+ assertTrue(schema.isEquivalent(batchLoader.getSchema()));
+ batchLoader.getContainer().zeroVectors();
+ }
+
+ // Drop a column: a: INT
+ // Schema change: ?
--- End diff --
Fixed.
---
[GitHub] drill issue #968: DRILL-5830: Resolve regressions to MapR DB from DRILL-5546
Posted by paul-rogers <gi...@git.apache.org>.
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/968
Thanks for the explanation! Bugs in the code are always a concern, but I personally prefer to fix them rather than write more code that tries to work around potential, unknown bugs in other code. (Working around known bugs is, of course, part of the game.)
Sounds like we'll need to test the MapR-DB case for the conditions you identified; that will tell is if the HBase solution (which seems to work now) can also be applied to MapR DB.
---
[GitHub] drill issue #968: DRILL-5830: Resolve regressions to MapR DB from DRILL-5546
Posted by paul-rogers <gi...@git.apache.org>.
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/968
@jinfengni, how unreliable is the project push-down rule? In general, it is hard to reason about software when parts are unreliable. Should every use of the projection push down rule implement a "just in case" fallback as was done for HBase? Should we add that to all storage plugins?
Or, can we fix the projection push-down rule to make it more reliable? It seems that one of your fixes did just that. An HBase query before your fix to the rule didn't expand the wildcard, but the same query, after your projection rule fix, does expand the wildcard.
What are the scenarios in which the rule does not work? Can we fix them in the planner?
---
[GitHub] drill pull request #968: DRILL-5830: Resolve regressions to MapR DB from DRI...
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/968#discussion_r143715206
--- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/util/VectorUtil.java ---
@@ -42,7 +43,12 @@ public static void showVectorAccessibleContent(VectorAccessible va, final String
System.out.println(rows + " row(s):");
List<String> columns = Lists.newArrayList();
for (VectorWrapper<?> vw : va) {
- columns.add(vw.getValueVector().getField().getName());
+ MaterializedField field = vw.getValueVector().getField();
--- End diff --
Maybe we can factor out this block of code (I've seen it at least three times)?
---
[GitHub] drill pull request #968: DRILL-5830: Resolve regressions to MapR DB from DRI...
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/968#discussion_r143748870
--- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/record/vector/TestLoad.java ---
@@ -119,7 +122,283 @@ public void testLoadValueVector() throws Exception {
}
}
assertEquals(100, recordCount);
+
+ // Free the original vectors
+
+ writableBatch.clear();
+
+ // Free the deserialized vectors
+
batchLoader.clear();
+
+ // The allocator will verify that the frees were done correctly.
+
+ allocator.close();
+ }
+
+ // TODO: Replace this low-level code with RowSet usage once
+ // DRILL-5657 is committed to master.
+
+ private static List<ValueVector> createVectors(BufferAllocator allocator, BatchSchema schema, int i) {
+ final List<ValueVector> vectors = new ArrayList<>();
+ for (MaterializedField field : schema) {
+ @SuppressWarnings("resource")
+ ValueVector v = TypeHelper.getNewVector(field, allocator);
+ AllocationHelper.allocate(v, 100, 50);
+ v.getMutator().generateTestData(100);
+ vectors.add(v);
+ }
+ return vectors;
+ }
+
+ private static DrillBuf serializeBatch(BufferAllocator allocator, WritableBatch writableBatch) {
+ final ByteBuf[] byteBufs = writableBatch.getBuffers();
+ int bytes = 0;
+ for (ByteBuf buf : byteBufs) {
+ bytes += buf.writerIndex();
+ }
+ final DrillBuf byteBuf = allocator.buffer(bytes);
+ int index = 0;
+ for (ByteBuf buf : byteBufs) {
+ buf.readBytes(byteBuf, index, buf.writerIndex());
+ index += buf.writerIndex();
+ }
+ byteBuf.writerIndex(bytes);
+ return byteBuf;
+ }
+
+ @SuppressWarnings("resource")
+ private boolean loadBatch(BufferAllocator allocator,
+ final RecordBatchLoader batchLoader,
+ BatchSchema schema) throws SchemaChangeException {
+ final List<ValueVector> vectors = createVectors(allocator, schema, 100);
+ final WritableBatch writableBatch = WritableBatch.getBatchNoHV(100, vectors, false);
+ final DrillBuf byteBuf = serializeBatch(allocator, writableBatch);
+ boolean result = batchLoader.load(writableBatch.getDef(), byteBuf);
+ byteBuf.release();
writableBatch.clear();
+ return result;
+ }
+
+ @Test
+ public void testSchemaChange() throws SchemaChangeException {
+ final BufferAllocator allocator = RootAllocatorFactory.newRoot(drillConfig);
+ final RecordBatchLoader batchLoader = new RecordBatchLoader(allocator);
+
+ // Initial schema: a: INT, b: VARCHAR
+ // Schema change: N/A
+
+ BatchSchema schema1 = new SchemaBuilder()
+ .add("a", MinorType.INT)
+ .add("b", MinorType.VARCHAR)
+ .build();
+ {
+ assertTrue(loadBatch(allocator, batchLoader, schema1));
+ assertTrue(schema1.isEquivalent(batchLoader.getSchema()));
+ batchLoader.getContainer().zeroVectors();
+ }
+
+ // Same schema
+ // Schema change: No
+
+ {
+ assertFalse(loadBatch(allocator, batchLoader, schema1));
+ assertTrue(schema1.isEquivalent(batchLoader.getSchema()));
+ batchLoader.getContainer().zeroVectors();
+ }
+
+ // Reverse columns: b: VARCHAR, a: INT
+ // Schema change: ?
+
+ {
+ BatchSchema schema = new SchemaBuilder()
+ .add("b", MinorType.VARCHAR)
+ .add("a", MinorType.INT)
+ .build();
+ assertFalse(loadBatch(allocator, batchLoader, schema));
+
+ // Potential bug: see DRILL-5828
+
+ assertTrue(schema.isEquivalent(batchLoader.getSchema()));
+ batchLoader.getContainer().zeroVectors();
+ }
+
+ // Drop a column: a: INT
+ // Schema change: ?
--- End diff --
If we drop column ,looks like we think schema change has occurred, right? Should we replace to `Schema change: Yes`?
---
[GitHub] drill pull request #968: DRILL-5830: Resolve regressions to MapR DB from DRI...
Posted by paul-rogers <gi...@git.apache.org>.
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/968#discussion_r142017916
--- Diff: exec/java-exec/src/test/java/org/apache/drill/test/QueryRowSetIterator.java ---
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.test;
+
+import java.util.Iterator;
+
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.memory.BufferAllocator;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.proto.helper.QueryIdHelper;
+import org.apache.drill.exec.record.RecordBatchLoader;
+import org.apache.drill.exec.record.VectorContainer;
+import org.apache.drill.exec.rpc.user.QueryDataBatch;
+import org.apache.drill.test.BufferingQueryEventListener.QueryEvent;
+import org.apache.drill.test.rowSet.DirectRowSet;
+
+public class QueryRowSetIterator implements Iterator<DirectRowSet>, Iterable<DirectRowSet> {
--- End diff --
Sorry, this is a Java 8 thing. In Java 8, `remove()` has a default implementation. In Java 7, the method must be implemented. Fixed.
---
[GitHub] drill issue #968: DRILL-5830: Resolve regressions to MapR DB from DRILL-5546
Posted by vdiravka <gi...@git.apache.org>.
Github user vdiravka commented on the issue:
https://github.com/apache/drill/pull/968
With separate table class for MapR-DB we can follow the same approach of expanding column star into list as for HBase plugin. For that reason I've created a [PR-973 (DRILL-5775)](https://github.com/apache/drill/pull/973) for it.
@jinfengni Please take a look at this PR as well.
---
[GitHub] drill pull request #968: DRILL-5830: Resolve regressions to MapR DB from DRI...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/drill/pull/968
---
[GitHub] drill pull request #968: DRILL-5830: Resolve regressions to MapR DB from DRI...
Posted by vdiravka <gi...@git.apache.org>.
Github user vdiravka commented on a diff in the pull request:
https://github.com/apache/drill/pull/968#discussion_r142017186
--- Diff: exec/java-exec/src/test/java/org/apache/drill/test/QueryRowSetIterator.java ---
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.test;
+
+import java.util.Iterator;
+
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.memory.BufferAllocator;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.proto.helper.QueryIdHelper;
+import org.apache.drill.exec.record.RecordBatchLoader;
+import org.apache.drill.exec.record.VectorContainer;
+import org.apache.drill.exec.rpc.user.QueryDataBatch;
+import org.apache.drill.test.BufferingQueryEventListener.QueryEvent;
+import org.apache.drill.test.rowSet.DirectRowSet;
+
+public class QueryRowSetIterator implements Iterator<DirectRowSet>, Iterable<DirectRowSet> {
--- End diff --
I have a compilation error here. Please implement remove() method from Iterator interface.
---
[GitHub] drill issue #968: DRILL-5830: Resolve regressions to MapR DB from DRILL-5546
Posted by jinfengni <gi...@git.apache.org>.
Github user jinfengni commented on the issue:
https://github.com/apache/drill/pull/968
Project push-down rule may not work in the way we want in at least two cases: 1) there is a bug in the code that we are not aware of, 2) it would not push the list of columns for "select *", when the table is a dynamic table. For HBase, that's not an issue. But it's an issue for MapRDB binary table, as it's processed as a table on a file system, in stead of following the logic of HBase in planning time. I could not comment why MapRDB binary table was handled that way. But the reality is that you still will see column star in execution for MapRDB, if we do not keep the "redundant" code in execution (which essentially converts * into row_key/column family).
If we keep column star in record reader's columns list, then we would end up with nullable-int vs map column. If we expand column star, then we may end up with empty map vs a non-empty map. To me, it's easier to handle the latter case. Plus it's something we have to deal with in case of HBase/JSON plugin.
---
[GitHub] drill pull request #968: DRILL-5830: Resolve regressions to MapR DB from DRI...
Posted by paul-rogers <gi...@git.apache.org>.
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/968#discussion_r144113545
--- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/util/VectorUtil.java ---
@@ -42,7 +43,12 @@ public static void showVectorAccessibleContent(VectorAccessible va, final String
System.out.println(rows + " row(s):");
List<String> columns = Lists.newArrayList();
for (VectorWrapper<?> vw : va) {
- columns.add(vw.getValueVector().getField().getName());
+ MaterializedField field = vw.getValueVector().getField();
--- End diff --
Fixed.
---
[GitHub] drill issue #968: DRILL-5830: Resolve regressions to MapR DB from DRILL-5546
Posted by paul-rogers <gi...@git.apache.org>.
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/968
Rebased onto latest master.
---