You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by paul-rogers <gi...@git.apache.org> on 2018/03/05 00:17:00 UTC
[GitHub] drill pull request #1150: DRILL-6210: Enhanced test schema utilities
GitHub user paul-rogers opened a pull request:
https://github.com/apache/drill/pull/1150
DRILL-6210: Enhanced test schema utilities
The result set loader project enhanced the schema builder used in tests to handle Drill's complex types: maps, unions, lists and repeated lists. The schema builder previously only handled maps.
This PR describes the addition of just this one part of the result set loader.
Also adds a run-time schema improvement: functions to create arrays when testing. Old style:
```
rowSetBuilder.addRow(1, new String[] {"foo", "bar"});
```
New style:
```
rowSetBuilder.addRow(1, strArray"foo", "bar"));
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/paul-rogers/drill DRILL-6210
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/1150.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1150
----
commit abae5200a86b3b68bb1f8e2bb85768a1b814df58
Author: Paul Rogers <pr...@...>
Date: 2018-03-05T00:14:33Z
DRILL-6210: Enhanced test schema utilities
----
---
[GitHub] drill issue #1150: DRILL-6210: Enhanced test schema utilities
Posted by Ben-Zvi <gi...@git.apache.org>.
Github user Ben-Zvi commented on the issue:
https://github.com/apache/drill/pull/1150
@paul-rogers this change also affects the new code (PR#1125) from @ppadma ; the required changes are proposed below; are these correct ? Can you rebase ( #1125 should be committed soon), fix, squash, etc. ?
```
--- a/exec/java-exec/src/test/java/org/apache/drill/exec/record/TestRecordBatchSizer.java
+++ b/exec/java-exec/src/test/java/org/apache/drill/exec/record/TestRecordBatchSizer.java
@@ -32,7 +32,7 @@ import org.apache.drill.test.SubOperatorTest;
import org.apache.drill.test.rowSet.RowSet;
import org.apache.drill.test.rowSet.RowSet.SingleRowSet;
import org.apache.drill.test.rowSet.RowSetBuilder;
-import org.apache.drill.test.rowSet.SchemaBuilder;
+import org.apache.drill.test.rowSet.schema.SchemaBuilder;
import org.junit.Test;
public class TestRecordBatchSizer extends SubOperatorTest {
@@ -506,7 +506,7 @@ public class TestRecordBatchSizer extends SubOperatorTest {
.addMap("map")
.add("key", MinorType.INT)
.add("value", MinorType.VARCHAR)
- .buildMap()
+ .resumeSchema()
.build();
RowSetBuilder builder = fixture.rowSetBuilder(schema);
@@ -589,7 +589,7 @@ public class TestRecordBatchSizer extends SubOperatorTest {
BatchSchema schema = new SchemaBuilder().addMapArray("map").
add("key", MinorType.INT).
add("value", MinorType.VARCHAR).
- buildMap().build();
+ resumeSchema().build();
RowSetBuilder builder = fixture.rowSetBuilder(schema);
@@ -694,8 +694,8 @@ public class TestRecordBatchSizer extends SubOperatorTest {
.addMap("childMap")
.add("childKey", MinorType.INT)
.add("childValue", MinorType.VARCHAR)
- .buildMap()
- .buildMap()
+ .resumeMap()
+ .resumeSchema()
.build();
RowSetBuilder builder = fixture.rowSetBuilder(schema);
```
---
[GitHub] drill issue #1150: DRILL-6210: Enhanced test schema utilities
Posted by paul-rogers <gi...@git.apache.org>.
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/1150
@padma, can you review this?
@ilooner, you might want to take a look also since you've taken an interest in improving our test structures.
---
[GitHub] drill issue #1150: DRILL-6210: Enhanced test schema utilities
Posted by ppadma <gi...@git.apache.org>.
Github user ppadma commented on the issue:
https://github.com/apache/drill/pull/1150
@Ben-Zvi Boaz, can you do a committer review for this PR and include it in this week's batch commit ?
---
[GitHub] drill issue #1150: DRILL-6210: Enhanced test schema utilities
Posted by paul-rogers <gi...@git.apache.org>.
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/1150
Rebased on latest master.
---
[GitHub] drill issue #1150: DRILL-6210: Enhanced test schema utilities
Posted by ppadma <gi...@git.apache.org>.
Github user ppadma commented on the issue:
https://github.com/apache/drill/pull/1150
This is very useful for testing complex types. LGTM. +1.
---
[GitHub] drill issue #1150: DRILL-6210: Enhanced test schema utilities
Posted by paul-rogers <gi...@git.apache.org>.
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/1150
Fixed the typo.
@Ben-Zvi, thanks for pointing out the unit tests: I do have test file in my branch; forgot to include it with the PR. Just did.
---
[GitHub] drill pull request #1150: DRILL-6210: Enhanced test schema utilities
Posted by Ben-Zvi <gi...@git.apache.org>.
Github user Ben-Zvi commented on a diff in the pull request:
https://github.com/apache/drill/pull/1150#discussion_r172735136
--- Diff: exec/java-exec/src/test/java/org/apache/drill/test/rowSet/schema/SchemaBuilder.java ---
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.test.rowSet.schema;
+
+import org.apache.drill.common.types.TypeProtos.DataMode;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.record.BatchSchema;
+import org.apache.drill.exec.record.BatchSchema.SelectionVectorMode;
+import org.apache.drill.exec.record.metadata.AbstractColumnMetadata;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.exec.record.MaterializedField;
+
+/**
+ * Builder of a row set schema expressed as a list of materialized
+ * fields. Optimized for use when creating schemas by hand in tests.
+ * <p>
+ * Example usage to create the following schema: <br>
+ * <tt>(c: INT, a: MAP(b: VARCHAR, d: INT, e: MAP(f: VARCHAR), g: INT),
+ * h: UNION(INT, MAP(h1: INT), LIST(BIGINT)),
+ * i: BIGINT[], j: VARCHAR[][][])</tt>
+ * <p>
+ * Code:<pre><code>
+ * BatchSchema batchSchema = new SchemaBuilder()
+ * .add("c", MinorType.INT)
+ * .addMap("a")
+ * .addNullable("b", MinorType.VARCHAR)
+ * .add("d", MinorType.INT)
+ * .addMap("e") // or .addMapArray("e")
+ * .add("f", MinorType.VARCHAR)
+ * .buildMap()
+ * .add("g", MinorType.INT)
+ * .buildMap()
+ * .addUnion("h") // or .addList("h")
+ * .addType(MinorType.INT)
+ * .addMap()
+ * .add("h1", MinorType.INT)
+ * .buildNested()
+ * .addList()
+ * .addType(MinorType.BIGINT)
+ * .buildNested()
+ * .build()
+ * .addArray("i", MinorType.BIGINT)
+ * .addRepeatedList("k")
--- End diff --
"k" should be "j" ....
---
[GitHub] drill issue #1150: DRILL-6210: Enhanced test schema utilities
Posted by paul-rogers <gi...@git.apache.org>.
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/1150
@Ben-Zvi, the changes look right. Old style was to have a "buildFoo" method for the nested structure "foo". The new style is to have a "resumeBar" method for the outer structure "bar".
If you have trouble, an alternative is to commit @ppadma's PR. Then I can immediately rebase and make the required fixes to her tests.
---
[GitHub] drill pull request #1150: DRILL-6210: Enhanced test schema utilities
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/drill/pull/1150
---