You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by paul-rogers <gi...@git.apache.org> on 2018/03/05 00:17:00 UTC

[GitHub] drill pull request #1150: DRILL-6210: Enhanced test schema utilities

GitHub user paul-rogers opened a pull request:

    https://github.com/apache/drill/pull/1150

    DRILL-6210: Enhanced test schema utilities

    The result set loader project enhanced the schema builder used in tests to handle Drill's complex types: maps, unions, lists and repeated lists. The schema builder previously only handled maps.
    This PR describes the addition of just this one part of the result set loader.
    
    Also adds a run-time schema improvement: functions to create arrays when testing. Old style:
    
    ```
      rowSetBuilder.addRow(1,  new String[] {"foo", "bar"});
    ```
    
    New style:
    ```
      rowSetBuilder.addRow(1,  strArray"foo", "bar"));
    ```


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/paul-rogers/drill DRILL-6210

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/drill/pull/1150.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1150
    
----
commit abae5200a86b3b68bb1f8e2bb85768a1b814df58
Author: Paul Rogers <pr...@...>
Date:   2018-03-05T00:14:33Z

    DRILL-6210: Enhanced test schema utilities

----


---

[GitHub] drill issue #1150: DRILL-6210: Enhanced test schema utilities

Posted by Ben-Zvi <gi...@git.apache.org>.
Github user Ben-Zvi commented on the issue:

    https://github.com/apache/drill/pull/1150
  
    @paul-rogers this change also affects the new code (PR#1125) from @ppadma ; the required changes are proposed below; are these correct ?   Can you rebase ( #1125 should be committed soon), fix, squash, etc. ? 
    
    ```
    --- a/exec/java-exec/src/test/java/org/apache/drill/exec/record/TestRecordBatchSizer.java
    +++ b/exec/java-exec/src/test/java/org/apache/drill/exec/record/TestRecordBatchSizer.java
    @@ -32,7 +32,7 @@ import org.apache.drill.test.SubOperatorTest;
     import org.apache.drill.test.rowSet.RowSet;
     import org.apache.drill.test.rowSet.RowSet.SingleRowSet;
     import org.apache.drill.test.rowSet.RowSetBuilder;
    -import org.apache.drill.test.rowSet.SchemaBuilder;
    +import org.apache.drill.test.rowSet.schema.SchemaBuilder;
     import org.junit.Test;
    
     public class TestRecordBatchSizer extends SubOperatorTest {
    @@ -506,7 +506,7 @@ public class TestRecordBatchSizer extends SubOperatorTest {
           .addMap("map")
             .add("key", MinorType.INT)
             .add("value", MinorType.VARCHAR)
    -      .buildMap()
    +      .resumeSchema()
           .build();
    
         RowSetBuilder builder = fixture.rowSetBuilder(schema);
    @@ -589,7 +589,7 @@ public class TestRecordBatchSizer extends SubOperatorTest {
         BatchSchema schema = new SchemaBuilder().addMapArray("map").
           add("key", MinorType.INT).
           add("value", MinorType.VARCHAR).
    -      buildMap().build();
    +      resumeSchema().build();
    
         RowSetBuilder builder = fixture.rowSetBuilder(schema);
    
    @@ -694,8 +694,8 @@ public class TestRecordBatchSizer extends SubOperatorTest {
             .addMap("childMap")
               .add("childKey", MinorType.INT)
               .add("childValue", MinorType.VARCHAR)
    -          .buildMap()
    -       .buildMap()
    +          .resumeMap()
    +       .resumeSchema()
           .build();
    
         RowSetBuilder builder = fixture.rowSetBuilder(schema);
    ```


---

[GitHub] drill issue #1150: DRILL-6210: Enhanced test schema utilities

Posted by paul-rogers <gi...@git.apache.org>.
Github user paul-rogers commented on the issue:

    https://github.com/apache/drill/pull/1150
  
    @padma, can you review this?
    
    @ilooner, you might want to take a look also since you've taken an interest in improving our test structures. 


---

[GitHub] drill issue #1150: DRILL-6210: Enhanced test schema utilities

Posted by ppadma <gi...@git.apache.org>.
Github user ppadma commented on the issue:

    https://github.com/apache/drill/pull/1150
  
    @Ben-Zvi Boaz, can you do a committer review for this PR and include it in this week's batch commit ? 


---

[GitHub] drill issue #1150: DRILL-6210: Enhanced test schema utilities

Posted by paul-rogers <gi...@git.apache.org>.
Github user paul-rogers commented on the issue:

    https://github.com/apache/drill/pull/1150
  
    Rebased on latest master.


---

[GitHub] drill issue #1150: DRILL-6210: Enhanced test schema utilities

Posted by ppadma <gi...@git.apache.org>.
Github user ppadma commented on the issue:

    https://github.com/apache/drill/pull/1150
  
    This is very useful for testing complex types.  LGTM.  +1. 


---

[GitHub] drill issue #1150: DRILL-6210: Enhanced test schema utilities

Posted by paul-rogers <gi...@git.apache.org>.
Github user paul-rogers commented on the issue:

    https://github.com/apache/drill/pull/1150
  
    Fixed the typo.
    
    @Ben-Zvi, thanks for pointing out the unit tests: I do have test file in my branch; forgot to include it with the PR. Just did.


---

[GitHub] drill pull request #1150: DRILL-6210: Enhanced test schema utilities

Posted by Ben-Zvi <gi...@git.apache.org>.
Github user Ben-Zvi commented on a diff in the pull request:

    https://github.com/apache/drill/pull/1150#discussion_r172735136
  
    --- Diff: exec/java-exec/src/test/java/org/apache/drill/test/rowSet/schema/SchemaBuilder.java ---
    @@ -0,0 +1,214 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + * http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.drill.test.rowSet.schema;
    +
    +import org.apache.drill.common.types.TypeProtos.DataMode;
    +import org.apache.drill.common.types.TypeProtos.MajorType;
    +import org.apache.drill.common.types.TypeProtos.MinorType;
    +import org.apache.drill.exec.record.BatchSchema;
    +import org.apache.drill.exec.record.BatchSchema.SelectionVectorMode;
    +import org.apache.drill.exec.record.metadata.AbstractColumnMetadata;
    +import org.apache.drill.exec.record.metadata.TupleMetadata;
    +import org.apache.drill.exec.record.MaterializedField;
    +
    +/**
    + * Builder of a row set schema expressed as a list of materialized
    + * fields. Optimized for use when creating schemas by hand in tests.
    + * <p>
    + * Example usage to create the following schema: <br>
    + * <tt>(c: INT, a: MAP(b: VARCHAR, d: INT, e: MAP(f: VARCHAR), g: INT),
    + * h: UNION(INT, MAP(h1: INT), LIST(BIGINT)),
    + * i: BIGINT[], j: VARCHAR[][][])</tt>
    + * <p>
    + * Code:<pre><code>
    + *     BatchSchema batchSchema = new SchemaBuilder()
    + *        .add("c", MinorType.INT)
    + *        .addMap("a")
    + *          .addNullable("b", MinorType.VARCHAR)
    + *          .add("d", MinorType.INT)
    + *          .addMap("e") // or .addMapArray("e")
    + *            .add("f", MinorType.VARCHAR)
    + *            .buildMap()
    + *          .add("g", MinorType.INT)
    + *          .buildMap()
    + *        .addUnion("h") // or .addList("h")
    + *          .addType(MinorType.INT)
    + *          .addMap()
    + *            .add("h1", MinorType.INT)
    + *            .buildNested()
    + *          .addList()
    + *            .addType(MinorType.BIGINT)
    + *            .buildNested()
    + *          .build()
    + *        .addArray("i", MinorType.BIGINT)
    + *        .addRepeatedList("k")
    --- End diff --
    
    "k" should be "j" ....


---

[GitHub] drill issue #1150: DRILL-6210: Enhanced test schema utilities

Posted by paul-rogers <gi...@git.apache.org>.
Github user paul-rogers commented on the issue:

    https://github.com/apache/drill/pull/1150
  
    @Ben-Zvi, the changes look right. Old style was to have a "buildFoo" method for the nested structure "foo". The new style is to have a "resumeBar" method for the outer structure "bar".
    
    If you have trouble, an alternative is to commit @ppadma's PR. Then I can immediately rebase and make the required fixes to her tests.


---

[GitHub] drill pull request #1150: DRILL-6210: Enhanced test schema utilities

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/drill/pull/1150


---