You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@drill.apache.org by pr...@apache.org on 2017/12/21 05:19:30 UTC
[03/15] drill git commit: DRILL-5657: Size-aware vector writer
structure
http://git-wip-us.apache.org/repos/asf/drill/blob/40de8ca4/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/package-info.java
----------------------------------------------------------------------
diff --git a/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/package-info.java b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/package-info.java
index f51c1a9..c90a734 100644
--- a/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/package-info.java
+++ b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/package-info.java
@@ -24,8 +24,40 @@
* framework for the java-exec project. That one implementation is specific to
* unit tests, but the accessor framework could easily be used for other
* purposes as well.
+ *
+ * <h4>Vector Overflow Handling</h4>
+ *
+ * The writers provide integrated support for detecting and handling vector
+ * overflow. Overflow occurs when a value exceeds some maximum, such as the
+ * 16MB block size in Netty. Overflow handling consists of replacing the
+ * "full" vector with a new, empty vector as part of a new batch. Overflow
+ * handing code must copy partially written values from the "overflow" row
+ * to the new vectors. The classes here do not provide overflow handling,
+ * rather they provide the framework on top of which overflow handling can be
+ * built by a higher level of abstraction.
+ *
+ * <h4>JSON-Like Model</h4>
+ *
+ * The object reader and writer provide a generic, JSON-like interface
+ * to allow any valid combination of readers or writers (generically
+ * accessors):<pre><code>
+ * row : tuple
+ * tuple : (name column) *
+ * column : scalar obj | array obj | tuple obj
+ * scalar obj : scalar accessor
+ * array obj : array accessor
+ * array accessor : element accessor
+ * tuple obj : tuple</code></pre>
* <p>
- * Drill provides a set of column readers and writers. Compared to those, this
+ * As seen above, the accessor tree starts with a tuple (a row in the form of
+ * a class provided by the consumer.) Each column in the tuple is represented
+ * by an object accesor. That object accessor contains a scalar, tuple or array
+ * accessor. This models Drill's JSON structure: a row can have a list of lists
+ * of tuples that contains lists of ints, say.
+ *
+ * <h4>Comparison with Previous Vector Readers and Writers</h4>
+ *
+ * Drill provides a set of vector readers and writers. Compared to those, this
* set:
* <ul>
* <li>Works with all Drill data types. The other set works only with repeated
@@ -36,23 +68,24 @@
* other set has accessors specific to each of the ~30 data types which Drill
* supports.</li>
* </ul>
- * The key difference is that this set is designed for developer ease-of-use, a
- * primary requirement for unit tests. The other set is designed to be used in
+ * The key difference is that this set is designed for both developer ease-of-use
+ * and performance. Developer eas-of-use is a
+ * primary requirement for unit tests. Performance is critical for production
+ * code. The other set is designed to be used in
* machine-generated or write-once code and so can be much more complex.
- * <p>
- * That is, the accessors here are optimized for test code: they trade
- * convenience for a slight decrease in speed (the performance hit comes from
- * the extra level of indirection which hides the complex, type-specific code
- * otherwise required.)
- * <p>
- * {@link ColumnReader} and {@link ColumnWriter} are the core abstractions: they
+ *
+ * <h4>Overview of the Code Structure</h4>
+ *
+ * {@link ScalarReader} and {@link ColumnWriter} are the core abstractions: they
* provide simplified access to the myriad of Drill column types via a
* simplified, uniform API. {@link TupleReader} and {@link TupleWriter} provide
* a simplified API to rows or maps (both of which are tuples in Drill.)
* {@link AccessorUtilities} provides a number of data conversion tools.
- * <p>
- * Overview of the code structure:
* <dl>
+ * <dt>ObjectWriter, ObjectReader</dt>
+ * <dd>Drill follows a JSON data model. A row is a tuple (AKA structure). Each
+ * column is a scalar, a map (AKA tuple, structure) or an array (AKA a repeated
+ * value.)</dd>
* <dt>TupleWriter, TupleReader</dt>
* <dd>In relational terms, a tuple is an ordered collection of values, where
* the meaning of the order is provided by a schema (usually a name/type pair.)
@@ -62,12 +95,8 @@
* But, doing so is slower than access by position (index). To provide efficient
* code, the tuple classes assume that the implementation imposes a column
* ordering which can be exposed via the indexes.</dd>
- * <dt>ColumnAccessor</dt>
- * <dd>A generic base class for column readers and writers that provides the
- * column data type.</dd>
- * <dt>ColumnWriter, ColumnReader</dt>
- * <dd>A uniform interface implemented for each column type ("major type" in
- * Drill terminology). The scalar types: Nullable (Drill optional) and
+ * <dt>ScalarWriter, ScalarReader</dt>
+ * <dd>A uniform interface for the scalar types: Nullable (Drill optional) and
* non-nullable (Drill required) fields use the same interface. Arrays (Drill
* repeated) are special. To handle the array aspect, even array fields use the
* same interface, but the <tt>getArray</tt> method returns another layer of
@@ -98,11 +127,11 @@
* <dd>The generated accessors: one for each combination of write/read, data
* (minor) type and cardinality (data model).
* <dd>
- * <dt>RowIndex</dt>
+ * <dt>ColumnReaderIndex, ColumnWriterIndex</dt>
* <dd>This nested class binds the accessor to the current row position for the
* entire record batch. That is, you don't ask for the value of column a for row
* 5, then the value of column b for row 5, etc. as with the "raw" vectors.
- * Instead, the implementation sets the row position (with, say an interator.)
+ * Instead, the implementation sets the row position (with, say an iterator.)
* Then, all columns implicitly return values for the current row.
* <p>
* Different implementations of the row index handle the case of no selection
@@ -122,6 +151,16 @@
* The column index picks out the x coordinate (horizontal position along the
* columns.)</dt>
* </dl>
+ * <h4>Column Writer Optimizations</h4>
+ * The writer classes here started as a simple abstraction on top of the existing
+ * vector mutators. The classes were then recruited for use in a new writer
+ * abstraction for Drill's record readers. At that point, performance became
+ * critical. The key to performance is to bypass the vector and the mutator and
+ * instead work with the Netty direct memory functions. This seems a risky
+ * approach until we realize that the writers form a very clear interface:
+ * the same interface supported the original mutator-based implementation and
+ * the revised Netty-based implementation. The benefit, however, is stark;
+ * the direct-to-Netty version is up to 4x faster (for repeated types).
*/
package org.apache.drill.exec.vector.accessor;
http://git-wip-us.apache.org/repos/asf/drill/blob/40de8ca4/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/AbstractArrayReader.java
----------------------------------------------------------------------
diff --git a/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/AbstractArrayReader.java b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/AbstractArrayReader.java
new file mode 100644
index 0000000..7fb0c9d
--- /dev/null
+++ b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/AbstractArrayReader.java
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.vector.accessor.reader;
+
+import org.apache.drill.exec.vector.UInt4Vector.Accessor;
+import org.apache.drill.exec.vector.accessor.ArrayReader;
+import org.apache.drill.exec.vector.accessor.ColumnReaderIndex;
+import org.apache.drill.exec.vector.accessor.ObjectReader;
+import org.apache.drill.exec.vector.accessor.ObjectType;
+import org.apache.drill.exec.vector.accessor.ScalarElementReader;
+import org.apache.drill.exec.vector.accessor.TupleReader;
+import org.apache.drill.exec.vector.complex.RepeatedValueVector;
+
+/**
+ * Reader for an array-valued column. This reader provides access to specific
+ * array members via an array index. This is an abstract base class;
+ * subclasses are generated for each repeated value vector type.
+ */
+
+public abstract class AbstractArrayReader implements ArrayReader {
+
+ /**
+ * Object representation of an array reader.
+ */
+
+ public static class ArrayObjectReader extends AbstractObjectReader {
+
+ private AbstractArrayReader arrayReader;
+
+ public ArrayObjectReader(AbstractArrayReader arrayReader) {
+ this.arrayReader = arrayReader;
+ }
+
+ @Override
+ public void bindIndex(ColumnReaderIndex index) {
+ arrayReader.bindIndex(index);
+ }
+
+ @Override
+ public ObjectType type() {
+ return ObjectType.ARRAY;
+ }
+
+ @Override
+ public ArrayReader array() {
+ return arrayReader;
+ }
+
+ @Override
+ public ScalarElementReader elements() {
+ return arrayReader.elements();
+ }
+
+ @Override
+ public Object getObject() {
+ return arrayReader.getObject();
+ }
+
+ @Override
+ public String getAsString() {
+ return arrayReader.getAsString();
+ }
+
+ @Override
+ public void reposition() {
+ arrayReader.reposition();
+ }
+ }
+
+ public static class BaseElementIndex {
+ private final ColumnReaderIndex base;
+ protected int startOffset;
+ protected int length;
+
+ public BaseElementIndex(ColumnReaderIndex base) {
+ this.base = base;
+ }
+
+ public int batchIndex() {
+ return base.batchIndex();
+ }
+
+ public void reset(int startOffset, int length) {
+ assert length >= 0;
+ assert startOffset >= 0;
+ this.startOffset = startOffset;
+ this.length = length;
+ }
+
+ public int size() { return length; }
+
+ public int elementIndex(int index) {
+ if (index < 0 || length <= index) {
+ throw new IndexOutOfBoundsException("Index = " + index + ", length = " + length);
+ }
+ return startOffset + index;
+ }
+ }
+
+ private final Accessor accessor;
+ private final VectorAccessor vectorAccessor;
+ protected ColumnReaderIndex baseIndex;
+ protected BaseElementIndex elementIndex;
+
+ public AbstractArrayReader(RepeatedValueVector vector) {
+ accessor = vector.getOffsetVector().getAccessor();
+ vectorAccessor = null;
+ }
+
+ public AbstractArrayReader(VectorAccessor vectorAccessor) {
+ accessor = null;
+ this.vectorAccessor = vectorAccessor;
+ }
+
+ public void bindIndex(ColumnReaderIndex index) {
+ baseIndex = index;
+ if (vectorAccessor != null) {
+ vectorAccessor.bind(index);
+ }
+ }
+
+ private Accessor accessor() {
+ if (accessor != null) {
+ return accessor;
+ }
+ return ((RepeatedValueVector) (vectorAccessor.vector())).getOffsetVector().getAccessor();
+ }
+
+ public void reposition() {
+ final int index = baseIndex.vectorIndex();
+ final Accessor curAccesssor = accessor();
+ final int startPosn = curAccesssor.get(index);
+ elementIndex.reset(startPosn, curAccesssor.get(index + 1) - startPosn);
+ }
+
+ @Override
+ public int size() { return elementIndex.size(); }
+
+ @Override
+ public ScalarElementReader elements() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public ObjectReader entry(int index) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public TupleReader tuple(int index) {
+ return entry(index).tuple();
+ }
+
+ @Override
+ public ArrayReader array(int index) {
+ return entry(index).array();
+ }
+
+ @Override
+ public ObjectReader entry() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public TupleReader tuple() {
+ return entry().tuple();
+ }
+
+ @Override
+ public ArrayReader array() {
+ return entry().array();
+ }
+}
http://git-wip-us.apache.org/repos/asf/drill/blob/40de8ca4/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/AbstractObjectReader.java
----------------------------------------------------------------------
diff --git a/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/AbstractObjectReader.java b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/AbstractObjectReader.java
new file mode 100644
index 0000000..59a066e
--- /dev/null
+++ b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/AbstractObjectReader.java
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.vector.accessor.reader;
+
+import org.apache.drill.exec.vector.accessor.ArrayReader;
+import org.apache.drill.exec.vector.accessor.ColumnReaderIndex;
+import org.apache.drill.exec.vector.accessor.ObjectReader;
+import org.apache.drill.exec.vector.accessor.ScalarElementReader;
+import org.apache.drill.exec.vector.accessor.ScalarReader;
+import org.apache.drill.exec.vector.accessor.TupleReader;
+
+public abstract class AbstractObjectReader implements ObjectReader {
+
+ public abstract void bindIndex(ColumnReaderIndex index);
+
+ public void reposition() { }
+
+ @Override
+ public ScalarReader scalar() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public TupleReader tuple() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public ArrayReader array() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public ScalarElementReader elements() {
+ throw new UnsupportedOperationException();
+ }
+}
http://git-wip-us.apache.org/repos/asf/drill/blob/40de8ca4/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/AbstractTupleReader.java
----------------------------------------------------------------------
diff --git a/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/AbstractTupleReader.java b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/AbstractTupleReader.java
new file mode 100644
index 0000000..afa0cb7
--- /dev/null
+++ b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/AbstractTupleReader.java
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.vector.accessor.reader;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.drill.exec.record.TupleMetadata;
+import org.apache.drill.exec.vector.accessor.ArrayReader;
+import org.apache.drill.exec.vector.accessor.ColumnReaderIndex;
+import org.apache.drill.exec.vector.accessor.ObjectReader;
+import org.apache.drill.exec.vector.accessor.ObjectType;
+import org.apache.drill.exec.vector.accessor.ScalarElementReader;
+import org.apache.drill.exec.vector.accessor.ScalarReader;
+import org.apache.drill.exec.vector.accessor.TupleReader;
+
+/**
+ * Reader for a tuple (a row or a map.) Provides access to each
+ * column using either a name or a numeric index.
+ */
+
+public abstract class AbstractTupleReader implements TupleReader {
+
+ public static class TupleObjectReader extends AbstractObjectReader {
+
+ private AbstractTupleReader tupleReader;
+
+ public TupleObjectReader(AbstractTupleReader tupleReader) {
+ this.tupleReader = tupleReader;
+ }
+
+ @Override
+ public void bindIndex(ColumnReaderIndex index) {
+ tupleReader.bindIndex(index);
+ }
+
+ @Override
+ public ObjectType type() {
+ return ObjectType.TUPLE;
+ }
+
+ @Override
+ public TupleReader tuple() {
+ return tupleReader;
+ }
+
+ @Override
+ public Object getObject() {
+ return tupleReader.getObject();
+ }
+
+ @Override
+ public String getAsString() {
+ return tupleReader.getAsString();
+ }
+
+ @Override
+ public void reposition() {
+ tupleReader.reposition();
+ }
+ }
+
+ protected final TupleMetadata schema;
+ private final AbstractObjectReader readers[];
+
+ protected AbstractTupleReader(TupleMetadata schema, AbstractObjectReader readers[]) {
+ this.schema = schema;
+ this.readers = readers;
+ }
+
+ public void bindIndex(ColumnReaderIndex index) {
+ for (int i = 0; i < readers.length; i++) {
+ readers[i].bindIndex(index);
+ }
+ }
+
+ @Override
+ public TupleMetadata schema() { return schema; }
+
+ @Override
+ public int columnCount() { return schema().size(); }
+
+ @Override
+ public ObjectReader column(int colIndex) {
+ return readers[colIndex];
+ }
+
+ @Override
+ public ObjectReader column(String colName) {
+ int index = schema.index(colName);
+ if (index == -1) {
+ return null; }
+ return readers[index];
+ }
+
+ @Override
+ public ScalarReader scalar(int colIndex) {
+ return column(colIndex).scalar();
+ }
+
+ @Override
+ public ScalarReader scalar(String colName) {
+ return column(colName).scalar();
+ }
+
+ @Override
+ public TupleReader tuple(int colIndex) {
+ return column(colIndex).tuple();
+ }
+
+ @Override
+ public TupleReader tuple(String colName) {
+ return column(colName).tuple();
+ }
+
+ @Override
+ public ArrayReader array(int colIndex) {
+ return column(colIndex).array();
+ }
+
+ @Override
+ public ArrayReader array(String colName) {
+ return column(colName).array();
+ }
+
+ @Override
+ public ObjectType type(int colIndex) {
+ return column(colIndex).type();
+ }
+
+ @Override
+ public ObjectType type(String colName) {
+ return column(colName).type();
+ }
+
+ @Override
+ public ScalarElementReader elements(int colIndex) {
+ return column(colIndex).elements();
+ }
+
+ @Override
+ public ScalarElementReader elements(String colName) {
+ return column(colName).elements();
+ }
+
+ public void reposition() {
+ for (int i = 0; i < columnCount(); i++) {
+ readers[i].reposition();
+ }
+ }
+
+ @Override
+ public Object getObject() {
+ List<Object> elements = new ArrayList<>();
+ for (int i = 0; i < columnCount(); i++) {
+ elements.add(readers[i].getObject());
+ }
+ return elements;
+ }
+
+ @Override
+ public String getAsString() {
+ StringBuilder buf = new StringBuilder();
+ buf.append("(");
+ for (int i = 0; i < columnCount(); i++) {
+ if (i > 0) {
+ buf.append( ", " );
+ }
+ buf.append(readers[i].getAsString());
+ }
+ buf.append(")");
+ return buf.toString();
+ }
+}
http://git-wip-us.apache.org/repos/asf/drill/blob/40de8ca4/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/BaseElementReader.java
----------------------------------------------------------------------
diff --git a/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/BaseElementReader.java b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/BaseElementReader.java
new file mode 100644
index 0000000..f32c101
--- /dev/null
+++ b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/BaseElementReader.java
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.vector.accessor.reader;
+
+import java.math.BigDecimal;
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.exec.vector.ValueVector;
+import org.apache.drill.exec.vector.accessor.ColumnReaderIndex;
+import org.apache.drill.exec.vector.accessor.ObjectType;
+import org.apache.drill.exec.vector.accessor.ScalarElementReader;
+import org.apache.drill.exec.vector.accessor.impl.AccessorUtilities;
+import org.joda.time.Period;
+
+public abstract class BaseElementReader implements ScalarElementReader {
+
+ public static class ScalarElementObjectReader extends AbstractObjectReader {
+
+ private BaseElementReader elementReader;
+
+ public ScalarElementObjectReader(BaseElementReader elementReader) {
+ this.elementReader = elementReader;
+ }
+
+ @Override
+ public void bindIndex(ColumnReaderIndex index) {
+ elementReader.bindIndex((ElementReaderIndex) index);
+ }
+
+ @Override
+ public ObjectType type() {
+ return ObjectType.SCALAR;
+ }
+
+ @Override
+ public ScalarElementReader elements() {
+ return elementReader;
+ }
+
+ @Override
+ public Object getObject() {
+ // Simple: return elements as an object list.
+ // If really needed, could return as a typed array, but that
+ // is a bit of a hassle.
+
+ List<Object> elements = new ArrayList<>();
+ for (int i = 0; i < elementReader.size(); i++) {
+ elements.add(elementReader.getObject(i));
+ }
+ return elements;
+ }
+
+ @Override
+ public String getAsString() {
+ StringBuilder buf = new StringBuilder();
+ buf.append("[");
+ for (int i = 0; i < elementReader.size(); i++) {
+ if (i > 0) {
+ buf.append( ", " );
+ }
+ buf.append(elementReader.getAsString(i));
+ }
+ buf.append("]");
+ return buf.toString();
+ }
+ }
+
+ protected ElementReaderIndex vectorIndex;
+ protected VectorAccessor vectorAccessor;
+
+ public abstract void bindVector(ValueVector vector);
+
+ public void bindVector(MajorType majorType, VectorAccessor va) {
+ vectorAccessor = va;
+ }
+
+ protected void bindIndex(ElementReaderIndex rowIndex) {
+ this.vectorIndex = rowIndex;
+ }
+
+ @Override
+ public int size() { return vectorIndex.size(); }
+
+ @Override
+ public Object getObject(int index) {
+ if (isNull(index)) {
+ return "null";
+ }
+ switch (valueType()) {
+ case BYTES:
+ return getBytes(index);
+ case DECIMAL:
+ return getDecimal(index);
+ case DOUBLE:
+ return getDouble(index);
+ case INTEGER:
+ return getInt(index);
+ case LONG:
+ return getLong(index);
+ case PERIOD:
+ return getPeriod(index);
+ case STRING:
+ return getString(index);
+ default:
+ throw new IllegalStateException("Unexpected type: " + valueType());
+ }
+ }
+
+ @Override
+ public String getAsString(int index) {
+ switch (valueType()) {
+ case BYTES:
+ return AccessorUtilities.bytesToString(getBytes(index));
+ case DOUBLE:
+ return Double.toString(getDouble(index));
+ case INTEGER:
+ return Integer.toString(getInt(index));
+ case LONG:
+ return Long.toString(getLong(index));
+ case STRING:
+ return "\"" + getString(index) + "\"";
+ case DECIMAL:
+ return getDecimal(index).toPlainString();
+ case PERIOD:
+ return getPeriod(index).normalizedStandard().toString();
+ default:
+ throw new IllegalArgumentException("Unsupported type " + valueType());
+ }
+ }
+
+ @Override
+ public boolean isNull(int index) {
+ return false;
+ }
+
+ @Override
+ public int getInt(int index) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public long getLong(int index) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public double getDouble(int index) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public String getString(int index) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte[] getBytes(int index) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public BigDecimal getDecimal(int index) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Period getPeriod(int index) {
+ throw new UnsupportedOperationException();
+ }
+}
http://git-wip-us.apache.org/repos/asf/drill/blob/40de8ca4/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/BaseScalarReader.java
----------------------------------------------------------------------
diff --git a/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/BaseScalarReader.java b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/BaseScalarReader.java
new file mode 100644
index 0000000..fb9a711
--- /dev/null
+++ b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/BaseScalarReader.java
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.vector.accessor.reader;
+
+import java.math.BigDecimal;
+
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.exec.vector.ValueVector;
+import org.apache.drill.exec.vector.accessor.ColumnReaderIndex;
+import org.apache.drill.exec.vector.accessor.ObjectType;
+import org.apache.drill.exec.vector.accessor.ScalarReader;
+import org.apache.drill.exec.vector.accessor.impl.AccessorUtilities;
+import org.joda.time.Period;
+
+/**
+ * Column reader implementation that acts as the basis for the
+ * generated, vector-specific implementations. All set methods
+ * throw an exception; subclasses simply override the supported
+ * method(s).
+ */
+
+public abstract class BaseScalarReader implements ScalarReader {
+
+ public static class ScalarObjectReader extends AbstractObjectReader {
+
+ private BaseScalarReader scalarReader;
+
+ public ScalarObjectReader(BaseScalarReader scalarReader) {
+ this.scalarReader = scalarReader;
+ }
+
+ @Override
+ public void bindIndex(ColumnReaderIndex index) {
+ scalarReader.bindIndex(index);
+ }
+
+ @Override
+ public ObjectType type() {
+ return ObjectType.SCALAR;
+ }
+
+ @Override
+ public ScalarReader scalar() {
+ return scalarReader;
+ }
+
+ @Override
+ public Object getObject() {
+ return scalarReader.getObject();
+ }
+
+ @Override
+ public String getAsString() {
+ return scalarReader.getAsString();
+ }
+ }
+
+ protected ColumnReaderIndex vectorIndex;
+ protected VectorAccessor vectorAccessor;
+
+ public static ScalarObjectReader build(ValueVector vector, BaseScalarReader reader) {
+ reader.bindVector(vector);
+ return new ScalarObjectReader(reader);
+ }
+
+ public static AbstractObjectReader build(MajorType majorType, VectorAccessor va,
+ BaseScalarReader reader) {
+ reader.bindVector(majorType, va);
+ return new ScalarObjectReader(reader);
+ }
+
+ public abstract void bindVector(ValueVector vector);
+
+ protected void bindIndex(ColumnReaderIndex rowIndex) {
+ this.vectorIndex = rowIndex;
+ if (vectorAccessor != null) {
+ vectorAccessor.bind(rowIndex);
+ }
+ }
+
+ public void bindVector(MajorType majorType, VectorAccessor va) {
+ vectorAccessor = va;
+ }
+
+ @Override
+ public Object getObject() {
+ if (isNull()) {
+ return null;
+ }
+ switch (valueType()) {
+ case BYTES:
+ return getBytes();
+ case DECIMAL:
+ return getDecimal();
+ case DOUBLE:
+ return getDouble();
+ case INTEGER:
+ return getInt();
+ case LONG:
+ return getLong();
+ case PERIOD:
+ return getPeriod();
+ case STRING:
+ return getString();
+ default:
+ throw new IllegalStateException("Unexpected type: " + valueType());
+ }
+ }
+
+ @Override
+ public String getAsString() {
+ if (isNull()) {
+ return "null";
+ }
+ switch (valueType()) {
+ case BYTES:
+ return AccessorUtilities.bytesToString(getBytes());
+ case DOUBLE:
+ return Double.toString(getDouble());
+ case INTEGER:
+ return Integer.toString(getInt());
+ case LONG:
+ return Long.toString(getLong());
+ case STRING:
+ return "\"" + getString() + "\"";
+ case DECIMAL:
+ return getDecimal().toPlainString();
+ case PERIOD:
+ return getPeriod().normalizedStandard().toString();
+ default:
+ throw new IllegalArgumentException("Unsupported type " + valueType());
+ }
+ }
+
+ @Override
+ public boolean isNull() {
+ return false;
+ }
+
+ @Override
+ public int getInt() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public long getLong() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public double getDouble() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public String getString() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public byte[] getBytes() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public BigDecimal getDecimal() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Period getPeriod() {
+ throw new UnsupportedOperationException();
+ }
+}
http://git-wip-us.apache.org/repos/asf/drill/blob/40de8ca4/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/ColumnReaderFactory.java
----------------------------------------------------------------------
diff --git a/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/ColumnReaderFactory.java b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/ColumnReaderFactory.java
new file mode 100644
index 0000000..0bcb6e2
--- /dev/null
+++ b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/ColumnReaderFactory.java
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.vector.accessor.reader;
+
+import org.apache.drill.common.types.TypeProtos.DataMode;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.vector.ValueVector;
+import org.apache.drill.exec.vector.accessor.ColumnAccessors;
+import org.apache.drill.exec.vector.complex.RepeatedValueVector;
+
+/**
+ * Gather generated reader classes into a set of class tables to allow rapid
+ * run-time creation of readers. Builds the reader and its object reader
+ * wrapper which binds the vector to the reader.
+ */
+
+@SuppressWarnings("unchecked")
+public class ColumnReaderFactory {
+
+ private static final int typeCount = MinorType.values().length;
+ private static final Class<? extends BaseScalarReader> requiredReaders[] = new Class[typeCount];
+ private static final Class<? extends BaseScalarReader> nullableReaders[] = new Class[typeCount];
+ private static final Class<? extends BaseElementReader> elementReaders[] = new Class[typeCount];
+
+ static {
+ ColumnAccessors.defineRequiredReaders(requiredReaders);
+ ColumnAccessors.defineNullableReaders(nullableReaders);
+ ColumnAccessors.defineArrayReaders(elementReaders);
+ }
+
+ public static AbstractObjectReader buildColumnReader(ValueVector vector) {
+ MajorType major = vector.getField().getType();
+ MinorType type = major.getMinorType();
+ DataMode mode = major.getMode();
+
+ switch (type) {
+ case GENERIC_OBJECT:
+ case LATE:
+ case NULL:
+ case LIST:
+ case MAP:
+ throw new UnsupportedOperationException(type.toString());
+ default:
+ switch (mode) {
+ case OPTIONAL:
+ return BaseScalarReader.build(vector, newAccessor(type, nullableReaders));
+ case REQUIRED:
+ return BaseScalarReader.build(vector, newAccessor(type, requiredReaders));
+ case REPEATED:
+ return ScalarArrayReader.build((RepeatedValueVector) vector, newAccessor(type, elementReaders));
+ default:
+ throw new UnsupportedOperationException(mode.toString());
+ }
+ }
+ }
+
+ public static AbstractObjectReader buildColumnReader(MajorType majorType, VectorAccessor va) {
+ MinorType type = majorType.getMinorType();
+ DataMode mode = majorType.getMode();
+
+ switch (type) {
+ case GENERIC_OBJECT:
+ case LATE:
+ case NULL:
+ case LIST:
+ case MAP:
+ throw new UnsupportedOperationException(type.toString());
+ default:
+ switch (mode) {
+ case OPTIONAL:
+ return BaseScalarReader.build(majorType, va, newAccessor(type, nullableReaders));
+ case REQUIRED:
+ return BaseScalarReader.build(majorType, va, newAccessor(type, requiredReaders));
+ case REPEATED:
+ return ScalarArrayReader.build(majorType, va, newAccessor(type, elementReaders));
+ default:
+ throw new UnsupportedOperationException(mode.toString());
+ }
+ }
+ }
+
+ public static <T> T newAccessor(MinorType type, Class<? extends T> accessors[]) {
+ try {
+ Class<? extends T> accessorClass = accessors[type.ordinal()];
+ if (accessorClass == null) {
+ throw new UnsupportedOperationException(type.toString());
+ }
+ return accessorClass.newInstance();
+ } catch (InstantiationException | IllegalAccessException e) {
+ throw new IllegalStateException(e);
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/drill/blob/40de8ca4/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/ElementReaderIndex.java
----------------------------------------------------------------------
diff --git a/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/ElementReaderIndex.java b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/ElementReaderIndex.java
new file mode 100644
index 0000000..9985edc
--- /dev/null
+++ b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/ElementReaderIndex.java
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.vector.accessor.reader;
+
+public interface ElementReaderIndex {
+ int batchIndex();
+ int size();
+ int vectorIndex(int posn);
+}
http://git-wip-us.apache.org/repos/asf/drill/blob/40de8ca4/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/FixedWidthElementReaderIndex.java
----------------------------------------------------------------------
diff --git a/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/FixedWidthElementReaderIndex.java b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/FixedWidthElementReaderIndex.java
new file mode 100644
index 0000000..4f3aeeb
--- /dev/null
+++ b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/FixedWidthElementReaderIndex.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.vector.accessor.reader;
+
+import org.apache.drill.exec.vector.accessor.ColumnReaderIndex;
+import org.apache.drill.exec.vector.accessor.reader.AbstractArrayReader.BaseElementIndex;
+
+/**
+ * Index into the vector of elements for a repeated vector.
+ * Keeps track of the current offset in terms of value positions.
+ */
+
+public class FixedWidthElementReaderIndex extends BaseElementIndex implements ElementReaderIndex {
+
+ public FixedWidthElementReaderIndex(ColumnReaderIndex base) {
+ super(base);
+ }
+
+ @Override
+ public int vectorIndex(int posn) {
+ return elementIndex(posn);
+ }
+}
http://git-wip-us.apache.org/repos/asf/drill/blob/40de8ca4/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/MapReader.java
----------------------------------------------------------------------
diff --git a/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/MapReader.java b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/MapReader.java
new file mode 100644
index 0000000..66bc067
--- /dev/null
+++ b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/MapReader.java
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.vector.accessor.reader;
+
+import java.util.List;
+
+import org.apache.drill.exec.record.ColumnMetadata;
+
+/**
+ * Reader for a Drill Map type. Maps are actually tuples, just like rows.
+ */
+
+public class MapReader extends AbstractTupleReader {
+
+ protected MapReader(ColumnMetadata schema, AbstractObjectReader readers[]) {
+ super(schema.mapSchema(), readers);
+ }
+
+ public static TupleObjectReader build(ColumnMetadata schema, AbstractObjectReader readers[]) {
+ return new TupleObjectReader(new MapReader(schema, readers));
+ }
+
+ public static AbstractObjectReader build(ColumnMetadata metadata,
+ List<AbstractObjectReader> readers) {
+ AbstractObjectReader readerArray[] = new AbstractObjectReader[readers.size()];
+ return build(metadata, readers.toArray(readerArray));
+ }
+}
http://git-wip-us.apache.org/repos/asf/drill/blob/40de8ca4/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/ObjectArrayReader.java
----------------------------------------------------------------------
diff --git a/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/ObjectArrayReader.java b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/ObjectArrayReader.java
new file mode 100644
index 0000000..9ed89f1
--- /dev/null
+++ b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/ObjectArrayReader.java
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.vector.accessor.reader;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.drill.exec.vector.accessor.ColumnReaderIndex;
+import org.apache.drill.exec.vector.accessor.ObjectReader;
+import org.apache.drill.exec.vector.accessor.ObjectType;
+import org.apache.drill.exec.vector.complex.RepeatedValueVector;
+
+/**
+ * Reader for an array of either tuples or other arrays.
+ */
+
+public class ObjectArrayReader extends AbstractArrayReader {
+
+ /**
+ * Index into the vector of elements for a repeated vector.
+ * Keeps track of the current offset in terms of value positions.
+ * This is a derived index. The base index points to an entry
+ * in the offset vector for the array. This inner index picks
+ * off elements within the range of offsets for that one entry.
+ * For example:<pre><code>
+ * [ ... 100 105 ...]
+ * </code></pre>In the above the value 100 might be at outer
+ * offset 5. The inner array will pick off the five values
+ * 100...104.
+ * <p>
+ * Because arrays allow random access on read, the inner offset
+ * is reset on each access to the array.
+ */
+
+ public static class ObjectElementReaderIndex extends BaseElementIndex implements ColumnReaderIndex {
+
+ private int posn;
+
+ public ObjectElementReaderIndex(ColumnReaderIndex base) {
+ super(base);
+ }
+
+ @Override
+ public int vectorIndex() {
+ return startOffset + posn;
+ }
+
+ public void set(int index) {
+ if (index < 0 || length <= index) {
+ throw new IndexOutOfBoundsException("Index = " + index + ", length = " + length);
+ }
+ posn = index;
+ }
+
+ public int posn() { return posn; }
+ }
+
+ /**
+ * Reader for each element.
+ */
+
+ private final AbstractObjectReader elementReader;
+
+ /**
+ * Index used to access elements.
+ */
+
+ private ObjectElementReaderIndex objElementIndex;
+
+ private ObjectArrayReader(RepeatedValueVector vector, AbstractObjectReader elementReader) {
+ super(vector);
+ this.elementReader = elementReader;
+ }
+
+ private ObjectArrayReader(VectorAccessor vectorAccessor, AbstractObjectReader elementReader) {
+ super(vectorAccessor);
+ this.elementReader = elementReader;
+ }
+
+ public static ArrayObjectReader build(RepeatedValueVector vector,
+ AbstractObjectReader elementReader) {
+ return new ArrayObjectReader(
+ new ObjectArrayReader(vector, elementReader));
+ }
+
+ public static AbstractObjectReader build(VectorAccessor vectorAccessor,
+ AbstractObjectReader elementReader) {
+ return new ArrayObjectReader(
+ new ObjectArrayReader(vectorAccessor, elementReader));
+ }
+
+ @Override
+ public void bindIndex(ColumnReaderIndex index) {
+ super.bindIndex(index);
+ objElementIndex = new ObjectElementReaderIndex(baseIndex);
+ elementIndex = objElementIndex;
+ elementReader.bindIndex(objElementIndex);
+ }
+
+ @Override
+ public ObjectType entryType() {
+ return elementReader.type();
+ }
+
+ @Override
+ public void setPosn(int index) {
+ objElementIndex.set(index);
+ elementReader.reposition();
+ }
+
+ @Override
+ public ObjectReader entry() {
+ return elementReader;
+ }
+
+ @Override
+ public ObjectReader entry(int index) {
+ setPosn(index);
+ return entry();
+ }
+
+ @Override
+ public Object getObject() {
+ List<Object> array = new ArrayList<>();
+ for (int i = 0; i < objElementIndex.size(); i++) {
+ array.add(entry(i).getObject());
+ }
+ return array;
+ }
+
+ @Override
+ public String getAsString() {
+ StringBuilder buf = new StringBuilder();
+ buf.append("[");
+ for (int i = 0; i < size(); i++) {
+ if (i > 0) {
+ buf.append( ", " );
+ }
+ buf.append(entry(i).getAsString());
+ }
+ buf.append("]");
+ return buf.toString();
+ }
+}
http://git-wip-us.apache.org/repos/asf/drill/blob/40de8ca4/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/ScalarArrayReader.java
----------------------------------------------------------------------
diff --git a/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/ScalarArrayReader.java b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/ScalarArrayReader.java
new file mode 100644
index 0000000..d93e4a5
--- /dev/null
+++ b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/ScalarArrayReader.java
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.vector.accessor.reader;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.exec.vector.accessor.ColumnReaderIndex;
+import org.apache.drill.exec.vector.accessor.ObjectType;
+import org.apache.drill.exec.vector.accessor.ScalarElementReader;
+import org.apache.drill.exec.vector.complex.RepeatedValueVector;
+
+public class ScalarArrayReader extends AbstractArrayReader {
+
+ private final BaseElementReader elementReader;
+
+ private ScalarArrayReader(RepeatedValueVector vector,
+ BaseElementReader elementReader) {
+ super(vector);
+ this.elementReader = elementReader;
+ }
+
+ private ScalarArrayReader(VectorAccessor va,
+ BaseElementReader elementReader) {
+ super(va);
+ this.elementReader = elementReader;
+ }
+
+ public static ArrayObjectReader build(RepeatedValueVector vector,
+ BaseElementReader elementReader) {
+ elementReader.bindVector(vector.getDataVector());
+ return new ArrayObjectReader(new ScalarArrayReader(vector, elementReader));
+ }
+
+ public static ArrayObjectReader build(MajorType majorType, VectorAccessor va,
+ BaseElementReader elementReader) {
+ elementReader.bindVector(majorType, va);
+ return new ArrayObjectReader(new ScalarArrayReader(va, elementReader));
+ }
+
+ @Override
+ public void bindIndex(ColumnReaderIndex index) {
+ super.bindIndex(index);
+ FixedWidthElementReaderIndex fwElementIndex = new FixedWidthElementReaderIndex(baseIndex);
+ elementIndex = fwElementIndex;
+ elementReader.bindIndex(fwElementIndex);
+ }
+
+ @Override
+ public ObjectType entryType() {
+ return ObjectType.SCALAR;
+ }
+
+ @Override
+ public ScalarElementReader elements() {
+ return elementReader;
+ }
+
+ @Override
+ public void setPosn(int index) {
+ throw new IllegalStateException("setPosn() not supported for scalar arrays");
+ }
+
+ @Override
+ public Object getObject() {
+ List<Object> elements = new ArrayList<>();
+ for (int i = 0; i < size(); i++) {
+ elements.add(elementReader.getObject(i));
+ }
+ return elements;
+ }
+
+ @Override
+ public String getAsString() {
+ StringBuilder buf = new StringBuilder();
+ buf.append("[");
+ for (int i = 0; i < size(); i++) {
+ if (i > 0) {
+ buf.append( ", " );
+ }
+ buf.append(elementReader.getAsString(i));
+ }
+ buf.append("]");
+ return buf.toString();
+ }
+}
http://git-wip-us.apache.org/repos/asf/drill/blob/40de8ca4/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/VectorAccessor.java
----------------------------------------------------------------------
diff --git a/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/VectorAccessor.java b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/VectorAccessor.java
new file mode 100644
index 0000000..1cf2a19
--- /dev/null
+++ b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/VectorAccessor.java
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.vector.accessor.reader;
+
+import org.apache.drill.exec.vector.ValueVector;
+import org.apache.drill.exec.vector.accessor.ColumnReaderIndex;
+
+public interface VectorAccessor {
+ void bind(ColumnReaderIndex index);
+ ValueVector vector();
+}
http://git-wip-us.apache.org/repos/asf/drill/blob/40de8ca4/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/package-info.java
----------------------------------------------------------------------
diff --git a/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/package-info.java b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/package-info.java
new file mode 100644
index 0000000..a94d2e8
--- /dev/null
+++ b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/reader/package-info.java
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+/**
+ * Provides the reader hierarchy as explained in the API package.
+ * The only caveat is that a simplification is provided for arrays of
+ * scalar values: rather than a scalar reader for each value, the
+ * {#link ScalarElementReader} class provides access to the entire array
+ * via indexed get methods.
+ */
+
+package org.apache.drill.exec.vector.accessor.reader;
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/40de8ca4/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/writer/AbstractArrayWriter.java
----------------------------------------------------------------------
diff --git a/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/writer/AbstractArrayWriter.java b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/writer/AbstractArrayWriter.java
new file mode 100644
index 0000000..e6e29b4
--- /dev/null
+++ b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/writer/AbstractArrayWriter.java
@@ -0,0 +1,348 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.vector.accessor.writer;
+
+import org.apache.drill.exec.record.ColumnMetadata;
+import org.apache.drill.exec.vector.UInt4Vector;
+import org.apache.drill.exec.vector.accessor.ArrayWriter;
+import org.apache.drill.exec.vector.accessor.ColumnWriterIndex;
+import org.apache.drill.exec.vector.accessor.ObjectType;
+import org.apache.drill.exec.vector.accessor.ObjectWriter;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.ScalarWriter.ColumnWriterListener;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter.TupleWriterListener;
+import org.apache.drill.exec.vector.accessor.impl.HierarchicalFormatter;
+
+/**
+ * Writer for an array-valued column. This writer appends values: once a value
+ * is written, it cannot be changed. As a result, writer methods have no item
+ * index; each set advances the array to the next position.
+ * <p>
+ * This class represents the array as a whole. In practice that means building
+ * the offset vector. The array is associated with an element object that
+ * manages writing to the scalar, array or tuple that is the array element. Note
+ * that this representation makes little use of the methods in the "Repeated"
+ * vector class: instead it works directly with the offset and element vectors.
+ * <p>
+ * An array has a one-to-many relationship with its children. Starting an array
+ * prepares for writing the first element. Each element must be saved by calling
+ * <tt>endValue()</tt>. This is done automatically for scalars (since there is
+ * exactly one value per element), but must be done via the client code for
+ * arrays of arrays or tuples. Valid state transitions:
+ *
+ * <table border=1>
+ * <tr><th>Public API</th><th>Array Event</th><th>Offset Event</th><th>Element Event</th></tr>
+ * <tr><td>startBatch()</td>
+ * <td>startWrite()</td>
+ * <td>startWrite()</td>
+ * <td>startWrite()</td></tr>
+ * <tr><td>start() (new row)</td>
+ * <td>startRow()</td>
+ * <td>startRow()</td>
+ * <td>startRow()</td></tr>
+ * <tr><td>start() (without save)</td>
+ * <td>restartRow()</td>
+ * <td>restartRow()</td>
+ * <td>restartRow()</td></tr>
+ * <tr><td>save() (array)</td>
+ * <td>saveValue()</td>
+ * <td>saveValue()</td>
+ * <td>saveValue()</td></tr>
+ * <tr><td>save() (row)</td>
+ * <td colspan=3>See subclasses.</td></tr>
+ * <tr><td>harvest()</td>
+ * <td>endWrite()</td>
+ * <td>endWrite()</td>
+ * <td>endWrite()</td></tr>
+ * </table>
+ *
+ * Some items to note:
+ * <ul>
+ * <li>Batch and row events are passed to the element.</li>
+ * <li>Each element is saved via a call to {@link #save()} on the array.
+ * Without this call, the element value is discarded. This is necessary
+ * because the array always has an active element: no "startElement"
+ * method is necessary. This also means that any unsaved element values
+ * can be discarded simply by omitting a call to <tt>save()</tt>.</li>
+ * <li>Since elements must be saved individually, the call to
+ * {@link #saveRow()} <i>does not</i> call <tt>saveValue()</tt>. This
+ * is an important distinction between an array and a tuple.</li>
+ * <li>The offset and element writers are treated equally: the same events
+ * are passed to both.</li>
+ * </ul>
+ */
+
+public abstract class AbstractArrayWriter implements ArrayWriter, WriterEvents {
+
+ /**
+ * Object representation of an array writer.
+ */
+
+ public static class ArrayObjectWriter extends AbstractObjectWriter {
+
+ private AbstractArrayWriter arrayWriter;
+
+ public ArrayObjectWriter(ColumnMetadata schema, AbstractArrayWriter arrayWriter) {
+ super(schema);
+ this.arrayWriter = arrayWriter;
+ }
+
+ @Override
+ public ObjectType type() { return ObjectType.ARRAY; }
+
+ @Override
+ public void set(Object value) {
+ arrayWriter.setObject(value);
+ }
+
+ @Override
+ public ArrayWriter array() { return arrayWriter; }
+
+ @Override
+ public WriterEvents events() { return arrayWriter; }
+
+ @Override
+ public void bindListener(ColumnWriterListener listener) {
+ arrayWriter.bindListener(listener);
+ }
+
+ @Override
+ public void bindListener(TupleWriterListener listener) {
+ arrayWriter.bindListener(listener);
+ }
+
+ @Override
+ public void dump(HierarchicalFormatter format) {
+ format
+ .startObject(this)
+ .attribute("arrayWriter");
+ arrayWriter.dump(format);
+ format.endObject();
+ }
+ }
+
+ public static abstract class BaseArrayWriter extends AbstractArrayWriter {
+
+ /**
+ * Index into the vector of elements for a repeated vector.
+ * Keeps track of the current offset in terms of value positions.
+ * Forwards overflow events to the base index.
+ */
+
+ public class ArrayElementWriterIndex implements ColumnWriterIndex {
+
+ private int elementIndex;
+
+ public void reset() { elementIndex = 0; }
+
+ @Override
+ public int vectorIndex() { return elementIndex + offsetsWriter.nextOffset(); }
+
+ @Override
+ public int rowStartIndex() { return offsetsWriter.rowStartOffset(); }
+
+ public int arraySize() { return elementIndex; }
+
+ @Override
+ public void nextElement() { }
+
+ public final void next() { elementIndex++; }
+
+ public int valueStartOffset() { return offsetsWriter.nextOffset(); }
+
+ @Override
+ public void rollover() { }
+
+ @Override
+ public ColumnWriterIndex outerIndex() {
+ return outerIndex;
+ }
+
+ @Override
+ public String toString() {
+ return new StringBuilder()
+ .append("[")
+ .append(getClass().getSimpleName())
+ .append(" elementIndex = ")
+ .append(elementIndex)
+ .append("]")
+ .toString();
+ }
+ }
+
+ private final OffsetVectorWriter offsetsWriter;
+ private ColumnWriterIndex outerIndex;
+ protected ArrayElementWriterIndex elementIndex;
+
+ public BaseArrayWriter(UInt4Vector offsetVector, AbstractObjectWriter elementObjWriter) {
+ super(elementObjWriter);
+ offsetsWriter = new OffsetVectorWriter(offsetVector);
+ }
+
+ @Override
+ public void bindIndex(ColumnWriterIndex index) {
+ assert elementIndex != null;
+ outerIndex = index;
+ offsetsWriter.bindIndex(index);
+ elementObjWriter.events().bindIndex(elementIndex);
+ }
+
+ @Override
+ public ColumnWriterIndex writerIndex() { return outerIndex; }
+
+ @Override
+ public int size() { return elementIndex.arraySize(); }
+
+ @Override
+ public void startWrite() {
+ elementIndex.reset();
+ offsetsWriter.startWrite();
+ elementObjWriter.events().startWrite();
+ }
+
+ @Override
+ public void startRow() {
+
+ // Starting an outer value automatically starts the first
+ // element value. If no elements are written, then this
+ // inner start will just be ignored.
+
+ offsetsWriter.startRow();
+ elementIndex.reset();
+ elementObjWriter.events().startRow();
+ }
+
+ @Override
+ public void endArrayValue() {
+ offsetsWriter.setNextOffset(elementIndex.vectorIndex());
+ elementIndex.reset();
+ }
+
+ @Override
+ public void restartRow() {
+ offsetsWriter.restartRow();
+ elementIndex.reset();
+ elementObjWriter.events().restartRow();
+ }
+
+ @Override
+ public void saveRow() {
+ offsetsWriter.saveRow();
+ elementObjWriter.events().saveRow();
+ }
+
+ @Override
+ public void endWrite() {
+ offsetsWriter.endWrite();
+ elementObjWriter.events().endWrite();
+ }
+
+ @Override
+ public void preRollover() {
+ elementObjWriter.events().preRollover();
+ offsetsWriter.preRollover();
+ }
+
+ @Override
+ public void postRollover() {
+ elementObjWriter.events().postRollover();
+
+ // Reset the index after the vectors: the vectors
+ // need the old row start index from the index.
+
+ offsetsWriter.postRollover();
+ elementIndex.rollover();
+ }
+
+ @Override
+ public int lastWriteIndex() { return outerIndex.vectorIndex(); }
+
+ /**
+ * Return the writer for the offset vector for this array. Primarily used
+ * to handle overflow; other clients should not attempt to muck about with
+ * the offset vector directly.
+ *
+ * @return the writer for the offset vector associated with this array
+ */
+
+ @Override
+ public OffsetVectorWriter offsetWriter() { return offsetsWriter; }
+
+ @Override
+ public void bindListener(ColumnWriterListener listener) {
+ elementObjWriter.bindListener(listener);
+ }
+
+ @Override
+ public void bindListener(TupleWriterListener listener) {
+ elementObjWriter.bindListener(listener);
+ }
+
+ @Override
+ public void dump(HierarchicalFormatter format) {
+ format.extend();
+ super.dump(format);
+ format
+ .attribute("elementIndex", elementIndex.vectorIndex())
+ .attribute("offsetsWriter");
+ offsetsWriter.dump(format);
+ }
+ }
+
+ protected final AbstractObjectWriter elementObjWriter;
+
+ public AbstractArrayWriter(AbstractObjectWriter elementObjWriter) {
+ this.elementObjWriter = elementObjWriter;
+ }
+
+ @Override
+ public ObjectType entryType() {
+ return elementObjWriter.type();
+ }
+
+ @Override
+ public ObjectWriter entry() { return elementObjWriter; }
+
+ @Override
+ public ScalarWriter scalar() {
+ return elementObjWriter.scalar();
+ }
+
+ @Override
+ public TupleWriter tuple() {
+ return elementObjWriter.tuple();
+ }
+
+ @Override
+ public ArrayWriter array() {
+ return elementObjWriter.array();
+ }
+
+ public abstract void bindListener(ColumnWriterListener listener);
+ public abstract void bindListener(TupleWriterListener listener);
+ public abstract OffsetVectorWriter offsetWriter();
+
+ public void dump(HierarchicalFormatter format) {
+ format
+ .startObject(this)
+ .attribute("elementObjWriter");
+ elementObjWriter.dump(format);
+ format.endObject();
+ }
+}
http://git-wip-us.apache.org/repos/asf/drill/blob/40de8ca4/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/writer/AbstractFixedWidthWriter.java
----------------------------------------------------------------------
diff --git a/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/writer/AbstractFixedWidthWriter.java b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/writer/AbstractFixedWidthWriter.java
new file mode 100644
index 0000000..e49f92c
--- /dev/null
+++ b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/writer/AbstractFixedWidthWriter.java
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.vector.accessor.writer;
+
+import org.apache.drill.exec.memory.BaseAllocator;
+import org.apache.drill.exec.vector.ValueVector;
+import org.apache.drill.exec.vector.accessor.impl.HierarchicalFormatter;
+
+/**
+ * Base class for writers for fixed-width vectors. Handles common
+ * tasks, leaving the generated code to handle only type-specific
+ * operations.
+ */
+
+public abstract class AbstractFixedWidthWriter extends BaseScalarWriter {
+
+ public static abstract class BaseFixedWidthWriter extends AbstractFixedWidthWriter {
+
+ /**
+ * Buffer of zeros used to back-fill vector buffers with
+ * zeros.
+ */
+
+ private static final byte ZERO_BUF[] = new byte[256];
+
+ /**
+ * Determine the write index, growing, overflowing and back-filling
+ * the vector as needed.
+ * <p>
+ * This is a bit tricky. This method has side effects, by design.
+ * The current vector buffer, and buffer address, will change in
+ * this method when a vector grows or overflows. So, don't use this
+ * method in inline calls of the form<br><code>
+ * vector.getBuffer().doSomething(writeIndex());</code></br>
+ * The buffer obtained by <tt>getBuffer()</tt> can be different than
+ * the current buffer after <tt>writeIndex()</tt>.
+ *
+ * @return the index at which to write the current value
+ */
+
+ protected final int writeIndex() {
+
+ // "Fast path" for the normal case of no fills, no overflow.
+ // This is the only bounds check we want to do for the entire
+ // set operation.
+
+ // This is performance critical code; every operation counts.
+ // Please be thoughtful when changing the code.
+
+ int writeIndex = vectorIndex.vectorIndex();
+ if (lastWriteIndex + 1 < writeIndex || writeIndex >= capacity) {
+ writeIndex = prepareWrite(writeIndex);
+ }
+
+ // Track the last write location for zero-fill use next time around.
+
+ lastWriteIndex = writeIndex;
+ return writeIndex;
+ }
+
+ protected final int prepareWrite(int writeIndex) {
+
+ // Either empties must be filed or the vector is full.
+
+ writeIndex = resize(writeIndex);
+
+ // Fill empties to the write position.
+
+ fillEmpties(writeIndex);
+ return writeIndex;
+ }
+
+ /**
+ * Fill empties. This is required because the allocated memory is not
+ * zero-filled.
+ */
+
+ @Override
+ protected final void fillEmpties(final int writeIndex) {
+ final int width = width();
+ final int stride = ZERO_BUF.length / width;
+ int dest = lastWriteIndex + 1;
+ while (dest < writeIndex) {
+ int length = writeIndex - dest;
+ length = Math.min(length, stride);
+ drillBuf.unsafeCopyMemory(ZERO_BUF, 0, dest * width, length * width);
+ dest += length;
+ }
+ }
+ }
+
+ /**
+ * The largest position to which the writer has written data. Used to allow
+ * "fill-empties" (AKA "back-fill") of missing values one each value write
+ * and at the end of a batch. Note that this is the position of the last
+ * write, not the next write position. Starts at -1 (no last write).
+ */
+
+ protected int lastWriteIndex;
+
+ @Override
+ public void startWrite() {
+ setBuffer();
+ lastWriteIndex = -1;
+ }
+
+ public abstract int width();
+
+ @Override
+ protected final void setBuffer() {
+ drillBuf = vector().getBuffer();
+ capacity = drillBuf.capacity() / width();
+ }
+
+ protected final void mandatoryResize(final int writeIndex) {
+ if (writeIndex < capacity) {
+ return;
+ }
+
+ // Since some vectors start off as 0 length, set a
+ // minimum size to avoid silly thrashing on early rows.
+
+ final int size = BaseAllocator.nextPowerOfTwo(
+ Math.max((writeIndex + 1) * width(), MIN_BUFFER_SIZE));
+ realloc(size);
+ }
+
+ protected final int resize(final int writeIndex) {
+ if (writeIndex < capacity) {
+ return writeIndex;
+ }
+ final int width = width();
+
+ // Since some vectors start off as 0 length, set a
+ // minimum size to avoid silly thrashing on early rows.
+
+ final int size = BaseAllocator.nextPowerOfTwo(
+ Math.max((writeIndex + 1) * width, MIN_BUFFER_SIZE));
+
+ // Two cases: grow this vector or allocate a new one.
+
+ // Grow the vector -- or overflow if the growth would make the batch
+ // consume too much memory. The idea is that we grow vectors as they
+ // fit the available memory budget, then we fill those vectors until
+ // one of them needs more space. At that point we trigger overflow to
+ // a new set of vectors. Internal fragmentation will result, but this
+ // approach (along with proper initial vector sizing), minimizes that
+ // fragmentation.
+
+ if (size <= ValueVector.MAX_BUFFER_SIZE &&
+ canExpand(size - capacity * width)) {
+
+ // Optimized form of reAlloc() which does not zero memory, does not do
+ // bounds checks (since they were already done above). The write index
+ // and offset remain unchanged.
+
+ realloc(size);
+ } else {
+
+ // Allocate a new vector, or throw an exception if overflow is not
+ // supported. If overflow is supported, the callback will call
+ // endWrite(), which will fill empties, so no need to do that here.
+ // The call to endWrite() will also set the final writer index for the
+ // current vector. Then, bindVector() will be called to provide the new
+ // vector. The write index changes with the new vector.
+
+ overflowed();
+ }
+
+ // Call to resize may cause rollover, so reset write index
+ // afterwards.
+
+ return vectorIndex.vectorIndex();
+ }
+
+ @Override
+ public int lastWriteIndex() { return lastWriteIndex; }
+
+ @Override
+ public void skipNulls() {
+
+ // Pretend we've written up to the previous value.
+ // This will leave null values (as specified by the
+ // caller) uninitialized.
+
+ lastWriteIndex = vectorIndex.vectorIndex() - 1;
+ }
+
+ @Override
+ public void restartRow() {
+ lastWriteIndex = Math.min(lastWriteIndex, vectorIndex.vectorIndex() - 1);
+ }
+
+ @Override
+ public void preRollover() {
+ setValueCount(vectorIndex.rowStartIndex());
+ }
+
+ @Override
+ public void postRollover() {
+ int newIndex = Math.max(lastWriteIndex - vectorIndex.rowStartIndex(), -1);
+ startWrite();
+ lastWriteIndex = newIndex;
+ }
+
+ @Override
+ public void endWrite() {
+ setValueCount(vectorIndex.vectorIndex());
+ }
+
+ protected abstract void fillEmpties(int writeIndex);
+
+ public void setValueCount(int valueCount) {
+
+ // Done this way to avoid another drill buf access in value set path.
+ // Though this calls writeOffset(), which handles vector overflow,
+ // such overflow should never occur because here we are simply
+ // finalizing a position already set. However, the vector size may
+ // grow and the "missing" values may be zero-filled. Note that, in
+ // odd cases, the call to writeOffset() might cause the vector to
+ // resize (as part of filling empties), so grab the buffer AFTER
+ // the call to writeOffset().
+
+ mandatoryResize(valueCount - 1);
+ fillEmpties(valueCount);
+ vector().getBuffer().writerIndex(valueCount * width());
+
+ // Last write index is either the last value we just filled,
+ // or it is the last actual write, if this is an overflow
+ // situation.
+
+ lastWriteIndex = Math.max(lastWriteIndex, valueCount - 1);
+ }
+
+ @Override
+ public void dump(HierarchicalFormatter format) {
+ format.extend();
+ super.dump(format);
+ format
+ .attribute("lastWriteIndex", lastWriteIndex)
+ .endObject();
+ }
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/40de8ca4/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/writer/AbstractObjectWriter.java
----------------------------------------------------------------------
diff --git a/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/writer/AbstractObjectWriter.java b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/writer/AbstractObjectWriter.java
new file mode 100644
index 0000000..a8f1c64
--- /dev/null
+++ b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/writer/AbstractObjectWriter.java
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.vector.accessor.writer;
+
+import org.apache.drill.exec.record.ColumnMetadata;
+import org.apache.drill.exec.vector.accessor.ArrayWriter;
+import org.apache.drill.exec.vector.accessor.ObjectWriter;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.ScalarWriter.ColumnWriterListener;
+import org.apache.drill.exec.vector.accessor.TupleWriter;
+import org.apache.drill.exec.vector.accessor.TupleWriter.TupleWriterListener;
+import org.apache.drill.exec.vector.accessor.impl.HierarchicalFormatter;
+
+/**
+ * Abstract base class for the object layer in writers. This class acts
+ * as the glue between a column and the data type of that column, per the
+ * JSON model which Drill uses. This base class provides stubs for most
+ * methods so that type-specific subclasses can simply fill in the bits
+ * needed for that particular class.
+ */
+
+public abstract class AbstractObjectWriter implements ObjectWriter {
+
+ private ColumnMetadata schema;
+
+ public AbstractObjectWriter(ColumnMetadata schema) {
+ this.schema = schema;
+ }
+
+ @Override
+ public ColumnMetadata schema() { return schema; }
+
+ @Override
+ public ScalarWriter scalar() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public TupleWriter tuple() {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public ArrayWriter array() {
+ throw new UnsupportedOperationException();
+ }
+
+ public abstract WriterEvents events();
+
+ @Override
+ public void bindListener(ColumnWriterListener listener) { }
+
+ @Override
+ public void bindListener(TupleWriterListener listener) { }
+
+ public abstract void dump(HierarchicalFormatter format);
+}
http://git-wip-us.apache.org/repos/asf/drill/blob/40de8ca4/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/writer/AbstractScalarWriter.java
----------------------------------------------------------------------
diff --git a/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/writer/AbstractScalarWriter.java b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/writer/AbstractScalarWriter.java
new file mode 100644
index 0000000..c02e2d9
--- /dev/null
+++ b/exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/writer/AbstractScalarWriter.java
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.vector.accessor.writer;
+
+import java.math.BigDecimal;
+
+import org.apache.drill.exec.record.ColumnMetadata;
+import org.apache.drill.exec.vector.BaseDataValueVector;
+import org.apache.drill.exec.vector.accessor.ObjectType;
+import org.apache.drill.exec.vector.accessor.ScalarWriter;
+import org.apache.drill.exec.vector.accessor.impl.HierarchicalFormatter;
+import org.joda.time.Period;
+
+/**
+ * Column writer implementation that acts as the basis for the
+ * generated, vector-specific implementations. All set methods
+ * throw an exception; subclasses simply override the supported
+ * method(s).
+ */
+
+public abstract class AbstractScalarWriter implements ScalarWriter, WriterEvents {
+
+ public static class ScalarObjectWriter extends AbstractObjectWriter {
+
+ private AbstractScalarWriter scalarWriter;
+
+ public ScalarObjectWriter(ColumnMetadata schema, AbstractScalarWriter scalarWriter) {
+ super(schema);
+ this.scalarWriter = scalarWriter;
+ }
+
+ @Override
+ public ObjectType type() { return ObjectType.SCALAR; }
+
+ @Override
+ public void set(Object value) { scalarWriter.setObject(value); }
+
+ @Override
+ public ScalarWriter scalar() { return scalarWriter; }
+
+ @Override
+ public WriterEvents events() { return scalarWriter; }
+
+ @Override
+ public void bindListener(ColumnWriterListener listener) {
+ scalarWriter.bindListener(listener);
+ }
+
+ @Override
+ public void dump(HierarchicalFormatter format) {
+ format
+ .startObject(this)
+ .attribute("scalarWriter");
+ scalarWriter.dump(format);
+ format.endObject();
+ }
+ }
+
+ public abstract BaseDataValueVector vector();
+
+ @Override
+ public void startWrite() { }
+
+ @Override
+ public void startRow() { }
+
+ @Override
+ public void endArrayValue() { }
+
+ @Override
+ public void saveRow() { }
+
+ @Override
+ public void setObject(Object value) {
+ if (value == null) {
+ setNull();
+ } else if (value instanceof Integer) {
+ setInt((Integer) value);
+ } else if (value instanceof Long) {
+ setLong((Long) value);
+ } else if (value instanceof String) {
+ setString((String) value);
+ } else if (value instanceof BigDecimal) {
+ setDecimal((BigDecimal) value);
+ } else if (value instanceof Period) {
+ setPeriod((Period) value);
+ } else if (value instanceof byte[]) {
+ byte[] bytes = (byte[]) value;
+ setBytes(bytes, bytes.length);
+ } else if (value instanceof Byte) {
+ setInt((Byte) value);
+ } else if (value instanceof Short) {
+ setInt((Short) value);
+ } else if (value instanceof Double) {
+ setDouble((Double) value);
+ } else if (value instanceof Float) {
+ setDouble((Float) value);
+ } else {
+ throw new IllegalArgumentException("Unsupported type " +
+ value.getClass().getSimpleName());
+ }
+ }
+
+ public void dump(HierarchicalFormatter format) {
+ format
+ .startObject(this)
+ .attributeIdentity("vector", vector())
+ .attribute("schema", vector().getField())
+ .endObject();
+ }
+}