You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by paul-rogers <gi...@git.apache.org> on 2017/03/14 04:07:31 UTC
[GitHub] drill pull request #783: DRILL-5324: Provide simplified column reader/writer...
GitHub user paul-rogers opened a pull request:
https://github.com/apache/drill/pull/783
DRILL-5324: Provide simplified column reader/writer for use in tests
The new "sub-operator" unit test framework provides simple ways to create row sets in code. This PR includes the column accessor code:
* Interfaces for column accessors
* Template for generated implementations
* Base implementation used by the generated code
* Factory class to create the proper reader or writer given a major
type (type and cardinality)
* Utilities for generic access, type conversions, etc.
Many vector types can be mapped to an int for get and set. One key
exception are the decimal types: decimals, by definition, require a
different representation. In Java, that is `BigDecimal`. Added get, set
and setSafe accessors as required for each decimal type that uses
`BigDecimal` to hold data.
Work remains to be done on other complex types: intervals and so on.
This will be added incrementally as work proceeds.
The generated code builds on the `valueVectorTypes.tdd` file, adding
additional properties needed to generate the accessors.
The PR also includes a number of code cleanups done while reviewing
existing code. In particular `DecimalUtility` was very roughly
formatted and thus hard to follow.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/paul-rogers/drill DRILL-5324
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/783.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #783
----
commit eb0b8bc33aeea27fd0aae582d19297bd0bda92e1
Author: Paul Rogers <pr...@maprtech.com>
Date: 2017-03-11T07:03:23Z
The PR includes the column accessor code:
* Interfaces described above
* Generated implementations
* Base implementation used by the generated code
* Factory class to create the proper reader or writer given a major
type (type and cardinality)
* Utilities for generic access, type conversions, etc.
Many vector types can be mapped to an int for get and set. One key
exception are the decimal types: decimals, by definition, require a
different representation. In Java, that is `BigDecimal`. Added get, set
and setSafe accessors as required for each decimal type that uses
`BigDecimal` to hold data.
Work remains to be done on other complex types: intervals and so on.
This will be added incrementally as work proceeds.
The generated code builds on the `valueVectorTypes.tdd` file, adding
additional properties needed to generate the accessors.
The PR also includes a number of code cleanups done while reviewing
existing code. In particular `DecimalUtility` was very roughly
formatted and thus hard to follow.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request #783: DRILL-5324: Provide simplified column reader/writer...
Posted by bitblender <gi...@git.apache.org>.
Github user bitblender commented on a diff in the pull request:
https://github.com/apache/drill/pull/783#discussion_r111495333
--- Diff: exec/vector/src/main/java/org/apache/drill/exec/vector/accessor/ScalarWriter.java ---
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.vector.accessor;
+
+import java.math.BigDecimal;
+
+import org.joda.time.Period;
+
--- End diff --
Adding a comment here saying that Writers use an implicit index would make it easier for readers to understand why there is no index parameter and why the ScalarWriter interface can be inherited-from by both the ColumnWriter and the ArrayWriter
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request #783: DRILL-5324: Provide simplified column reader/writer...
Posted by bitblender <gi...@git.apache.org>.
Github user bitblender commented on a diff in the pull request:
https://github.com/apache/drill/pull/783#discussion_r111289506
--- Diff: exec/vector/src/main/codegen/templates/ColumnAccessors.java ---
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+<@pp.dropOutputFile />
+<@pp.changeOutputFile name="/org/apache/drill/exec/vector/accessor/ColumnAccessors.java" />
+<#include "/@includes/license.ftl" />
+<#macro getType label>
+ @Override
+ public ValueType valueType() {
+ <#if label == "Int">
+ return ValueType.INTEGER;
+ <#else>
+ return ValueType.${label?upper_case};
+ </#if>
+ }
+</#macro>
+<#macro bindReader prefix drillType>
+ <#if drillType = "Decimal9" || drillType == "Decimal18">
+ private MaterializedField field;
+ </#if>
+ private ${prefix}${drillType}Vector.Accessor accessor;
+
+ @Override
+ public void bind(RowIndex vectorIndex, ValueVector vector) {
+ bind(vectorIndex);
+ <#if drillType = "Decimal9" || drillType == "Decimal18">
+ field = vector.getField();
+ </#if>
+ accessor = ((${prefix}${drillType}Vector) vector).getAccessor();
+ }
+
+ <#if drillType = "Decimal9" || drillType == "Decimal18">
+ @Override
+ public void bind(RowIndex vectorIndex, MaterializedField field, VectorAccessor va) {
+ bind(vectorIndex, field, va);
+ this.field = field;
+ }
+
+ </#if>
+ private ${prefix}${drillType}Vector.Accessor accessor() {
+ if (vectorAccessor == null) {
+ return accessor;
+ } else {
+ return ((${prefix}${drillType}Vector) vectorAccessor.vector()).getAccessor();
+ }
+ }
+</#macro>
+<#macro get drillType accessorType label isArray>
+ @Override
+ public ${accessorType} get${label}(<#if isArray>int index</#if>) {
+ <#if isArray>
+ <#assign index=", index"/>
+ <#assign getObject="getSingleObject">
+ <#else>
+ <#assign index=""/>
+ <#assign getObject="getObject">
+ </#if>
+ <#if drillType == "VarChar">
+ return new String(accessor().get(vectorIndex.index()${index}), Charsets.UTF_8);
+ <#elseif drillType == "Var16Char">
+ return new String(accessor().get(vectorIndex.index()${index}), Charsets.UTF_16);
+ <#elseif drillType == "VarBinary">
+ return accessor().get(vectorIndex.index()${index});
+ <#elseif drillType == "Decimal9" || drillType == "Decimal18">
+ return DecimalUtility.getBigDecimalFromPrimitiveTypes(
+ accessor().get(vectorIndex.index()${index}),
+ field.getScale(),
+ field.getPrecision());
+ <#elseif accessorType == "Decimal18">
+ return DecimalUtilities.getBigDecimalFromPrimitiveTypes(accessor().${getObject}(vectorIndex.index()${index});
--- End diff --
As discusses offline, this seems to be deadcode as there is no DecimalUtilities class in the Drill source base.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill issue #783: DRILL-5324: Provide simplified column reader/writer for te...
Posted by amansinha100 <gi...@git.apache.org>.
Github user amansinha100 commented on the issue:
https://github.com/apache/drill/pull/783
This PR combines the main changes for simplified column readers/writers with the code cleanup/hygiene changes which made it harder to review. For future, please separate the hygiene changes into a separate PR.
Rest LGTM. +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill issue #783: DRILL-5324: Provide simplified column reader/writer for te...
Posted by amansinha100 <gi...@git.apache.org>.
Github user amansinha100 commented on the issue:
https://github.com/apache/drill/pull/783
@paul-rogers can you squash the commits for merging ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request #783: DRILL-5324: Provide simplified column reader/writer...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/drill/pull/783
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill issue #783: DRILL-5324: Provide simplified column reader/writer for te...
Posted by paul-rogers <gi...@git.apache.org>.
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/783
Done. Also rebased on master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---