You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hawq.apache.org by sansanichfb <gi...@git.apache.org> on 2016/04/29 02:34:25 UTC
[GitHub] incubator-hawq pull request: HAWQ-703. Serialize HCatalog Complex ...
GitHub user sansanichfb opened a pull request:
https://github.com/apache/incubator-hawq/pull/633
HAWQ-703. Serialize HCatalog Complex Types to plain text (as Hive pro…
…file).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/incubator-hawq HAWQ-703
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-hawq/pull/633.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #633
----
commit 8c5e6f8b2408329f250ad6d69f0487dc5999b597
Author: Oleksandr Diachenko <od...@pivotal.io>
Date: 2016-04-22T23:34:42Z
HAWQ-703. Serialize HCatalog Complex Types to plain text (as Hive profile).
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-703. Serialize HCatalog Complex ...
Posted by kavinderd <gi...@git.apache.org>.
Github user kavinderd commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/633#discussion_r62552154
--- Diff: pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/utilities/HiveUtilities.java ---
@@ -102,80 +103,49 @@ public static Table getHiveTable(HiveMetaStoreClient client, Metadata.Item itemN
* <li>{@code decimal(precision, scale) -> numeric(precision, scale)}</li>
* <li>{@code varchar(size) -> varchar(size)}</li>
* <li>{@code char(size) -> bpchar(size)}</li>
+ * <li>{@code array<dataType> -> text}</li>
+ * <li>{@code map<keyDataType, valueDataType> -> text}</li>
+ * <li>{@code struct<field1:dataType,...,fieldN:dataType> -> text}</li>
+ * <li>{@code uniontype<...> -> text}</li>
* </ul>
*
- * @param hiveColumn hive column schema
+ * @param hiveColumn
+ * hive column schema
* @return field with mapped HAWQ type and modifiers
- * @throws UnsupportedTypeException if the column type is not supported
+ * @throws UnsupportedTypeException
+ * if the column type is not supported
+ * @see EnumHiveToHawqType
*/
public static Metadata.Field mapHiveType(FieldSchema hiveColumn) throws UnsupportedTypeException {
String fieldName = hiveColumn.getName();
- String hiveType = hiveColumn.getType();
- String mappedType;
- String[] modifiers = null;
+ String hiveType = hiveColumn.getType(); // Type name and modifiers if any
+ String hiveTypeName; // Type name
+ String[] modifiers = null; // Modifiers
+ EnumHiveToHawqType hiveToHawqType = EnumHiveToHawqType.getHiveToHawqType(hiveType);
+ EnumHawqType hawqType = hiveToHawqType.getHawqType();
- // check parameterized types:
- if (hiveType.startsWith("varchar(") ||
- hiveType.startsWith("char(")) {
- String[] toks = hiveType.split("[(,)]");
- if (toks.length != 2) {
- throw new UnsupportedTypeException( "HAWQ does not support type " + hiveType + " (Field " + fieldName + "), " +
- "expected type of the form <type name>(<parameter>)");
- }
- mappedType = toks[0];
- if (mappedType.equals("char")) {
- mappedType = "bpchar";
- }
- modifiers = new String[] {toks[1]};
- } else if (hiveType.startsWith("decimal(")) {
- String[] toks = hiveType.split("[(,)]");
- if (toks.length != 3) {
- throw new UnsupportedTypeException( "HAWQ does not support type " + hiveType + " (Field " + fieldName + "), " +
- "expected type of the form <type name>(<parameter>,<parameter>)");
+ if (hiveToHawqType.getSplitExpression() != null) {
+ String[] tokens = hiveType.split(hiveToHawqType.getSplitExpression());
+ hiveTypeName = tokens[0];
--- End diff --
Since you already execute similar logic to parse the tokens in `getHiveToHawqType()` why not set `tokens` as a field on `EnumHiveToHawqType` to consolidate the code to one place?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-703. Serialize HCatalog Complex ...
Posted by hornn <gi...@git.apache.org>.
Github user hornn commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/633#discussion_r61818778
--- Diff: pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/utilities/HiveUtilities.java ---
@@ -88,94 +90,61 @@ public static Table getHiveTable(HiveMetaStoreClient client, Metadata.Item itemN
* Unsupported types will result in an exception.
* <br>
* The supported mappings are:<ul>
- * <li>{@code tinyint -> int2}</li>
- * <li>{@code smallint -> int2}</li>
- * <li>{@code int -> int4}</li>
- * <li>{@code bigint -> int8}</li>
- * <li>{@code boolean -> bool}</li>
- * <li>{@code float -> float4}</li>
- * <li>{@code double -> float8}</li>
- * <li>{@code string -> text}</li>
- * <li>{@code binary -> bytea}</li>
- * <li>{@code timestamp -> timestamp}</li>
- * <li>{@code date -> date}</li>
- * <li>{@code decimal(precision, scale) -> numeric(precision, scale)}</li>
- * <li>{@code varchar(size) -> varchar(size)}</li>
- * <li>{@code char(size) -> bpchar(size)}</li>
+ * <li>{@code tinyint -> int2}</li>
--- End diff --
indentation?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-703. Serialize HCatalog Complex ...
Posted by shivzone <gi...@git.apache.org>.
Github user shivzone commented on the pull request:
https://github.com/apache/incubator-hawq/pull/633#issuecomment-218238748
nice job with the refactoring of hawq types !
+!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-703. Serialize HCatalog Complex ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-hawq/pull/633
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-703. Serialize HCatalog Complex ...
Posted by hornn <gi...@git.apache.org>.
Github user hornn commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/633#discussion_r61818073
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/Metadata.java ---
@@ -67,36 +68,43 @@ public String toString() {
}
/**
- * Class representing item field - name and type.
+ * Class representing item field - name, type, source type, modifiers.
+ * Type - exposed type of field
+ * Source type - type of field in underlying source
+ * Modifiers - additional attributes which describe type or field
*/
public static class Field {
private String name;
- private String type; // TODO: change to enum
+ private EnumHawqType type; // field type which PXF exposes
+ private String sourceType; // filed type PXF reads from
private String[] modifiers; // type modifiers, optional field
- public Field(String name, String type) {
-
- if (StringUtils.isBlank(name) || StringUtils.isBlank(type)) {
- throw new IllegalArgumentException("Field name and type cannot be empty");
- }
-
- this.name = name;
- this.type = type;
+ public Field(String name, EnumHawqType type, String sourceType) {
+ if (StringUtils.isBlank(name) || StringUtils.isBlank(type.getTypeName())
--- End diff --
type could be null, right? maybe just check that it's not null - it probably can't be an empty string because it's an enum.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-703. Serialize HCatalog Complex ...
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/633#discussion_r61972977
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/Metadata.java ---
@@ -67,36 +68,43 @@ public String toString() {
}
/**
- * Class representing item field - name and type.
+ * Class representing item field - name, type, source type, modifiers.
+ * Type - exposed type of field
+ * Source type - type of field in underlying source
+ * Modifiers - additional attributes which describe type or field
*/
public static class Field {
private String name;
- private String type; // TODO: change to enum
+ private EnumHawqType type; // field type which PXF exposes
+ private String sourceType; // filed type PXF reads from
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-703. Serialize HCatalog Complex ...
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/633#discussion_r62565599
--- Diff: pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/utilities/EnumHiveToHawqType.java ---
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hawq.pxf.plugins.hive.utilities;
+
+import org.apache.hawq.pxf.api.utilities.EnumHawqType;
+import org.apache.hawq.pxf.api.UnsupportedTypeException;
+
+/**
+ *
+ * Hive types, which are supported by plugin, mapped to HAWQ's types
+ * @see EnumHawqType
+ */
+public enum EnumHiveToHawqType {
+
+ TinyintType("tinyint", EnumHawqType.Int2Type),
+ SmallintType("smallint", EnumHawqType.Int2Type),
+ IntType("int", EnumHawqType.Int4Type),
+ BigintType("bigint", EnumHawqType.Int8Type),
+ BooleanType("boolean", EnumHawqType.BoolType),
+ FloatType("float", EnumHawqType.Float4Type),
+ DoubleType("double", EnumHawqType.Float8Type),
+ StringType("string", EnumHawqType.TextType),
+ BinaryType("binary", EnumHawqType.ByteaType),
+ TimestampType("timestamp", EnumHawqType.TimestampType),
+ DateType("date", EnumHawqType.DateType),
+ DecimalType("decimal", EnumHawqType.NumericType, "[(,)]"),
+ VarcharType("varchar", EnumHawqType.VarcharType, "[(,)]"),
+ CharType("char", EnumHawqType.BpcharType, "[(,)]"),
+ ArrayType("array", EnumHawqType.TextType, "[<,>]"),
+ MapType("map", EnumHawqType.TextType, "[<,>]"),
+ StructType("struct", EnumHawqType.TextType, "[<,>]"),
+ UnionType("uniontype", EnumHawqType.TextType, "[<,>]");
+
+ private String typeName;
+ private EnumHawqType hawqType;
+ private String splitExpression;
+
+ EnumHiveToHawqType(String typeName, EnumHawqType hawqType) {
+ this.typeName = typeName;
+ this.hawqType = hawqType;
+ }
+
+ EnumHiveToHawqType(String typeName, EnumHawqType hawqType, String splitExpression) {
+ this(typeName, hawqType);
+ this.splitExpression = splitExpression;
+ }
+
+ /**
+ *
+ * @return name of type
+ */
+ public String getTypeName() {
+ return this.typeName;
+ }
+
+ /**
+ *
+ * @return corresponding HAWQ type
+ */
+ public EnumHawqType getHawqType() {
+ return this.hawqType;
+ }
+
+ /**
+ *
+ * @return split by expression
+ */
+ public String getSplitExpression() {
+ return this.splitExpression;
+ }
+
+ /**
+ * Returns Hive to HAWQ type mapping entry for given Hive type
+ *
+ * @param hiveType full Hive type with modifiers, for example - decimal(10, 0), char(5), binary, array<string>, map<string,float> etc
+ * @return corresponding Hive to HAWQ type mapping entry
+ * @throws UnsupportedTypeException if there is no corresponding HAWQ type
+ */
+ public static EnumHiveToHawqType getHiveToHawqType(String hiveType) {
+ for (EnumHiveToHawqType t : values()) {
+ String hiveTypeName = hiveType;
+ String splitExpression = t.getSplitExpression();
+ if (splitExpression != null) {
+ String[] tokens = hiveType.split(splitExpression);
+ hiveTypeName = tokens[0];
+ }
+
+ if (t.getTypeName().toLowerCase().equals(hiveTypeName.toLowerCase())) {
--- End diff --
As it is enum, and instances creation are controlled by itself, getTypeName() will always return not null value
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-703. Serialize HCatalog Complex ...
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/633#discussion_r61972982
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/Metadata.java ---
@@ -67,36 +68,43 @@ public String toString() {
}
/**
- * Class representing item field - name and type.
+ * Class representing item field - name, type, source type, modifiers.
+ * Type - exposed type of field
+ * Source type - type of field in underlying source
+ * Modifiers - additional attributes which describe type or field
*/
public static class Field {
private String name;
- private String type; // TODO: change to enum
+ private EnumHawqType type; // field type which PXF exposes
+ private String sourceType; // filed type PXF reads from
private String[] modifiers; // type modifiers, optional field
- public Field(String name, String type) {
-
- if (StringUtils.isBlank(name) || StringUtils.isBlank(type)) {
- throw new IllegalArgumentException("Field name and type cannot be empty");
- }
-
- this.name = name;
- this.type = type;
+ public Field(String name, EnumHawqType type, String sourceType) {
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-703. Serialize HCatalog Complex ...
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/633#discussion_r61975316
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/Metadata.java ---
@@ -67,36 +68,43 @@ public String toString() {
}
/**
- * Class representing item field - name and type.
+ * Class representing item field - name, type, source type, modifiers.
+ * Type - exposed type of field
+ * Source type - type of field in underlying source
+ * Modifiers - additional attributes which describe type or field
*/
public static class Field {
private String name;
- private String type; // TODO: change to enum
+ private EnumHawqType type; // field type which PXF exposes
+ private String sourceType; // filed type PXF reads from
private String[] modifiers; // type modifiers, optional field
- public Field(String name, String type) {
-
- if (StringUtils.isBlank(name) || StringUtils.isBlank(type)) {
- throw new IllegalArgumentException("Field name and type cannot be empty");
- }
-
- this.name = name;
- this.type = type;
+ public Field(String name, EnumHawqType type, String sourceType) {
+ if (StringUtils.isBlank(name) || StringUtils.isBlank(type.getTypeName())
--- End diff --
Makes sense, updated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-703. Serialize HCatalog Complex ...
Posted by hornn <gi...@git.apache.org>.
Github user hornn commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/633#discussion_r61826885
--- Diff: src/bin/psql/describe.c ---
@@ -4263,8 +4263,13 @@ describePxfTable(const char *profile, const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
printTableContent cont;
int cols = 0;
+ if (verbose)
+ {
--- End diff --
remove brackets here or add them to the else block
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-703. Serialize HCatalog Complex ...
Posted by hornn <gi...@git.apache.org>.
Github user hornn commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/633#discussion_r61818664
--- Diff: pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/utilities/EnumHiveToHawqType.java ---
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hawq.pxf.plugins.hive.utilities;
+
+import org.apache.hawq.pxf.api.utilities.EnumHawqType;
+import org.apache.hawq.pxf.api.UnsupportedTypeException;
+
+/**
+ *
+ * Hive types, which are supported by plugin, mapped to HAWQ's types
+ * @see EnumHawqType
+ */
+public enum EnumHiveToHawqType {
+
+ TinyintType("tinyint", EnumHawqType.Int2Type),
+ SmallintType("smallint", EnumHawqType.Int2Type),
+ IntType("int", EnumHawqType.Int4Type),
+ BigintType("bigint", EnumHawqType.Int8Type),
+ BooleanType("boolean", EnumHawqType.BoolType),
+ FloatType("float", EnumHawqType.Float4Type),
+ DoubleType("double", EnumHawqType.Float8Type),
+ StringType("string", EnumHawqType.TextType),
+ BinaryType("binary", EnumHawqType.ByteaType),
+ TimestampType("timestamp", EnumHawqType.TimestampType),
+ DateType("date", EnumHawqType.DateType),
+ DecimalType("decimal", EnumHawqType.NumericType, "[(,)]"),
+ VarcharType("varchar", EnumHawqType.VarcharType, "[(,)]"),
+ CharType("char", EnumHawqType.BpcharType, "[(,)]"),
+ ArrayType("array", EnumHawqType.TextType, "[<,>]"),
+ MapType("map", EnumHawqType.TextType, "[<,>]"),
+ StructType("struct", EnumHawqType.TextType, "[<,>]"),
+ UnionType("uniontype", EnumHawqType.TextType, "[<,>]");
+
+ private String typeName;
+ private EnumHawqType hawqType;
+ private String splitExpression;
+
+ EnumHiveToHawqType(String typeName, EnumHawqType hawqType) {
+ this.typeName = typeName;
+ this.hawqType = hawqType;
+ }
+
+ EnumHiveToHawqType(String typeName, EnumHawqType hawqType, String splitExpression) {
+ this(typeName, hawqType);
+ this.splitExpression = splitExpression;
+ }
+
+ /**
+ *
+ * @return name of type
+ */
+ public String getTypeName() {
+ return this.typeName;
+ }
+
+ /**
+ *
+ * @return corresponding HAWQ type
+ */
+ public EnumHawqType getHawqType() {
+ return this.hawqType;
+ }
+
+ /**
+ *
+ * @return split by expression
+ */
+ public String getSplitExpression() {
+ return this.splitExpression;
+ }
+
+ /**
+ * Returns Hive to HAWQ type mapping entry for given Hive type
+ *
+ * @param hiveType full Hive type with modifiers, for example - decimal(10, 0), char(5), binary, array<string>, map<string,float> etc
+ * @return corresponding Hive to HAWQ type mapping entry
+ * @throws UnsupportedTypeException if there is no corresponding HAWQ type
+ */
+ public static EnumHiveToHawqType getHiveToHawqType(String hiveType) {
+ for (EnumHiveToHawqType t : values()) {
+ String hiveTypeName = hiveType;
+ if (t.getSplitExpression() != null) {
--- End diff --
minor - since t.getSplitExpression() is called twice here, it can be saved in a variable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-703. Serialize HCatalog Complex ...
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/633#discussion_r61975914
--- Diff: pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/utilities/HiveUtilities.java ---
@@ -88,94 +90,61 @@ public static Table getHiveTable(HiveMetaStoreClient client, Metadata.Item itemN
* Unsupported types will result in an exception.
* <br>
* The supported mappings are:<ul>
- * <li>{@code tinyint -> int2}</li>
- * <li>{@code smallint -> int2}</li>
- * <li>{@code int -> int4}</li>
- * <li>{@code bigint -> int8}</li>
- * <li>{@code boolean -> bool}</li>
- * <li>{@code float -> float4}</li>
- * <li>{@code double -> float8}</li>
- * <li>{@code string -> text}</li>
- * <li>{@code binary -> bytea}</li>
- * <li>{@code timestamp -> timestamp}</li>
- * <li>{@code date -> date}</li>
- * <li>{@code decimal(precision, scale) -> numeric(precision, scale)}</li>
- * <li>{@code varchar(size) -> varchar(size)}</li>
- * <li>{@code char(size) -> bpchar(size)}</li>
+ * <li>{@code tinyint -> int2}</li>
--- End diff --
thanks, fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-703. Serialize HCatalog Complex ...
Posted by shivzone <gi...@git.apache.org>.
Github user shivzone commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/633#discussion_r61800082
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/Metadata.java ---
@@ -67,36 +68,43 @@ public String toString() {
}
/**
- * Class representing item field - name and type.
+ * Class representing item field - name, type, source type, modifiers.
+ * Type - exposed type of field
+ * Source type - type of field in underlying source
+ * Modifiers - additional attributes which describe type or field
*/
public static class Field {
private String name;
- private String type; // TODO: change to enum
+ private EnumHawqType type; // field type which PXF exposes
+ private String sourceType; // filed type PXF reads from
private String[] modifiers; // type modifiers, optional field
- public Field(String name, String type) {
-
- if (StringUtils.isBlank(name) || StringUtils.isBlank(type)) {
- throw new IllegalArgumentException("Field name and type cannot be empty");
- }
-
- this.name = name;
- this.type = type;
+ public Field(String name, EnumHawqType type, String sourceType) {
--- End diff --
Correct indentation in Field class
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-703. Serialize HCatalog Complex ...
Posted by hornn <gi...@git.apache.org>.
Github user hornn commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/633#discussion_r61819180
--- Diff: pxf/pxf-service/src/test/java/org/apache/hawq/pxf/service/MetadataResponseFormatterTest.java ---
@@ -85,26 +86,45 @@ public void formatResponseStringWithModifiers() throws Exception {
List<Metadata.Field> fields = new ArrayList<Metadata.Field>();
Metadata.Item itemName = new Metadata.Item("default", "table1");
Metadata metadata = new Metadata(itemName, fields);
- fields.add(new Metadata.Field("field1", "int"));
- fields.add(new Metadata.Field("field2", "numeric",
+ fields.add(new Metadata.Field("field1", EnumHawqType.Int8Type, "bigint"));
+ fields.add(new Metadata.Field("field2", EnumHawqType.NumericType, "decimal",
new String[] {"1349", "1789"}));
- fields.add(new Metadata.Field("field3", "char",
+ fields.add(new Metadata.Field("field3", EnumHawqType.BpcharType, "char",
new String[] {"50"}));
metadataList.add(metadata);
response = MetadataResponseFormatter.formatResponse(metadataList, "path.file");
StringBuilder expected = new StringBuilder("{\"PXFMetadata\":[{");
expected.append("\"item\":{\"path\":\"default\",\"name\":\"table1\"},")
.append("\"fields\":[")
- .append("{\"name\":\"field1\",\"type\":\"int\"},")
- .append("{\"name\":\"field2\",\"type\":\"numeric\",\"modifiers\":[\"1349\",\"1789\"]},")
- .append("{\"name\":\"field3\",\"type\":\"char\",\"modifiers\":[\"50\"]}")
+ .append("{\"name\":\"field1\",\"type\":\"int8\",\"sourceType\":\"bigint\"},")
+ .append("{\"name\":\"field2\",\"type\":\"numeric\",\"sourceType\":\"decimal\",\"modifiers\":[\"1349\",\"1789\"]},")
+ .append("{\"name\":\"field3\",\"type\":\"bpchar\",\"sourceType\":\"char\",\"modifiers\":[\"50\"]}")
.append("]}]}");
assertEquals(expected.toString(), convertResponseToString(response));
}
@Test
+ public void formatResponseStringWithSourceType() throws Exception {
+ List<Metadata> metadataList = new ArrayList<Metadata>();
+ List<Metadata.Field> fields = new ArrayList<Metadata.Field>();
+ Metadata.Item itemName = new Metadata.Item("default", "table1");
+ Metadata metadata = new Metadata(itemName, fields);
+ fields.add(new Metadata.Field("field1", EnumHawqType.Float8Type, "double"));
+ metadataList.add(metadata);
+
+ response = MetadataResponseFormatter.formatResponse(metadataList, "path.file");
+ StringBuilder expected = new StringBuilder("{\"PXFMetadata\":[{");
+ expected.append("\"item\":{\"path\":\"default\",\"name\":\"table1\"},")
+ .append("\"fields\":[")
+ .append("{\"name\":\"field1\",\"type\":\"float8\",\"sourceType\":\"double\"}")
+ .append("]}]}");
+
+// assertEquals(expected.toString(), convertResponseToString(response));
--- End diff --
uncomment?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-703. Serialize HCatalog Complex ...
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/633#discussion_r61976525
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/utilities/EnumHawqType.java ---
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hawq.pxf.api.utilities;
+
+import java.io.IOException;
+import org.codehaus.jackson.JsonGenerator;
+import org.codehaus.jackson.map.JsonSerializer;
+import org.codehaus.jackson.map.annotate.JsonSerialize;
+import org.codehaus.jackson.map.SerializerProvider;
+import org.codehaus.jackson.JsonProcessingException;
+
+class EnumHawqTypeSerializer extends JsonSerializer<EnumHawqType> {
+
+ @Override
+ public void serialize(EnumHawqType value, JsonGenerator generator,
+ SerializerProvider provider) throws IOException,
+ JsonProcessingException {
+ generator.writeString(value.getTypeName());
--- End diff --
Enum instances are not storing actual modifiers, just number and type of modifiers.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-703. Serialize HCatalog Complex ...
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/633#discussion_r62571923
--- Diff: pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/utilities/HiveUtilities.java ---
@@ -102,80 +103,49 @@ public static Table getHiveTable(HiveMetaStoreClient client, Metadata.Item itemN
* <li>{@code decimal(precision, scale) -> numeric(precision, scale)}</li>
* <li>{@code varchar(size) -> varchar(size)}</li>
* <li>{@code char(size) -> bpchar(size)}</li>
+ * <li>{@code array<dataType> -> text}</li>
+ * <li>{@code map<keyDataType, valueDataType> -> text}</li>
+ * <li>{@code struct<field1:dataType,...,fieldN:dataType> -> text}</li>
+ * <li>{@code uniontype<...> -> text}</li>
* </ul>
*
- * @param hiveColumn hive column schema
+ * @param hiveColumn
+ * hive column schema
* @return field with mapped HAWQ type and modifiers
- * @throws UnsupportedTypeException if the column type is not supported
+ * @throws UnsupportedTypeException
+ * if the column type is not supported
+ * @see EnumHiveToHawqType
*/
public static Metadata.Field mapHiveType(FieldSchema hiveColumn) throws UnsupportedTypeException {
String fieldName = hiveColumn.getName();
- String hiveType = hiveColumn.getType();
- String mappedType;
- String[] modifiers = null;
+ String hiveType = hiveColumn.getType(); // Type name and modifiers if any
+ String hiveTypeName; // Type name
+ String[] modifiers = null; // Modifiers
+ EnumHiveToHawqType hiveToHawqType = EnumHiveToHawqType.getHiveToHawqType(hiveType);
+ EnumHawqType hawqType = hiveToHawqType.getHawqType();
- // check parameterized types:
- if (hiveType.startsWith("varchar(") ||
- hiveType.startsWith("char(")) {
- String[] toks = hiveType.split("[(,)]");
- if (toks.length != 2) {
- throw new UnsupportedTypeException( "HAWQ does not support type " + hiveType + " (Field " + fieldName + "), " +
- "expected type of the form <type name>(<parameter>)");
- }
- mappedType = toks[0];
- if (mappedType.equals("char")) {
- mappedType = "bpchar";
- }
- modifiers = new String[] {toks[1]};
- } else if (hiveType.startsWith("decimal(")) {
- String[] toks = hiveType.split("[(,)]");
- if (toks.length != 3) {
- throw new UnsupportedTypeException( "HAWQ does not support type " + hiveType + " (Field " + fieldName + "), " +
- "expected type of the form <type name>(<parameter>,<parameter>)");
+ if (hiveToHawqType.getSplitExpression() != null) {
+ String[] tokens = hiveType.split(hiveToHawqType.getSplitExpression());
+ hiveTypeName = tokens[0];
--- End diff --
EnumHiveToHawqType stores Hive type name, corresponding Hawq type and parse expression(if any), and there is one-to-many relation between "raw" Hive type and EnumHiveToHawqType instance. For example, Hive "raw" types ARRAY<INT>, ARRAY<FLOAT>, ARRAY<STRING> correspond to one enum instance EnumHiveToHawqType.ArrayType. So we cannot tie up tokens to one enum instance on creation stage.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-703. Serialize HCatalog Complex ...
Posted by hornn <gi...@git.apache.org>.
Github user hornn commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/633#discussion_r61818748
--- Diff: pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/utilities/HiveUtilities.java ---
@@ -20,9 +20,12 @@
*/
+import java.util.Arrays;
import java.util.List;
import java.util.ArrayList;
+import org.apache.hawq.pxf.api.utilities.EnumHawqType;
--- End diff --
please put these imports together with the other pxf imports below.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-703. Serialize HCatalog Complex ...
Posted by kavinderd <gi...@git.apache.org>.
Github user kavinderd commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/633#discussion_r62551107
--- Diff: pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/utilities/EnumHiveToHawqType.java ---
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hawq.pxf.plugins.hive.utilities;
+
+import org.apache.hawq.pxf.api.utilities.EnumHawqType;
+import org.apache.hawq.pxf.api.UnsupportedTypeException;
+
+/**
+ *
+ * Hive types, which are supported by plugin, mapped to HAWQ's types
+ * @see EnumHawqType
+ */
+public enum EnumHiveToHawqType {
+
+ TinyintType("tinyint", EnumHawqType.Int2Type),
+ SmallintType("smallint", EnumHawqType.Int2Type),
+ IntType("int", EnumHawqType.Int4Type),
+ BigintType("bigint", EnumHawqType.Int8Type),
+ BooleanType("boolean", EnumHawqType.BoolType),
+ FloatType("float", EnumHawqType.Float4Type),
+ DoubleType("double", EnumHawqType.Float8Type),
+ StringType("string", EnumHawqType.TextType),
+ BinaryType("binary", EnumHawqType.ByteaType),
+ TimestampType("timestamp", EnumHawqType.TimestampType),
+ DateType("date", EnumHawqType.DateType),
+ DecimalType("decimal", EnumHawqType.NumericType, "[(,)]"),
+ VarcharType("varchar", EnumHawqType.VarcharType, "[(,)]"),
+ CharType("char", EnumHawqType.BpcharType, "[(,)]"),
+ ArrayType("array", EnumHawqType.TextType, "[<,>]"),
+ MapType("map", EnumHawqType.TextType, "[<,>]"),
+ StructType("struct", EnumHawqType.TextType, "[<,>]"),
+ UnionType("uniontype", EnumHawqType.TextType, "[<,>]");
+
+ private String typeName;
+ private EnumHawqType hawqType;
+ private String splitExpression;
+
+ EnumHiveToHawqType(String typeName, EnumHawqType hawqType) {
+ this.typeName = typeName;
+ this.hawqType = hawqType;
+ }
+
+ EnumHiveToHawqType(String typeName, EnumHawqType hawqType, String splitExpression) {
+ this(typeName, hawqType);
+ this.splitExpression = splitExpression;
+ }
+
+ /**
+ *
+ * @return name of type
+ */
+ public String getTypeName() {
+ return this.typeName;
+ }
+
+ /**
+ *
+ * @return corresponding HAWQ type
+ */
+ public EnumHawqType getHawqType() {
+ return this.hawqType;
+ }
+
+ /**
+ *
+ * @return split by expression
+ */
+ public String getSplitExpression() {
+ return this.splitExpression;
+ }
+
+ /**
+ * Returns Hive to HAWQ type mapping entry for given Hive type
+ *
+ * @param hiveType full Hive type with modifiers, for example - decimal(10, 0), char(5), binary, array<string>, map<string,float> etc
+ * @return corresponding Hive to HAWQ type mapping entry
+ * @throws UnsupportedTypeException if there is no corresponding HAWQ type
+ */
+ public static EnumHiveToHawqType getHiveToHawqType(String hiveType) {
+ for (EnumHiveToHawqType t : values()) {
+ String hiveTypeName = hiveType;
+ String splitExpression = t.getSplitExpression();
+ if (splitExpression != null) {
+ String[] tokens = hiveType.split(splitExpression);
+ hiveTypeName = tokens[0];
+ }
+
+ if (t.getTypeName().toLowerCase().equals(hiveTypeName.toLowerCase())) {
--- End diff --
Will `t.getTypeName()` ever return NULL?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-703. Serialize HCatalog Complex ...
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/633#discussion_r61978803
--- Diff: pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/utilities/EnumHiveToHawqType.java ---
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hawq.pxf.plugins.hive.utilities;
+
+import org.apache.hawq.pxf.api.utilities.EnumHawqType;
+import org.apache.hawq.pxf.api.UnsupportedTypeException;
+
+/**
+ *
+ * Hive types, which are supported by plugin, mapped to HAWQ's types
+ * @see EnumHawqType
+ */
+public enum EnumHiveToHawqType {
+
+ TinyintType("tinyint", EnumHawqType.Int2Type),
+ SmallintType("smallint", EnumHawqType.Int2Type),
+ IntType("int", EnumHawqType.Int4Type),
+ BigintType("bigint", EnumHawqType.Int8Type),
+ BooleanType("boolean", EnumHawqType.BoolType),
+ FloatType("float", EnumHawqType.Float4Type),
+ DoubleType("double", EnumHawqType.Float8Type),
+ StringType("string", EnumHawqType.TextType),
+ BinaryType("binary", EnumHawqType.ByteaType),
+ TimestampType("timestamp", EnumHawqType.TimestampType),
+ DateType("date", EnumHawqType.DateType),
+ DecimalType("decimal", EnumHawqType.NumericType, "[(,)]"),
+ VarcharType("varchar", EnumHawqType.VarcharType, "[(,)]"),
+ CharType("char", EnumHawqType.BpcharType, "[(,)]"),
+ ArrayType("array", EnumHawqType.TextType, "[<,>]"),
+ MapType("map", EnumHawqType.TextType, "[<,>]"),
+ StructType("struct", EnumHawqType.TextType, "[<,>]"),
+ UnionType("uniontype", EnumHawqType.TextType, "[<,>]");
+
+ private String typeName;
+ private EnumHawqType hawqType;
+ private String splitExpression;
+
+ EnumHiveToHawqType(String typeName, EnumHawqType hawqType) {
+ this.typeName = typeName;
+ this.hawqType = hawqType;
+ }
+
+ EnumHiveToHawqType(String typeName, EnumHawqType hawqType, String splitExpression) {
+ this(typeName, hawqType);
+ this.splitExpression = splitExpression;
+ }
+
+ /**
+ *
+ * @return name of type
+ */
+ public String getTypeName() {
+ return this.typeName;
+ }
+
+ /**
+ *
+ * @return corresponding HAWQ type
+ */
+ public EnumHawqType getHawqType() {
+ return this.hawqType;
+ }
+
+ /**
+ *
+ * @return split by expression
+ */
+ public String getSplitExpression() {
+ return this.splitExpression;
+ }
+
+ /**
+ * Returns Hive to HAWQ type mapping entry for given Hive type
+ *
+ * @param hiveType full Hive type with modifiers, for example - decimal(10, 0), char(5), binary, array<string>, map<string,float> etc
+ * @return corresponding Hive to HAWQ type mapping entry
+ * @throws UnsupportedTypeException if there is no corresponding HAWQ type
+ */
+ public static EnumHiveToHawqType getHiveToHawqType(String hiveType) {
+ for (EnumHiveToHawqType t : values()) {
+ String hiveTypeName = hiveType;
+ if (t.getSplitExpression() != null) {
--- End diff --
Good catch
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-703. Serialize HCatalog Complex ...
Posted by hornn <gi...@git.apache.org>.
Github user hornn commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/633#discussion_r61827025
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/utilities/EnumHawqType.java ---
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hawq.pxf.api.utilities;
+
+import java.io.IOException;
+import org.codehaus.jackson.JsonGenerator;
+import org.codehaus.jackson.map.JsonSerializer;
+import org.codehaus.jackson.map.annotate.JsonSerialize;
+import org.codehaus.jackson.map.SerializerProvider;
+import org.codehaus.jackson.JsonProcessingException;
+
+class EnumHawqTypeSerializer extends JsonSerializer<EnumHawqType> {
+
+ @Override
+ public void serialize(EnumHawqType value, JsonGenerator generator,
+ SerializerProvider provider) throws IOException,
+ JsonProcessingException {
+ generator.writeString(value.getTypeName());
--- End diff --
is it planned to add the modifiers to the serialized data? if so, maybe add a TODO here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-703. Serialize HCatalog Complex ...
Posted by hornn <gi...@git.apache.org>.
Github user hornn commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/633#discussion_r61809892
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/Metadata.java ---
@@ -67,36 +68,43 @@ public String toString() {
}
/**
- * Class representing item field - name and type.
+ * Class representing item field - name, type, source type, modifiers.
+ * Type - exposed type of field
+ * Source type - type of field in underlying source
+ * Modifiers - additional attributes which describe type or field
*/
public static class Field {
private String name;
- private String type; // TODO: change to enum
+ private EnumHawqType type; // field type which PXF exposes
+ private String sourceType; // filed type PXF reads from
--- End diff --
typo field
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-703. Serialize HCatalog Complex ...
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/633#discussion_r61976195
--- Diff: pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/utilities/HiveUtilities.java ---
@@ -186,7 +155,7 @@ public static Table getHiveTable(HiveMetaStoreClient client, Metadata.Item itemN
* @param modifiers type modifiers to be verified
* @return whether modifiers are null or integers
*/
- private static boolean verifyModifers(String[] modifiers) {
+ private static boolean verifyIntegerModifers(String[] modifiers) {
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-703. Serialize HCatalog Complex ...
Posted by hornn <gi...@git.apache.org>.
Github user hornn commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/633#discussion_r61818951
--- Diff: pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/utilities/HiveUtilities.java ---
@@ -186,7 +155,7 @@ public static Table getHiveTable(HiveMetaStoreClient client, Metadata.Item itemN
* @param modifiers type modifiers to be verified
* @return whether modifiers are null or integers
*/
- private static boolean verifyModifers(String[] modifiers) {
+ private static boolean verifyIntegerModifers(String[] modifiers) {
--- End diff --
typo Modifiers
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-703. Serialize HCatalog Complex ...
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/633#discussion_r61978916
--- Diff: pxf/pxf-service/src/test/java/org/apache/hawq/pxf/service/MetadataResponseFormatterTest.java ---
@@ -85,26 +86,45 @@ public void formatResponseStringWithModifiers() throws Exception {
List<Metadata.Field> fields = new ArrayList<Metadata.Field>();
Metadata.Item itemName = new Metadata.Item("default", "table1");
Metadata metadata = new Metadata(itemName, fields);
- fields.add(new Metadata.Field("field1", "int"));
- fields.add(new Metadata.Field("field2", "numeric",
+ fields.add(new Metadata.Field("field1", EnumHawqType.Int8Type, "bigint"));
+ fields.add(new Metadata.Field("field2", EnumHawqType.NumericType, "decimal",
new String[] {"1349", "1789"}));
- fields.add(new Metadata.Field("field3", "char",
+ fields.add(new Metadata.Field("field3", EnumHawqType.BpcharType, "char",
new String[] {"50"}));
metadataList.add(metadata);
response = MetadataResponseFormatter.formatResponse(metadataList, "path.file");
StringBuilder expected = new StringBuilder("{\"PXFMetadata\":[{");
expected.append("\"item\":{\"path\":\"default\",\"name\":\"table1\"},")
.append("\"fields\":[")
- .append("{\"name\":\"field1\",\"type\":\"int\"},")
- .append("{\"name\":\"field2\",\"type\":\"numeric\",\"modifiers\":[\"1349\",\"1789\"]},")
- .append("{\"name\":\"field3\",\"type\":\"char\",\"modifiers\":[\"50\"]}")
+ .append("{\"name\":\"field1\",\"type\":\"int8\",\"sourceType\":\"bigint\"},")
+ .append("{\"name\":\"field2\",\"type\":\"numeric\",\"sourceType\":\"decimal\",\"modifiers\":[\"1349\",\"1789\"]},")
+ .append("{\"name\":\"field3\",\"type\":\"bpchar\",\"sourceType\":\"char\",\"modifiers\":[\"50\"]}")
.append("]}]}");
assertEquals(expected.toString(), convertResponseToString(response));
}
@Test
+ public void formatResponseStringWithSourceType() throws Exception {
+ List<Metadata> metadataList = new ArrayList<Metadata>();
+ List<Metadata.Field> fields = new ArrayList<Metadata.Field>();
+ Metadata.Item itemName = new Metadata.Item("default", "table1");
+ Metadata metadata = new Metadata(itemName, fields);
+ fields.add(new Metadata.Field("field1", EnumHawqType.Float8Type, "double"));
+ metadataList.add(metadata);
+
+ response = MetadataResponseFormatter.formatResponse(metadataList, "path.file");
+ StringBuilder expected = new StringBuilder("{\"PXFMetadata\":[{");
+ expected.append("\"item\":{\"path\":\"default\",\"name\":\"table1\"},")
+ .append("\"fields\":[")
+ .append("{\"name\":\"field1\",\"type\":\"float8\",\"sourceType\":\"double\"}")
+ .append("]}]}");
+
+// assertEquals(expected.toString(), convertResponseToString(response));
--- End diff --
yes, sure
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---