You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hawq.apache.org by sansanichfb <gi...@git.apache.org> on 2017/01/07 00:14:42 UTC
[GitHub] incubator-hawq pull request #1076: HAWQ-1228. Initial commit.
GitHub user sansanichfb opened a pull request:
https://github.com/apache/incubator-hawq/pull/1076
HAWQ-1228. Initial commit.
Work still in progress, but the main part is done. I want to have earlier feedback on it, it's quite a big diff.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sansanichfb/incubator-hawq HAWQ-1228
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-hawq/pull/1076.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1076
----
commit 0574e75fa972f6ccddd1f55a98972223b3860759
Author: Oleksandr Diachenko <od...@pivotal.io>
Date: 2016-12-28T22:03:50Z
HAWQ-1228. Initial commit.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1076: HAWQ-1228. Initial commit.
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1076#discussion_r97923193
--- Diff: pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveAccessor.java ---
@@ -138,12 +135,11 @@ protected Object getReader(JobConf jobConf, InputSplit split)
*/
private InputFormat<?, ?> createInputFormat(InputData input)
throws Exception {
- String userData = new String(input.getFragmentUserData());
- String[] toks = userData.split(HiveDataFragmenter.HIVE_UD_DELIM);
- initPartitionFields(toks[3]);
- filterInFragmenter = new Boolean(toks[4]);
+ HiveUserData hiveUserData = HiveUtilities.parseHiveUserData(input);
+ initPartitionFields(hiveUserData.getPartitionKeys());
--- End diff --
Yes, even if a table has no partitions, we have "!HNPT!" value.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1076: HAWQ-1228. Use profile based on file form...
Posted by shivzone <gi...@git.apache.org>.
Github user shivzone commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1076#discussion_r98127170
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/utilities/InputData.java ---
@@ -31,6 +31,8 @@
*/
public class InputData {
+ public static final String DELIMITER_KEY = "DELIMITER";
+
--- End diff --
Remove extra line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1076: HAWQ-1228. Use profile based on file form...
Posted by kavinderd <gi...@git.apache.org>.
Github user kavinderd commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1076#discussion_r98115361
--- Diff: src/bin/gpfusion/gpbridgeapi.c ---
@@ -68,7 +68,6 @@ void free_token_resources(PxfInputData *inputData);
Datum gpbridge_import(PG_FUNCTION_ARGS)
{
gpbridge_check_inside_extproto(fcinfo, "gpbridge_import");
-// ExternalSelectDesc desc = EXTPROTOCOL_GET_SELECTDESC(fcinfo);
--- End diff --
remove instead of commenting out
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1076: HAWQ-1228. Initial commit.
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1076#discussion_r97715444
--- Diff: pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveStringPassResolver.java ---
@@ -42,21 +46,36 @@ public HiveStringPassResolver(InputData input) throws Exception {
@Override
void parseUserData(InputData input) throws Exception {
- String userData = new String(input.getFragmentUserData());
- String[] toks = userData.split(HiveDataFragmenter.HIVE_UD_DELIM);
+ HiveUserData hiveUserData = HiveUtilities.parseHiveUserData(input);
parseDelimiterChar(input);
parts = new StringBuilder();
- partitionKeys = toks[HiveInputFormatFragmenter.TOK_KEYS];
+ partitionKeys = hiveUserData.getPartitionKeys();
+ serdeClassName = hiveUserData.getSerdeClassName();
+ collectionDelim = input.getUserProperty("COLLECTION_DELIM") == null ? COLLECTION_DELIM : input.getUserProperty("COLLECTION_DELIM");
+ mapkeyDelim = input.getUserProperty("MAPKEY_DELIM") == null ? MAPKEY_DELIM : input.getUserProperty("MAPKEY_DELIM");
+
+ /* Needed only for BINARY format*/
+ if (((ProtocolData) inputData).outputFormat() == OutputFormat.BINARY) {
+ propsString = hiveUserData.getPropertiesString();
+ }
}
@Override
- void initSerde(InputData input) {
- /* nothing to do here */
+ void initSerde(InputData input) throws Exception {
+ if (((ProtocolData) inputData).outputFormat() == OutputFormat.TEXT) {
--- End diff --
Reverse check.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1076: HAWQ-1228. Initial commit.
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1076#discussion_r97715215
--- Diff: pxf/pxf-service/src/main/resources/pxf-profiles-default.xml ---
@@ -49,6 +49,7 @@ under the License.
<accessor>org.apache.hawq.pxf.plugins.hive.HiveAccessor</accessor>
<resolver>org.apache.hawq.pxf.plugins.hive.HiveResolver</resolver>
<metadata>org.apache.hawq.pxf.plugins.hive.HiveMetadataFetcher</metadata>
+ <outputFormat>BINARY</outputFormat>
--- End diff --
defaultOutputFormat
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1076: HAWQ-1228. Use profile based on file form...
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb closed the pull request at:
https://github.com/apache/incubator-hawq/pull/1076
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1076: HAWQ-1228. Use profile based on file form...
Posted by kavinderd <gi...@git.apache.org>.
Github user kavinderd commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1076#discussion_r98277093
--- Diff: pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/utilities/HiveUtilities.java ---
@@ -432,32 +447,144 @@ public static String serializePartitionKeys(HiveTablePartition partData) throws
@SuppressWarnings("unchecked")
public static byte[] makeUserData(String fragmenterClassName, HiveTablePartition partData, boolean filterInFragmenter) throws Exception {
- String userData = null;
+ HiveUserData hiveUserData = null;
if (fragmenterClassName == null) {
throw new IllegalArgumentException("No fragmenter provided.");
}
Class fragmenterClass = Class.forName(fragmenterClassName);
+ String inputFormatName = partData.storageDesc.getInputFormat();
+ String serdeClassName = partData.storageDesc.getSerdeInfo().getSerializationLib();
+ String propertiesString = serializeProperties(partData.properties);
+ String partitionKeys = serializePartitionKeys(partData);
+ String delimiter = getDelimiterCode(partData.storageDesc).toString();
+
if (HiveInputFormatFragmenter.class.isAssignableFrom(fragmenterClass)) {
- String inputFormatName = partData.storageDesc.getInputFormat();
- String serdeName = partData.storageDesc.getSerdeInfo().getSerializationLib();
- String partitionKeys = serializePartitionKeys(partData);
assertFileType(inputFormatName, partData);
- userData = assertSerde(serdeName, partData) + HiveDataFragmenter.HIVE_UD_DELIM
- + partitionKeys + HiveDataFragmenter.HIVE_UD_DELIM + filterInFragmenter;
- } else if (HiveDataFragmenter.class.isAssignableFrom(fragmenterClass)){
- String inputFormatName = partData.storageDesc.getInputFormat();
- String serdeName = partData.storageDesc.getSerdeInfo().getSerializationLib();
- String propertiesString = serializeProperties(partData.properties);
- String partitionKeys = serializePartitionKeys(partData);
- userData = inputFormatName + HiveDataFragmenter.HIVE_UD_DELIM + serdeName
- + HiveDataFragmenter.HIVE_UD_DELIM + propertiesString + HiveDataFragmenter.HIVE_UD_DELIM
- + partitionKeys + HiveDataFragmenter.HIVE_UD_DELIM + filterInFragmenter;
+ }
+
+ hiveUserData = new HiveUserData(inputFormatName, serdeClassName, propertiesString, partitionKeys, filterInFragmenter, delimiter);
+
+ return hiveUserData.toString().getBytes();
+ }
+
+ public static HiveUserData parseHiveUserData(InputData input, PXF_HIVE_SERDES... supportedSerdes) throws UserDataException{
+ String userData = new String(input.getFragmentUserData());
+ String[] toks = userData.split(HiveUserData.HIVE_UD_DELIM, EXPECTED_NUM_OF_TOKS);
+
+ if (toks.length != (EXPECTED_NUM_OF_TOKS)) {
+ throw new UserDataException("HiveInputFormatFragmenter expected "
+ + EXPECTED_NUM_OF_TOKS + " tokens, but got " + toks.length);
+ }
+
+ HiveUserData hiveUserData = new HiveUserData(toks[0], toks[1], toks[2], toks[3], Boolean.valueOf(toks[4]), toks[5]);
+
+ if (supportedSerdes.length > 0) {
--- End diff --
Indentation
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #1076: HAWQ-1228. Initial commit.
Posted by shivzone <gi...@git.apache.org>.
Github user shivzone commented on the issue:
https://github.com/apache/incubator-hawq/pull/1076
Each property in the profile was meant to be a particular class which allowed any user to customize/extend it. In the same theme, instead of definiging outputFormat in the pxf-profile, lets go with <formatter> and get the classname.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #1076: HAWQ-1228. Initial commit.
Posted by shivzone <gi...@git.apache.org>.
Github user shivzone commented on the issue:
https://github.com/apache/incubator-hawq/pull/1076
Please make sure you resolve the conflicts when you merge with master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1076: HAWQ-1228. Initial commit.
Posted by shivzone <gi...@git.apache.org>.
Github user shivzone commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1076#discussion_r95266566
--- Diff: pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveLineBreakAccessor.java ---
@@ -43,9 +43,9 @@
public HiveLineBreakAccessor(InputData input) throws Exception {
super(input, new TextInputFormat());
((TextInputFormat) inputFormat).configure(jobConf);
- String[] toks = HiveInputFormatFragmenter.parseToks(input, PXF_HIVE_SERDES.LAZY_SIMPLE_SERDE.name());
- initPartitionFields(toks[HiveInputFormatFragmenter.TOK_KEYS]);
- filterInFragmenter = new Boolean(toks[HiveInputFormatFragmenter.TOK_FILTER_DONE]);
+ HiveUserData hiveUserData = HiveUtilities.parseHiveUserData(input, PXF_HIVE_SERDES.LAZY_SIMPLE_SERDE);
+ initPartitionFields(hiveUserData.getPartitionKeys());
--- End diff --
Possible NPE in this function
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1076: HAWQ-1228. Initial commit.
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1076#discussion_r95267705
--- Diff: pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveColumnarSerdeResolver.java ---
@@ -233,4 +238,9 @@ private void resolvePrimitive(Object o, PrimitiveObjectInspector oi) throws IOEx
}
firstColumn = false;
}
+
+ @Override
+ void parseDelimiterChar(InputData input) {
+ delimiter = 44; //,
--- End diff --
Yes, it's temporary, work is in progress to actually use appropriate parameters from Hive's SerDe.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1076: HAWQ-1228. Use profile based on file form...
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1076#discussion_r98125062
--- Diff: src/backend/catalog/external/externalmd.c ---
@@ -631,3 +630,79 @@ static int ComputeTypeMod(Oid typeOid, const char *colname, int *typemod, int nT
return VARHDRSZ + result;
}
+static Datum GetFormatTypeForProfile(const List *outputFormats)
+{
+
+ /* if table is homogeneous and output format is text - use text*/
+ if (list_length(outputFormats) == 1 && strcmp(lfirst(list_head(outputFormats)), TextFormatName) == 0)
+ {
+ return CharGetDatum(TextFormatType);
+ } else
+ {
+ return CharGetDatum(CustomFormatType);
+ }
+}
+
+static Datum GetFormatOptionsForProfile(const List *outputFormats, int delimiter)
+{
+ StringInfoData formatStr;
+ initStringInfo(&formatStr);
+
+ /* "delimiter 'delimiter' null '\\N' escape '\\'"*/
+ char formatArr[35] = { 0x64, 0x65, 0x6c, 0x69, 0x6d, 0x69, 0x74, 0x65,
+ 0x72, 0x20, 0x27, delimiter, 0x27, 0x20, 0x6e, 0x75, 0x6c, 0x6c,
+ 0x20, 0x27, 0x5c, 0x4e, 0x27, 0x20, 0x65, 0x73, 0x63, 0x61, 0x70,
+ 0x65, 0x20, 0x27, 0x5c, 0x27, 0x00 };
+
+ if (list_length(outputFormats) == 1 && strcmp(lfirst(list_head(outputFormats)),TextFormatName) == 0)
+ {
+ appendStringInfo(&formatStr, "%s", formatArr);
+ } else {
+ appendStringInfo(&formatStr, "formatter 'pxfwritable_import'");
+ }
+ Datum format_opts = DirectFunctionCall1(textin, CStringGetDatum(formatStr.data));
+ pfree(formatStr.data);
+ return format_opts;
+}
+
+/* location - should be an array of text with one element:
+ * pxf://<ip:port/namaservice>/<db>.<path>?Profile=profileName&delimiter=delimiterCode */
+static Datum GetLocationForFormat(char *profile, List *outputFormats, char *pxf_service_address, char *path, char *name, int delimiter)
--- End diff --
Sure
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1076: HAWQ-1228. Use profile based on file form...
Posted by shivzone <gi...@git.apache.org>.
Github user shivzone commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1076#discussion_r98127047
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/Metadata.java ---
@@ -91,12 +94,17 @@ public Field(String name, EnumHawqType type, String sourceType) {
this.sourceType = sourceType;
}
- public Field(String name, EnumHawqType type, String sourceType,
- String[] modifiers) {
+ public Field(String name, EnumHawqType type, String sourceType, String[] modifiers) {
this(name, type, sourceType);
this.modifiers = modifiers;
}
+ public Field(String name, EnumHawqType type, boolean isComplexType, String sourceType, String[] modifiers) {
--- End diff --
Add isComplexType as the final argument to keep the method overloading structure clean
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1076: HAWQ-1228. Initial commit.
Posted by shivzone <gi...@git.apache.org>.
Github user shivzone commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1076#discussion_r95266450
--- Diff: pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveColumnarSerdeResolver.java ---
@@ -57,11 +59,11 @@
*/
public class HiveColumnarSerdeResolver extends HiveResolver {
private static final Log LOG = LogFactory.getLog(HiveColumnarSerdeResolver.class);
- private ColumnarSerDeBase deserializer;
+ //private ColumnarSerDeBase deserializer;
--- End diff --
Remove line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1076: HAWQ-1228. Initial commit.
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1076#discussion_r97923218
--- Diff: pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveLineBreakAccessor.java ---
@@ -43,9 +43,9 @@
public HiveLineBreakAccessor(InputData input) throws Exception {
super(input, new TextInputFormat());
((TextInputFormat) inputFormat).configure(jobConf);
- String[] toks = HiveInputFormatFragmenter.parseToks(input, PXF_HIVE_SERDES.LAZY_SIMPLE_SERDE.name());
- initPartitionFields(toks[HiveInputFormatFragmenter.TOK_KEYS]);
- filterInFragmenter = new Boolean(toks[HiveInputFormatFragmenter.TOK_FILTER_DONE]);
+ HiveUserData hiveUserData = HiveUtilities.parseHiveUserData(input, PXF_HIVE_SERDES.LAZY_SIMPLE_SERDE);
+ initPartitionFields(hiveUserData.getPartitionKeys());
--- End diff --
partitionKeys always have not-null value.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1076: HAWQ-1228. Initial commit.
Posted by shivzone <gi...@git.apache.org>.
Github user shivzone commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1076#discussion_r95265666
--- Diff: pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveColumnarSerdeResolver.java ---
@@ -233,4 +238,9 @@ private void resolvePrimitive(Object o, PrimitiveObjectInspector oi) throws IOEx
}
firstColumn = false;
}
+
+ @Override
+ void parseDelimiterChar(InputData input) {
+ delimiter = 44; //,
--- End diff --
Is this a temporary workaround ?
Why not use a static variable to store the delimiter value
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1076: HAWQ-1228. Use profile based on file form...
Posted by kavinderd <gi...@git.apache.org>.
Github user kavinderd commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1076#discussion_r98115518
--- Diff: src/backend/catalog/external/externalmd.c ---
@@ -631,3 +630,79 @@ static int ComputeTypeMod(Oid typeOid, const char *colname, int *typemod, int nT
return VARHDRSZ + result;
}
+static Datum GetFormatTypeForProfile(const List *outputFormats)
+{
+
+ /* if table is homogeneous and output format is text - use text*/
+ if (list_length(outputFormats) == 1 && strcmp(lfirst(list_head(outputFormats)), TextFormatName) == 0)
+ {
+ return CharGetDatum(TextFormatType);
+ } else
+ {
+ return CharGetDatum(CustomFormatType);
+ }
+}
+
+static Datum GetFormatOptionsForProfile(const List *outputFormats, int delimiter)
+{
+ StringInfoData formatStr;
+ initStringInfo(&formatStr);
+
+ /* "delimiter 'delimiter' null '\\N' escape '\\'"*/
+ char formatArr[35] = { 0x64, 0x65, 0x6c, 0x69, 0x6d, 0x69, 0x74, 0x65,
+ 0x72, 0x20, 0x27, delimiter, 0x27, 0x20, 0x6e, 0x75, 0x6c, 0x6c,
+ 0x20, 0x27, 0x5c, 0x4e, 0x27, 0x20, 0x65, 0x73, 0x63, 0x61, 0x70,
+ 0x65, 0x20, 0x27, 0x5c, 0x27, 0x00 };
+
+ if (list_length(outputFormats) == 1 && strcmp(lfirst(list_head(outputFormats)),TextFormatName) == 0)
+ {
+ appendStringInfo(&formatStr, "%s", formatArr);
+ } else {
+ appendStringInfo(&formatStr, "formatter 'pxfwritable_import'");
+ }
+ Datum format_opts = DirectFunctionCall1(textin, CStringGetDatum(formatStr.data));
+ pfree(formatStr.data);
+ return format_opts;
+}
+
+/* location - should be an array of text with one element:
+ * pxf://<ip:port/namaservice>/<db>.<path>?Profile=profileName&delimiter=delimiterCode */
+static Datum GetLocationForFormat(char *profile, List *outputFormats, char *pxf_service_address, char *path, char *name, int delimiter)
--- End diff --
Maybe rename path to `db` and name to `path`, the comment and the actual variable names are confusing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1076: HAWQ-1228. Initial commit.
Posted by shivzone <gi...@git.apache.org>.
Github user shivzone commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1076#discussion_r95266362
--- Diff: pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveAccessor.java ---
@@ -138,12 +135,11 @@ protected Object getReader(JobConf jobConf, InputSplit split)
*/
private InputFormat<?, ?> createInputFormat(InputData input)
throws Exception {
- String userData = new String(input.getFragmentUserData());
- String[] toks = userData.split(HiveDataFragmenter.HIVE_UD_DELIM);
- initPartitionFields(toks[3]);
- filterInFragmenter = new Boolean(toks[4]);
+ HiveUserData hiveUserData = HiveUtilities.parseHiveUserData(input);
+ initPartitionFields(hiveUserData.getPartitionKeys());
--- End diff --
Will we always have value for partitionKeys ? If not, initParitionFields will throw NPE
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---