You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hawq.apache.org by chiyang10000 <gi...@git.apache.org> on 2017/12/08 00:44:27 UTC
[GitHub] incubator-hawq pull request #1321: HAWQ-1565. Include Pluggable Storage Form...
GitHub user chiyang10000 opened a pull request:
https://github.com/apache/incubator-hawq/pull/1321
HAWQ-1565. Include Pluggable Storage Format Framework in External Table Scan
Rewrite the tuple construct and consume working flow in the external table, which leads to data copy cost.
@huor
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/chiyang10000/incubator-hawq HAWQ-1565
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-hawq/pull/1321.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1321
----
commit 223c8cb4738724b3c42d316a987ab506ca6eda08
Author: Chiyang Wan <ch...@gmail.com>
Date: 2017-12-07T15:05:40Z
HAWQ-1565. Include Pluggable Storage Format Framework in External Table Scan
Rewrite the tuple construct and consume working flow in the external table, which leads to data copy cost.
----
---
[GitHub] incubator-hawq issue #1321: HAWQ-1565. Include Pluggable Storage Format Fram...
Posted by chiyang10000 <gi...@git.apache.org>.
Github user chiyang10000 commented on the issue:
https://github.com/apache/incubator-hawq/pull/1321
There will be a contrast performance test in the near days.
---
[GitHub] incubator-hawq pull request #1321: HAWQ-1565. Include Pluggable Storage Form...
Posted by interma <gi...@git.apache.org>.
Github user interma commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1321#discussion_r157174345
--- Diff: src/backend/executor/nodeExternalscan.c ---
@@ -80,8 +83,61 @@ ExternalNext(ExternalScanState *node)
/*
* get the next tuple from the file access methods
*/
- externalSelectDesc = external_getnext_init(&(node->ss.ps), node);
- tuple = external_getnext(scandesc, direction, externalSelectDesc);
+ if (scandesc->fs_formatter_type == ExternalTableType_Invalid)
+ {
+ elog(ERROR, "invalid formatter type for external table: %s", __func__);
+ }
+ else if (scandesc->fs_formatter_type != ExternalTableType_PLUG)
+ {
+ externalSelectDesc = external_getnext_init(&(node->ss.ps), node);
+
+ returnTuple = external_getnext(scandesc, direction, externalSelectDesc,
+ &(node->ss), slot);
+ }
+ else
+ {
+ Assert(scandesc->fs_formatter_name);
+
+ FmgrInfo *getnextInitFunc = scandesc->fs_ps_scan_funcs.getnext_init;
+
+ if (getnextInitFunc)
+ {
+ /*
+ * pg_strncasecmp(scandesc->fs_formatter_name, "orc", strlen("orc"))
+ * Performance improvement for string comparison.
+ */
+ const char *formatter_name = "orc";
+ if (*(int *)(scandesc->fs_formatter_name) != *(int *)formatter_name)
--- End diff --
I think pg_strncasecmp() or other some string compare function is easier to understand.
Just a advise, do not change is also ok.
---
[GitHub] incubator-hawq pull request #1321: HAWQ-1565. Include Pluggable Storage Form...
Posted by chiyang10000 <gi...@git.apache.org>.
Github user chiyang10000 commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1321#discussion_r157335240
--- Diff: src/backend/executor/nodeExternalscan.c ---
@@ -80,8 +83,61 @@ ExternalNext(ExternalScanState *node)
/*
* get the next tuple from the file access methods
*/
- externalSelectDesc = external_getnext_init(&(node->ss.ps), node);
- tuple = external_getnext(scandesc, direction, externalSelectDesc);
+ if (scandesc->fs_formatter_type == ExternalTableType_Invalid)
+ {
+ elog(ERROR, "invalid formatter type for external table: %s", __func__);
+ }
+ else if (scandesc->fs_formatter_type != ExternalTableType_PLUG)
+ {
+ externalSelectDesc = external_getnext_init(&(node->ss.ps), node);
+
+ returnTuple = external_getnext(scandesc, direction, externalSelectDesc,
+ &(node->ss), slot);
+ }
+ else
+ {
+ Assert(scandesc->fs_formatter_name);
+
+ FmgrInfo *getnextInitFunc = scandesc->fs_ps_scan_funcs.getnext_init;
+
+ if (getnextInitFunc)
+ {
+ /*
+ * pg_strncasecmp(scandesc->fs_formatter_name, "orc", strlen("orc"))
+ * Performance improvement for string comparison.
+ */
+ const char *formatter_name = "orc";
+ if (*(int *)(scandesc->fs_formatter_name) != *(int *)formatter_name)
--- End diff --
Thanks for your advise. I will update this PR with a simple demo in the near days.
---
[GitHub] incubator-hawq issue #1321: HAWQ-1565. Include Pluggable Storage Format Fram...
Posted by huor <gi...@git.apache.org>.
Github user huor commented on the issue:
https://github.com/apache/incubator-hawq/pull/1321
Looks great to have performance gain in pluggable storage framework. +1
Is there any evaluation regarding the performance boost comparing with existing external table framework?
---
[GitHub] incubator-hawq pull request #1321: HAWQ-1565. Include Pluggable Storage Form...
Posted by chiyang10000 <gi...@git.apache.org>.
Github user chiyang10000 closed the pull request at:
https://github.com/apache/incubator-hawq/pull/1321
---