You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hawq.apache.org by chiyang10000 <gi...@git.apache.org> on 2017/12/08 00:44:27 UTC

[GitHub] incubator-hawq pull request #1321: HAWQ-1565. Include Pluggable Storage Form...

GitHub user chiyang10000 opened a pull request:

    https://github.com/apache/incubator-hawq/pull/1321

    HAWQ-1565. Include Pluggable Storage Format Framework in External Table Scan

    Rewrite the tuple construct and consume working flow in the external table, which leads to data copy cost.
    
    @huor 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/chiyang10000/incubator-hawq HAWQ-1565

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-hawq/pull/1321.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1321
    
----
commit 223c8cb4738724b3c42d316a987ab506ca6eda08
Author: Chiyang Wan <ch...@gmail.com>
Date:   2017-12-07T15:05:40Z

    HAWQ-1565. Include Pluggable Storage Format Framework in External Table Scan
    
    Rewrite the tuple construct and consume working flow in the external table, which leads to data copy cost.

----


---

[GitHub] incubator-hawq issue #1321: HAWQ-1565. Include Pluggable Storage Format Fram...

Posted by chiyang10000 <gi...@git.apache.org>.
Github user chiyang10000 commented on the issue:

    https://github.com/apache/incubator-hawq/pull/1321
  
    There will be a contrast performance test in the near days.


---

[GitHub] incubator-hawq pull request #1321: HAWQ-1565. Include Pluggable Storage Form...

Posted by interma <gi...@git.apache.org>.
Github user interma commented on a diff in the pull request:

    https://github.com/apache/incubator-hawq/pull/1321#discussion_r157174345
  
    --- Diff: src/backend/executor/nodeExternalscan.c ---
    @@ -80,8 +83,61 @@ ExternalNext(ExternalScanState *node)
     	/*
     	 * get the next tuple from the file access methods
     	 */
    -	externalSelectDesc = external_getnext_init(&(node->ss.ps), node);
    -	tuple = external_getnext(scandesc, direction, externalSelectDesc);
    +	if (scandesc->fs_formatter_type == ExternalTableType_Invalid)
    +	{
    +		elog(ERROR, "invalid formatter type for external table: %s", __func__);
    +	}
    +	else if (scandesc->fs_formatter_type != ExternalTableType_PLUG)
    +	{
    +		externalSelectDesc = external_getnext_init(&(node->ss.ps), node);
    +
    +		returnTuple = external_getnext(scandesc, direction, externalSelectDesc,
    +		                               &(node->ss), slot);
    +	}
    +	else
    +	{
    +		Assert(scandesc->fs_formatter_name);
    +
    +		FmgrInfo *getnextInitFunc = scandesc->fs_ps_scan_funcs.getnext_init;
    +
    +		if (getnextInitFunc)
    +		{
    +			/*
    +			 * pg_strncasecmp(scandesc->fs_formatter_name, "orc", strlen("orc"))
    +			 * Performance improvement for string comparison.
    +			 */
    +			const char *formatter_name = "orc";
    +			if (*(int *)(scandesc->fs_formatter_name) != *(int *)formatter_name)
    --- End diff --
    
    I think pg_strncasecmp() or other some string compare function is easier to understand.
    Just a advise, do not change is also ok.


---

[GitHub] incubator-hawq pull request #1321: HAWQ-1565. Include Pluggable Storage Form...

Posted by chiyang10000 <gi...@git.apache.org>.
Github user chiyang10000 commented on a diff in the pull request:

    https://github.com/apache/incubator-hawq/pull/1321#discussion_r157335240
  
    --- Diff: src/backend/executor/nodeExternalscan.c ---
    @@ -80,8 +83,61 @@ ExternalNext(ExternalScanState *node)
     	/*
     	 * get the next tuple from the file access methods
     	 */
    -	externalSelectDesc = external_getnext_init(&(node->ss.ps), node);
    -	tuple = external_getnext(scandesc, direction, externalSelectDesc);
    +	if (scandesc->fs_formatter_type == ExternalTableType_Invalid)
    +	{
    +		elog(ERROR, "invalid formatter type for external table: %s", __func__);
    +	}
    +	else if (scandesc->fs_formatter_type != ExternalTableType_PLUG)
    +	{
    +		externalSelectDesc = external_getnext_init(&(node->ss.ps), node);
    +
    +		returnTuple = external_getnext(scandesc, direction, externalSelectDesc,
    +		                               &(node->ss), slot);
    +	}
    +	else
    +	{
    +		Assert(scandesc->fs_formatter_name);
    +
    +		FmgrInfo *getnextInitFunc = scandesc->fs_ps_scan_funcs.getnext_init;
    +
    +		if (getnextInitFunc)
    +		{
    +			/*
    +			 * pg_strncasecmp(scandesc->fs_formatter_name, "orc", strlen("orc"))
    +			 * Performance improvement for string comparison.
    +			 */
    +			const char *formatter_name = "orc";
    +			if (*(int *)(scandesc->fs_formatter_name) != *(int *)formatter_name)
    --- End diff --
    
    Thanks for your advise. I will update this PR with a simple demo in the near days.


---

[GitHub] incubator-hawq issue #1321: HAWQ-1565. Include Pluggable Storage Format Fram...

Posted by huor <gi...@git.apache.org>.
Github user huor commented on the issue:

    https://github.com/apache/incubator-hawq/pull/1321
  
    Looks great to have performance gain in pluggable storage framework. +1
    
    Is there any evaluation regarding the performance boost comparing with existing external table framework?


---

[GitHub] incubator-hawq pull request #1321: HAWQ-1565. Include Pluggable Storage Form...

Posted by chiyang10000 <gi...@git.apache.org>.
Github user chiyang10000 closed the pull request at:

    https://github.com/apache/incubator-hawq/pull/1321


---