You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hawq.apache.org by kuien <gi...@git.apache.org> on 2018/09/20 10:29:09 UTC

[GitHub] hawq pull request #1397: HAWQ-1660. Optimize parquet scan when bloom filter ...

GitHub user kuien opened a pull request:

    https://github.com/apache/hawq/pull/1397

    HAWQ-1660. Optimize parquet scan when bloom filter enabled.

    

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/kuien/incubator-hawq bf-pq

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/hawq/pull/1397.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1397
    
----
commit 2320cf08eadb52921427e07ed678abf787d41385
Author: Kuien Liu <ku...@...>
Date:   2018-09-20T09:15:27Z

    HAWQ-1660. Optimize parquet scan when bloom filter enabled.

----


---

[GitHub] hawq issue #1397: HAWQ-1660. Optimize parquet scan when bloom filter enabled...

Posted by interma <gi...@git.apache.org>.
Github user interma commented on the issue:

    https://github.com/apache/hawq/pull/1397
  
    @kuien I do a perf test on your pr, two issues:
    1. query result error
    2. performance downgrade
    
    Details see below, please check code, thanks.
    
    TPCH1G data on my mac, master code
    ```
    tpch=# select count (*) from part, lineitem where p_partkey = l_partkey and p_brand = 'Brand#23' and p_container = 'MED BOX';
    count
    -------
      6088
    (1 row)
    
    Time: 3150.873 ms
    tpch=# set hawq_hashjoin_bloomfilter to on;
    SET
    Time: 2.903 ms
    tpch=# select count (*) from part, lineitem where p_partkey = l_partkey and p_brand = 'Brand#23' and p_container = 'MED BOX';
    count
    -------
      6088
    (1 row)
    
    Time: 1512.782 ms
    ```
     your code
    ```
    tpch=# select count (*) from part, lineitem where p_partkey = l_partkey and p_brand = 'Brand#23' and p_container = 'MED BOX';
     count
    -------
      6088
    (1 row)
    
    Time: 49466.999 ms #<-- result ok, but bad performance
    tpch=# set hawq_hashjoin_bloomfilter to on;                                                                             SET
    Time: 13.106 ms
    tpch=# select count (*) from part, lineitem where p_partkey = l_partkey and p_brand = 'Brand#23' and p_container = 'MED BOX';
     count
    -------
         0 #<-- result error
    (1 row)
    
    Time: 1888.176 ms 
    ```


---

[GitHub] hawq issue #1397: HAWQ-1660. Optimize parquet scan when bloom filter enabled...

Posted by linwen <gi...@git.apache.org>.
Github user linwen commented on the issue:

    https://github.com/apache/hawq/pull/1397
  
    It is a good optimization point. If a lot of columns will be projected, we can only fetch joinkey and do a bloomfilter check, if doesn't match, no need to fetch other columns. 
    
    But in this PR, if bloomfilter is not enable, it will fetch joinkey in the first loop, and fetch other columns in the second loop, which needs a little refine further. 


---

[GitHub] hawq issue #1397: HAWQ-1660. Optimize parquet scan when bloom filter enabled...

Posted by interma <gi...@git.apache.org>.
Github user interma commented on the issue:

    https://github.com/apache/hawq/pull/1397
  
    @kuien 
    Btw: If also test on mac, you can generate tpch data via my dbgen tools:
    https://github.com/interma/misc/tree/master/hawq/dbgen_mac


---

[GitHub] hawq pull request #1397: HAWQ-1660. Optimize parquet scan when bloom filter ...

Posted by kuien <gi...@git.apache.org>.
Github user kuien commented on a diff in the pull request:

    https://github.com/apache/hawq/pull/1397#discussion_r219430232
  
    --- Diff: src/backend/cdb/cdbparquetrowgroup.c ---
    @@ -322,6 +306,92 @@ ParquetRowGroupReader_ScanNextTuple(
     	return false;
     }
     
    +/*
    + * Get specified attributes of a tuple from current row group into slot.
    + */
    +void
    +ParquetRowGroupReader_ScanNextTupleColumns(
    +	TupleDesc 				tupDesc,
    +	ParquetRowGroupReader	*rowGroupReader,
    +	int						*hawqAttrToParquetColNum,
    +	bool 					*projs,
    +	TupleTableSlot 			*slot,
    +	List					*attsList)
    +{
    +	int natts = slot->tts_tupleDescriptor->natts;
    +	Assert(natts <= tupDesc->natts);
    +
    +	Datum *values = slot_get_values(slot);
    +	bool *nulls = slot_get_isnull(slot);
    +
    +	int colReaderIndex = 0;
    +	for(int i = 0; i < natts; i++)
    +	{
    +		/* it is not expensive to do twice in case of bloomfilter */
    +		if(projs[i] == false)
    +		{
    +			nulls[i] = true;
    +			continue;
    +		}
    +
    +		/* skip those attributes not in given list */
    +		if (attsList != NIL && list_find_int(attsList, i) >= 0)
    --- End diff --
    
    typo, should be < 0


---