You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hawq.apache.org by kuien <gi...@git.apache.org> on 2018/09/20 10:29:09 UTC
[GitHub] hawq pull request #1397: HAWQ-1660. Optimize parquet scan when bloom filter ...
GitHub user kuien opened a pull request:
https://github.com/apache/hawq/pull/1397
HAWQ-1660. Optimize parquet scan when bloom filter enabled.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kuien/incubator-hawq bf-pq
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/hawq/pull/1397.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1397
----
commit 2320cf08eadb52921427e07ed678abf787d41385
Author: Kuien Liu <ku...@...>
Date: 2018-09-20T09:15:27Z
HAWQ-1660. Optimize parquet scan when bloom filter enabled.
----
---
[GitHub] hawq issue #1397: HAWQ-1660. Optimize parquet scan when bloom filter enabled...
Posted by interma <gi...@git.apache.org>.
Github user interma commented on the issue:
https://github.com/apache/hawq/pull/1397
@kuien I do a perf test on your pr, two issues:
1. query result error
2. performance downgrade
Details see below, please check code, thanks.
TPCH1G data on my mac, master code
```
tpch=# select count (*) from part, lineitem where p_partkey = l_partkey and p_brand = 'Brand#23' and p_container = 'MED BOX';
count
-------
6088
(1 row)
Time: 3150.873 ms
tpch=# set hawq_hashjoin_bloomfilter to on;
SET
Time: 2.903 ms
tpch=# select count (*) from part, lineitem where p_partkey = l_partkey and p_brand = 'Brand#23' and p_container = 'MED BOX';
count
-------
6088
(1 row)
Time: 1512.782 ms
```
your code
```
tpch=# select count (*) from part, lineitem where p_partkey = l_partkey and p_brand = 'Brand#23' and p_container = 'MED BOX';
count
-------
6088
(1 row)
Time: 49466.999 ms #<-- result ok, but bad performance
tpch=# set hawq_hashjoin_bloomfilter to on; SET
Time: 13.106 ms
tpch=# select count (*) from part, lineitem where p_partkey = l_partkey and p_brand = 'Brand#23' and p_container = 'MED BOX';
count
-------
0 #<-- result error
(1 row)
Time: 1888.176 ms
```
---
[GitHub] hawq issue #1397: HAWQ-1660. Optimize parquet scan when bloom filter enabled...
Posted by linwen <gi...@git.apache.org>.
Github user linwen commented on the issue:
https://github.com/apache/hawq/pull/1397
It is a good optimization point. If a lot of columns will be projected, we can only fetch joinkey and do a bloomfilter check, if doesn't match, no need to fetch other columns.
But in this PR, if bloomfilter is not enable, it will fetch joinkey in the first loop, and fetch other columns in the second loop, which needs a little refine further.
---
[GitHub] hawq issue #1397: HAWQ-1660. Optimize parquet scan when bloom filter enabled...
Posted by interma <gi...@git.apache.org>.
Github user interma commented on the issue:
https://github.com/apache/hawq/pull/1397
@kuien
Btw: If also test on mac, you can generate tpch data via my dbgen tools:
https://github.com/interma/misc/tree/master/hawq/dbgen_mac
---
[GitHub] hawq pull request #1397: HAWQ-1660. Optimize parquet scan when bloom filter ...
Posted by kuien <gi...@git.apache.org>.
Github user kuien commented on a diff in the pull request:
https://github.com/apache/hawq/pull/1397#discussion_r219430232
--- Diff: src/backend/cdb/cdbparquetrowgroup.c ---
@@ -322,6 +306,92 @@ ParquetRowGroupReader_ScanNextTuple(
return false;
}
+/*
+ * Get specified attributes of a tuple from current row group into slot.
+ */
+void
+ParquetRowGroupReader_ScanNextTupleColumns(
+ TupleDesc tupDesc,
+ ParquetRowGroupReader *rowGroupReader,
+ int *hawqAttrToParquetColNum,
+ bool *projs,
+ TupleTableSlot *slot,
+ List *attsList)
+{
+ int natts = slot->tts_tupleDescriptor->natts;
+ Assert(natts <= tupDesc->natts);
+
+ Datum *values = slot_get_values(slot);
+ bool *nulls = slot_get_isnull(slot);
+
+ int colReaderIndex = 0;
+ for(int i = 0; i < natts; i++)
+ {
+ /* it is not expensive to do twice in case of bloomfilter */
+ if(projs[i] == false)
+ {
+ nulls[i] = true;
+ continue;
+ }
+
+ /* skip those attributes not in given list */
+ if (attsList != NIL && list_find_int(attsList, i) >= 0)
--- End diff --
typo, should be < 0
---