You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hawq.apache.org by linwen <gi...@git.apache.org> on 2016/10/26 09:41:21 UTC
[GitHub] incubator-hawq pull request #976: HAWQ-1104. extract and register tupcount/v...
GitHub user linwen opened a pull request:
https://github.com/apache/incubator-hawq/pull/976
HAWQ-1104. extract and register tupcount/varblocknumber/eofuncompressed
Please review, thanks!
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/linwen/incubator-hawq hawq_1104
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-hawq/pull/976.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #976
----
commit e907c819528c469636374d23d517d17d38e90a53
Author: Wen Lin <wl...@pivotal.io>
Date: 2016-10-26T08:29:53Z
HAWQ-1104. extract and register tupcount/varblocknumber/eofuncompressed
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #976: HAWQ-1104. extract and register tupcount/varblock...
Posted by xunzhang <gi...@git.apache.org>.
Github user xunzhang commented on the issue:
https://github.com/apache/incubator-hawq/pull/976
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #976: HAWQ-1104. extract and register tupcount/v...
Posted by xunzhang <gi...@git.apache.org>.
Github user xunzhang commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/976#discussion_r85255431
--- Diff: tools/bin/hawqregister ---
@@ -539,6 +550,9 @@ class HawqRegister(object):
self.file_format = 'Parquet'
self.files_same_path = []
self.sizes_same_path = []
+ self.tupcounts_same_path = []
+ self.varblockcounts_same_path = []
+ self.eofuncompresseds_same_path = []
--- End diff --
Remove these lines.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #976: HAWQ-1104. extract and register tupcount/v...
Posted by xunzhang <gi...@git.apache.org>.
Github user xunzhang commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/976#discussion_r85254860
--- Diff: tools/bin/hawqregister ---
@@ -394,6 +397,9 @@ class HawqRegister(object):
self.encoding = encoding
self.files_same_path = []
self.sizes_same_path = []
+ self.tupcounts_same_path = []
--- End diff --
Move these lines into `__init__` function to avoid the issue we met before?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #976: HAWQ-1104. extract and register tupcount/v...
Posted by linwen <gi...@git.apache.org>.
Github user linwen closed the pull request at:
https://github.com/apache/incubator-hawq/pull/976
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #976: HAWQ-1104. extract and register tupcount/varblock...
Posted by zhangh43 <gi...@git.apache.org>.
Github user zhangh43 commented on the issue:
https://github.com/apache/incubator-hawq/pull/976
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #976: HAWQ-1104. extract and register tupcount/v...
Posted by xunzhang <gi...@git.apache.org>.
Github user xunzhang commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/976#discussion_r85255184
--- Diff: src/test/feature/lib/sql_util.cpp ---
@@ -268,6 +268,17 @@ std::string SQLUtility::getQueryResult(const std::string &query) {
return value;
}
+std::string SQLUtility::getQueryResultSetString(const std::string &query) {
+ const hawq::test::PSQLQueryResult &result = executeQuery(query);
+ std::vector<std::vector<string> > resultString = result.getRows();
--- End diff --
Useless line.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #976: HAWQ-1104. extract and register tupcount/v...
Posted by xunzhang <gi...@git.apache.org>.
Github user xunzhang commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/976#discussion_r85255575
--- Diff: tools/bin/hawqregister ---
@@ -778,45 +802,67 @@ class HawqRegister(object):
append_eofs = self.sizes_append
update_eofs = self.sizes_update
same_path_eofs = self.sizes_same_path
+ append_tupcounts = self.tupcounts_append
+ update_tupcounts = self.tupcounts_update
+ same_path_tupcounts = self.tupcounts_same_path
+ append_eofuncompresseds = self.eofuncompresseds_append
+ update_eofuncompresseds = self.eofuncompresseds_update
+ same_path_eofuncompresseds = self.eofuncompresseds_same_path
+ append_varblockcounts = self.varblockcounts_append
+ update_varblockcounts = self.varblockcounts_update
+ same_path_varblockcounts = self.varblockcounts_same_path
update_segno_lst = [f.split('/')[-1] for f in self.files_update]
same_path_segno_lst = [seg for seg in self.segnos_same_path]
query = ""
+
if mode == 'force':
query += "delete from pg_aoseg.%s;" % (self.seg_name)
if self.file_format == 'Parquet':
if len(update_eofs) > 0:
- query += 'insert into pg_aoseg.%s values(%s, %d, %d, %d)' % (self.seg_name, update_segno_lst[0], update_eofs[0], -1, -1)
- for k, update_eof in enumerate(update_eofs[1:]):
- query += ',(%s, %d, %d, %d)' % (update_segno_lst[k + 1], update_eof, -1, -1)
+ query += 'insert into pg_aoseg.%s values(%s, %d, %d, %d)' % (self.seg_name, update_segno_lst[0], update_eofs[0], update_tupcounts[0], update_eofuncompresseds[0])
+ k = 0
--- End diff --
I do not like this C-style implementation..
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #976: HAWQ-1104. extract and register tupcount/v...
Posted by xunzhang <gi...@git.apache.org>.
Github user xunzhang commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/976#discussion_r85255318
--- Diff: tools/bin/hawqextract ---
@@ -147,17 +147,17 @@ class GpMetadataAccessor:
def get_aoseg_files(self, oid):
'''
- Return rows in pg_aoseg_`oid` table, excluding
- rows whose content id is -1.
+ Return rows in pg_aoseg_`oid` table
Example:
>>> accessor.get_aoseg_files(35709)
- >>> [{'fileno':'1', 'filesize':'320'},
- ... {'fileno':'2', 'filesize':'880'},
- ... {'fileno':'3', 'filesize':'160'}]
+ >>> [{'fileno':'1', 'filesize':'320', 'tupcount':'10', 'varblockcount':'2', eofuncompressed:'320'},
+ ... {'fileno':'2', 'filesize':'880', 'tupcount':'27', 'varblockcount':'3', eofuncompressed:'880'},
+ ... {'fileno':'3', 'filesize':'160', 'tupcount':'5', 'varblockcount':'2', eofuncompressed:'160'}]
--- End diff --
I like these lines.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #976: HAWQ-1104. extract and register tupcount/v...
Posted by xunzhang <gi...@git.apache.org>.
Github user xunzhang commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/976#discussion_r85255147
--- Diff: src/test/feature/lib/sql_util.cpp ---
@@ -268,6 +268,17 @@ std::string SQLUtility::getQueryResult(const std::string &query) {
return value;
}
+std::string SQLUtility::getQueryResultSetString(const std::string &query) {
+ const hawq::test::PSQLQueryResult &result = executeQuery(query);
+ std::vector<std::vector<string> > resultString = result.getRows();
+ string resultStr;
+ for (auto row : result.getRows()) {
--- End diff --
Use reference to avoid copy.
```
for(auto & row: result.getRows()) {
for(auto & column : row) resultStr += column + "|";
resultStr += '\n';
...
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---