You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hawq.apache.org by "Lili Ma (JIRA)" <ji...@apache.org> on 2017/04/07 05:10:42 UTC
[jira] [Created] (HAWQ-1426) hawq extract meets error after the
table was reorganized.
Lili Ma created HAWQ-1426:
-----------------------------
Summary: hawq extract meets error after the table was reorganized.
Key: HAWQ-1426
URL: https://issues.apache.org/jira/browse/HAWQ-1426
Project: Apache HAWQ
Issue Type: Bug
Components: Command Line Tools
Reporter: Lili Ma
Assignee: Ed Espino
Fix For: 2.3.0.0-incubating
After one table is reorganized, hawq extract the table will meet error.
Reproduce Steps:
1. create an AO table
2. insert into several records into it
3. Get the table reorganized. "alter table a set with (reorganize=true);"
4. run hawq extract, error thrown out.
For the bug fix, we should also guarantee that hawq extract should work if the table is truncated and re-inserted.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
Re: [jira] [Created] (HAWQ-1426) hawq extract meets error after the
table was reorganized.
Posted by Lili Ma <lm...@pivotal.io>.
Hi Cyrille,
Thanks for your reply!
Passing relfilenode instead of oid can fix the problem of reorganize, but
not truncate. Since truncate modify relfilenode, but does not create
pg_aosge_$ table name.
Below is the behavior after reorganized:
1. HDFS directory for the table is recreated
2. relfilenode in pg_class for this table changed to new HDFS directory
name.
3. segrelid in pg_appendonly for this table changed to a new value.
4. new pg_aoseg.pg_aoseg_$ table generated.
Below is the behavior after truncate:
1. HDFS directory for the table is recreated
2. relfilenode in pg_class for this table changed to new HDFS directory
name.
3. pg_aoseg.pg_aoseg_$ table name remain the same, but with content cleared.
So I think the more accurate way to identify is finding the table table
through segrelid.
About truncate implementation, may someone know the reason for it?
Thanks
Lili
On Fri, Apr 7, 2017 at 1:56 PM, Cyrille Lintz <cl...@pivotal.io> wrote:
> Hello,
>
> I think the issue comes from here (in /usr/local/hawq/bin/hawqextract):
>
> def get_paqseg_files(self, *oid*):
> '''
> Return rows in pg_paqseg_*`oid`* table, excluding
> rows whose content id is -1.
>
> Example:
> >>> accessor.get_paqseg_files(35709)
> >>> [{'fileno':'1', 'filesize':'320', 'tupcount':'10',
> 'eofuncompressed':'320'},
> ... {'fileno':'2', 'filesize':'880', 'tupcount':'27',
> 'eofuncompressed':'880'},
> ... {'fileno':'3', 'filesize':'160', 'tupcount':'5',
> 'eofuncompressed':'160'}]
> '''
> qry = """
> SELECT segno as fileno, eof as filesize, tupcount, eofuncompressed
> FROM* pg_aoseg.pg_paqseg_%d*
> ORDER by fileno;
> """ % oid
> return self.exec_query(qry)
>
> With reorganize, the pg_paqseg_* table is recreated as
> pg_paqseg_<relfilenode> instead of pg_paqseg_<oid>.
>
> You should pass to get_paqseg_files the relfilenode instead of oid.
> It will work also with truncate.
>
> Regards,
>
>
> *Cyrille LINTZ*Advisory Solution Architect | Pivotal Europe South
> Mobile: + 33 (0)6 11 48 71 10 | clintz@pivotal.io
>
> 2017-04-07 7:10 GMT+02:00 Lili Ma (JIRA) <ji...@apache.org>:
>
> > Lili Ma created HAWQ-1426:
> > -----------------------------
> >
> > Summary: hawq extract meets error after the table was
> > reorganized.
> > Key: HAWQ-1426
> > URL: https://issues.apache.org/jira/browse/HAWQ-1426
> > Project: Apache HAWQ
> > Issue Type: Bug
> > Components: Command Line Tools
> > Reporter: Lili Ma
> > Assignee: Ed Espino
> > Fix For: 2.3.0.0-incubating
> >
> >
> > After one table is reorganized, hawq extract the table will meet error.
> >
> > Reproduce Steps:
> > 1. create an AO table
> > 2. insert into several records into it
> > 3. Get the table reorganized. "alter table a set with
> (reorganize=true);"
> > 4. run hawq extract, error thrown out.
> >
> >
> > For the bug fix, we should also guarantee that hawq extract should work
> if
> > the table is truncated and re-inserted.
> >
> >
> >
> > --
> > This message was sent by Atlassian JIRA
> > (v6.3.15#6346)
> >
>
Re: [jira] [Created] (HAWQ-1426) hawq extract meets error after the
table was reorganized.
Posted by Cyrille Lintz <cl...@pivotal.io>.
Hello,
I think the issue comes from here (in /usr/local/hawq/bin/hawqextract):
def get_paqseg_files(self, *oid*):
'''
Return rows in pg_paqseg_*`oid`* table, excluding
rows whose content id is -1.
Example:
>>> accessor.get_paqseg_files(35709)
>>> [{'fileno':'1', 'filesize':'320', 'tupcount':'10',
'eofuncompressed':'320'},
... {'fileno':'2', 'filesize':'880', 'tupcount':'27',
'eofuncompressed':'880'},
... {'fileno':'3', 'filesize':'160', 'tupcount':'5',
'eofuncompressed':'160'}]
'''
qry = """
SELECT segno as fileno, eof as filesize, tupcount, eofuncompressed
FROM* pg_aoseg.pg_paqseg_%d*
ORDER by fileno;
""" % oid
return self.exec_query(qry)
With reorganize, the pg_paqseg_* table is recreated as
pg_paqseg_<relfilenode> instead of pg_paqseg_<oid>.
You should pass to get_paqseg_files the relfilenode instead of oid.
It will work also with truncate.
Regards,
*Cyrille LINTZ*Advisory Solution Architect | Pivotal Europe South
Mobile: + 33 (0)6 11 48 71 10 | clintz@pivotal.io
2017-04-07 7:10 GMT+02:00 Lili Ma (JIRA) <ji...@apache.org>:
> Lili Ma created HAWQ-1426:
> -----------------------------
>
> Summary: hawq extract meets error after the table was
> reorganized.
> Key: HAWQ-1426
> URL: https://issues.apache.org/jira/browse/HAWQ-1426
> Project: Apache HAWQ
> Issue Type: Bug
> Components: Command Line Tools
> Reporter: Lili Ma
> Assignee: Ed Espino
> Fix For: 2.3.0.0-incubating
>
>
> After one table is reorganized, hawq extract the table will meet error.
>
> Reproduce Steps:
> 1. create an AO table
> 2. insert into several records into it
> 3. Get the table reorganized. "alter table a set with (reorganize=true);"
> 4. run hawq extract, error thrown out.
>
>
> For the bug fix, we should also guarantee that hawq extract should work if
> the table is truncated and re-inserted.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.15#6346)
>