You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hawq.apache.org by Gagan Brahmi <ga...@gmail.com> on 2017/02/03 03:11:11 UTC
HAWQ Upgarde Issues
Hi All,
I tried to upgrade from HDB 2.0.0.0 to HDB 2.0.1.0. The upgrade went fine
as per the mentioned documentation. However, I am seeing some odd behavior
around PXF Hive integration via HCatalog.
When I try to describe the Hive table (or list the fields) using PXF
HCatalog integration it seems to run into issues. The expected fields out
of the function pxf_get_item_fields still seems to be 4. I know the newer
HDB 2.0.1.0 returns 5 fields from this function.
However, the data access seems to be alright and no issues encountered
there.
The following is how my testing looks like.
---------
postgres=# \d hcatalog.default.hive_table
ERROR: function return row and query-specified return row do not match
DETAIL: Returned row contains 5 attributes, but query expects 4.
postgres=# \d hcatalog.default.*
ERROR: function return row and query-specified return row do not match
DETAIL: Returned row contains 5 attributes, but query expects 4.
postgres=# \d hcatalog.*.*
ERROR: function return row and query-specified return row do not match
DETAIL: Returned row contains 5 attributes, but query expects 4.
postgres=# SELECT * FROM pxf_get_item_fields('Hive','default.hive_table');
ERROR: function return row and query-specified return row do not match
DETAIL: Returned row contains 5 attributes, but query expects 4.
---------
The following is the expected result from a HDB 2.0.1.0 cluster. This is a
clean install 2.0.1.0 cluster (not an upgrade).
---------
postgres=# SELECT * FROM pxf_get_item_fields('Hive','default.hive_table');
path | itemname | fieldname | fieldtype | sourcefieldtype
---------+------------+-----------+-----------+-----------------
default | hive_table | id | int4 | int
default | hive_table | fname | text | string
default | hive_table | lname | text | string
(3 rows)
---------
Is this a known issue? Am I missing something here?
Regards,
Gagan Brahmi
Re: HAWQ Upgarde Issues
Posted by Shivram Mani <sh...@gmail.com>.
Gagan, good catch !
It does look like there was a schema change which shouldn't have happened
given that this is just a minor version upgrade. The additional attribute
was added in order to capture the raw external field type apart from just
storing the hawq compatible field type. We will check if we can provide a
workaround for this issue.
The schema update you have suggested, is the right workaround to support \d
hcatalog.x.y functionlity when someone upgrades from a version prior to HDB
2.0.1.0 to a newer version. We will coordinate with the engineering and
docs team to update the docs.
On Mon, Feb 6, 2017 at 2:34 PM, Gagan Brahmi <ga...@gmail.com> wrote:
> I updated the pg_catalog.pg_proc table to modify the values for
> proallargtypes, proargmodes and proargnames.
>
> postgres=# UPDATE pg_proc SET proallargtypes = '{25,25,25,25,25,25,25}'
> WHERE proname = 'pxf_get_item_fields';
> UPDATE 1
>
> postgres=# UPDATE pg_proc SET proargmodes = '{i,i,o,o,o,o,o}' WHERE proname
> = 'pxf_get_item_fields';
> UPDATE 1
>
> postgres=# UPDATE pg_proc SET proargnames = '{profile,pattern,path,
> itemname,fieldname,fieldtype,sourcefieldtype}' WHERE proname =
> 'pxf_get_item_fields';
> UPDATE 1
>
>
> Regards,
> Gagan Brahmi
>
> On Mon, Feb 6, 2017 at 12:59 PM, Vineet Goel <vv...@apache.org> wrote:
>
> > Gagan - thank you for reporting this. To be specific - how exactly did
> you
> > update pg_proc catalog table?
> >
> >
> >
> > On Sun, Feb 5, 2017 at 8:59 AM Gagan Brahmi <ga...@gmail.com>
> wrote:
> >
> > > So finally I was able to figure out the problem.
> > >
> > > The system catalog table for pg_proc needed to be updated. It looks
> like
> > > the upgrade right now does not have the ability to update the pg_proc
> to
> > > reflect the changed behavior in the OUT for the procedure/function
> > > pxf_get_item_fields.
> > >
> > > Here's how the function looked earlier.
> > >
> > > postgres=# \df pxf_get_item_fields
> > >
> > > List of functions
> > > Schema | Name | Result data type |
> > > Argument data types
> > > | Type
> > >
> > > ------------+---------------------+------------------+------
> > ------------------------------------------------------------
> > ------------------------------------+--------
> > > pg_catalog | pxf_get_item_fields | SETOF record | profile text,
> > > pattern text, OUT path text, OUT itemname text, OUT fieldname text, OUT
> > > fieldtype text | normal
> > > (1 row)
> > >
> > > And here is how the function looks like after updating the pg_proc
> > catalog
> > > table:
> > >
> > > postgres=# \df pxf_get_item_fields
> > >
> > > List of functions
> > > Schema | Name | Result data type |
> > > Argument data types
> > > | Type
> > >
> > > ------------+---------------------+------------------+------
> > ------------------------------------------------------------
> > --------------------------------------------------------------+--------
> > > pg_catalog | pxf_get_item_fields | SETOF record | profile text,
> > > pattern text, OUT path text, OUT itemname text, OUT fieldname text, OUT
> > > fieldtype text, OUT sourcefieldtype text | normal
> > > (1 row)
> > >
> > > The function is now working as expected.
> > >
> > > postgres=# \d hcatalog.default.hive_table
> > > PXF Hive Table "default.hive_table"
> > > Column | Type
> > > --------+------
> > > id | int4
> > > fname | text
> > > lname | text
> > >
> > > postgres=# SELECT * FROM pxf_get_item_fields('Hive','
> > default.hive_table');
> > > path | itemname | fieldname | fieldtype | sourcefieldtype
> > > ---------+------------+-----------+-----------+-----------------
> > > default | hive_table | id | int4 | int
> > > default | hive_table | fname | text | string
> > > default | hive_table | lname | text | string
> > > (3 rows)
> > >
> > > I have created an issue HAWQ-1314 to either update the documentation to
> > > mention this step or improve the upgrade process which can take care of
> > the
> > > catalog table update.
> > >
> > > https://issues.apache.org/jira/browse/HAWQ-1314
> > >
> > >
> > >
> > > Regards,
> > > Gagan Brahmi
> > >
> > >
> > > On Thu, Feb 2, 2017 at 9:42 PM, Gagan Brahmi <ga...@gmail.com>
> > > wrote:
> > >
> > > > Yes the PXF was upgraded as well. It is at 3.0.1 right now. The steps
> > > > followed are the same mentioned in the document.
> > > >
> > > >
> > > > Regards,
> > > > Gagan Brahmi
> > > >
> > > > On Thu, Feb 2, 2017 at 9:30 PM, Pratheesh Nair <pr...@pivotal.io>
> > > wrote:
> > > >
> > > >> Can you confirm pxf has also upgraded in the cluster ? Please have a
> > > look
> > > >> into the below document for the steps .
> > > >>
> > > >> http://hdb.docs.pivotal.io/201/hdb/install/HDB20to201Upgrade
> > > >> .html#20to201up_pxfup
> > > >>
> > > >> Thanks
> > > >> Pratheesh Nair
> > > >>
> > > >> On Thu, Feb 2, 2017 at 7:11 PM, Gagan Brahmi <gaganbrahmi@gmail.com
> >
> > > >> wrote:
> > > >>
> > > >>> Hi All,
> > > >>>
> > > >>> I tried to upgrade from HDB 2.0.0.0 to HDB 2.0.1.0. The upgrade
> went
> > > >>> fine as per the mentioned documentation. However, I am seeing some
> > odd
> > > >>> behavior around PXF Hive integration via HCatalog.
> > > >>>
> > > >>> When I try to describe the Hive table (or list the fields) using
> PXF
> > > >>> HCatalog integration it seems to run into issues. The expected
> fields
> > > out
> > > >>> of the function pxf_get_item_fields still seems to be 4. I know the
> > > newer
> > > >>> HDB 2.0.1.0 returns 5 fields from this function.
> > > >>>
> > > >>> However, the data access seems to be alright and no issues
> > encountered
> > > >>> there.
> > > >>>
> > > >>> The following is how my testing looks like.
> > > >>>
> > > >>> ---------
> > > >>>
> > > >>> postgres=# \d hcatalog.default.hive_table
> > > >>> ERROR: function return row and query-specified return row do not
> > match
> > > >>> DETAIL: Returned row contains 5 attributes, but query expects 4.
> > > >>>
> > > >>> postgres=# \d hcatalog.default.*
> > > >>> ERROR: function return row and query-specified return row do not
> > match
> > > >>> DETAIL: Returned row contains 5 attributes, but query expects 4.
> > > >>>
> > > >>> postgres=# \d hcatalog.*.*
> > > >>> ERROR: function return row and query-specified return row do not
> > match
> > > >>> DETAIL: Returned row contains 5 attributes, but query expects 4.
> > > >>>
> > > >>> postgres=# SELECT * FROM pxf_get_item_fields('Hive','de
> > > >>> fault.hive_table');
> > > >>> ERROR: function return row and query-specified return row do not
> > match
> > > >>> DETAIL: Returned row contains 5 attributes, but query expects 4.
> > > >>>
> > > >>> ---------
> > > >>>
> > > >>> The following is the expected result from a HDB 2.0.1.0 cluster.
> This
> > > is
> > > >>> a clean install 2.0.1.0 cluster (not an upgrade).
> > > >>>
> > > >>> ---------
> > > >>>
> > > >>> postgres=# SELECT * FROM pxf_get_item_fields('Hive','de
> > > >>> fault.hive_table');
> > > >>> path | itemname | fieldname | fieldtype | sourcefieldtype
> > > >>> ---------+------------+-----------+-----------+-----------------
> > > >>> default | hive_table | id | int4 | int
> > > >>> default | hive_table | fname | text | string
> > > >>> default | hive_table | lname | text | string
> > > >>> (3 rows)
> > > >>>
> > > >>> ---------
> > > >>>
> > > >>> Is this a known issue? Am I missing something here?
> > > >>>
> > > >>>
> > > >>>
> > > >>> Regards,
> > > >>> Gagan Brahmi
> > > >>>
> > > >>
> > > >>
> > > >
> > >
> >
>
--
shivram mani
Re: HAWQ Upgarde Issues
Posted by Gagan Brahmi <ga...@gmail.com>.
I updated the pg_catalog.pg_proc table to modify the values for
proallargtypes, proargmodes and proargnames.
postgres=# UPDATE pg_proc SET proallargtypes = '{25,25,25,25,25,25,25}'
WHERE proname = 'pxf_get_item_fields';
UPDATE 1
postgres=# UPDATE pg_proc SET proargmodes = '{i,i,o,o,o,o,o}' WHERE proname
= 'pxf_get_item_fields';
UPDATE 1
postgres=# UPDATE pg_proc SET proargnames = '{profile,pattern,path,
itemname,fieldname,fieldtype,sourcefieldtype}' WHERE proname =
'pxf_get_item_fields';
UPDATE 1
Regards,
Gagan Brahmi
On Mon, Feb 6, 2017 at 12:59 PM, Vineet Goel <vv...@apache.org> wrote:
> Gagan - thank you for reporting this. To be specific - how exactly did you
> update pg_proc catalog table?
>
>
>
> On Sun, Feb 5, 2017 at 8:59 AM Gagan Brahmi <ga...@gmail.com> wrote:
>
> > So finally I was able to figure out the problem.
> >
> > The system catalog table for pg_proc needed to be updated. It looks like
> > the upgrade right now does not have the ability to update the pg_proc to
> > reflect the changed behavior in the OUT for the procedure/function
> > pxf_get_item_fields.
> >
> > Here's how the function looked earlier.
> >
> > postgres=# \df pxf_get_item_fields
> >
> > List of functions
> > Schema | Name | Result data type |
> > Argument data types
> > | Type
> >
> > ------------+---------------------+------------------+------
> ------------------------------------------------------------
> ------------------------------------+--------
> > pg_catalog | pxf_get_item_fields | SETOF record | profile text,
> > pattern text, OUT path text, OUT itemname text, OUT fieldname text, OUT
> > fieldtype text | normal
> > (1 row)
> >
> > And here is how the function looks like after updating the pg_proc
> catalog
> > table:
> >
> > postgres=# \df pxf_get_item_fields
> >
> > List of functions
> > Schema | Name | Result data type |
> > Argument data types
> > | Type
> >
> > ------------+---------------------+------------------+------
> ------------------------------------------------------------
> --------------------------------------------------------------+--------
> > pg_catalog | pxf_get_item_fields | SETOF record | profile text,
> > pattern text, OUT path text, OUT itemname text, OUT fieldname text, OUT
> > fieldtype text, OUT sourcefieldtype text | normal
> > (1 row)
> >
> > The function is now working as expected.
> >
> > postgres=# \d hcatalog.default.hive_table
> > PXF Hive Table "default.hive_table"
> > Column | Type
> > --------+------
> > id | int4
> > fname | text
> > lname | text
> >
> > postgres=# SELECT * FROM pxf_get_item_fields('Hive','
> default.hive_table');
> > path | itemname | fieldname | fieldtype | sourcefieldtype
> > ---------+------------+-----------+-----------+-----------------
> > default | hive_table | id | int4 | int
> > default | hive_table | fname | text | string
> > default | hive_table | lname | text | string
> > (3 rows)
> >
> > I have created an issue HAWQ-1314 to either update the documentation to
> > mention this step or improve the upgrade process which can take care of
> the
> > catalog table update.
> >
> > https://issues.apache.org/jira/browse/HAWQ-1314
> >
> >
> >
> > Regards,
> > Gagan Brahmi
> >
> >
> > On Thu, Feb 2, 2017 at 9:42 PM, Gagan Brahmi <ga...@gmail.com>
> > wrote:
> >
> > > Yes the PXF was upgraded as well. It is at 3.0.1 right now. The steps
> > > followed are the same mentioned in the document.
> > >
> > >
> > > Regards,
> > > Gagan Brahmi
> > >
> > > On Thu, Feb 2, 2017 at 9:30 PM, Pratheesh Nair <pr...@pivotal.io>
> > wrote:
> > >
> > >> Can you confirm pxf has also upgraded in the cluster ? Please have a
> > look
> > >> into the below document for the steps .
> > >>
> > >> http://hdb.docs.pivotal.io/201/hdb/install/HDB20to201Upgrade
> > >> .html#20to201up_pxfup
> > >>
> > >> Thanks
> > >> Pratheesh Nair
> > >>
> > >> On Thu, Feb 2, 2017 at 7:11 PM, Gagan Brahmi <ga...@gmail.com>
> > >> wrote:
> > >>
> > >>> Hi All,
> > >>>
> > >>> I tried to upgrade from HDB 2.0.0.0 to HDB 2.0.1.0. The upgrade went
> > >>> fine as per the mentioned documentation. However, I am seeing some
> odd
> > >>> behavior around PXF Hive integration via HCatalog.
> > >>>
> > >>> When I try to describe the Hive table (or list the fields) using PXF
> > >>> HCatalog integration it seems to run into issues. The expected fields
> > out
> > >>> of the function pxf_get_item_fields still seems to be 4. I know the
> > newer
> > >>> HDB 2.0.1.0 returns 5 fields from this function.
> > >>>
> > >>> However, the data access seems to be alright and no issues
> encountered
> > >>> there.
> > >>>
> > >>> The following is how my testing looks like.
> > >>>
> > >>> ---------
> > >>>
> > >>> postgres=# \d hcatalog.default.hive_table
> > >>> ERROR: function return row and query-specified return row do not
> match
> > >>> DETAIL: Returned row contains 5 attributes, but query expects 4.
> > >>>
> > >>> postgres=# \d hcatalog.default.*
> > >>> ERROR: function return row and query-specified return row do not
> match
> > >>> DETAIL: Returned row contains 5 attributes, but query expects 4.
> > >>>
> > >>> postgres=# \d hcatalog.*.*
> > >>> ERROR: function return row and query-specified return row do not
> match
> > >>> DETAIL: Returned row contains 5 attributes, but query expects 4.
> > >>>
> > >>> postgres=# SELECT * FROM pxf_get_item_fields('Hive','de
> > >>> fault.hive_table');
> > >>> ERROR: function return row and query-specified return row do not
> match
> > >>> DETAIL: Returned row contains 5 attributes, but query expects 4.
> > >>>
> > >>> ---------
> > >>>
> > >>> The following is the expected result from a HDB 2.0.1.0 cluster. This
> > is
> > >>> a clean install 2.0.1.0 cluster (not an upgrade).
> > >>>
> > >>> ---------
> > >>>
> > >>> postgres=# SELECT * FROM pxf_get_item_fields('Hive','de
> > >>> fault.hive_table');
> > >>> path | itemname | fieldname | fieldtype | sourcefieldtype
> > >>> ---------+------------+-----------+-----------+-----------------
> > >>> default | hive_table | id | int4 | int
> > >>> default | hive_table | fname | text | string
> > >>> default | hive_table | lname | text | string
> > >>> (3 rows)
> > >>>
> > >>> ---------
> > >>>
> > >>> Is this a known issue? Am I missing something here?
> > >>>
> > >>>
> > >>>
> > >>> Regards,
> > >>> Gagan Brahmi
> > >>>
> > >>
> > >>
> > >
> >
>
Re: HAWQ Upgarde Issues
Posted by Vineet Goel <vv...@apache.org>.
Gagan - thank you for reporting this. To be specific - how exactly did you
update pg_proc catalog table?
On Sun, Feb 5, 2017 at 8:59 AM Gagan Brahmi <ga...@gmail.com> wrote:
> So finally I was able to figure out the problem.
>
> The system catalog table for pg_proc needed to be updated. It looks like
> the upgrade right now does not have the ability to update the pg_proc to
> reflect the changed behavior in the OUT for the procedure/function
> pxf_get_item_fields.
>
> Here's how the function looked earlier.
>
> postgres=# \df pxf_get_item_fields
>
> List of functions
> Schema | Name | Result data type |
> Argument data types
> | Type
>
> ------------+---------------------+------------------+------------------------------------------------------------------------------------------------------+--------
> pg_catalog | pxf_get_item_fields | SETOF record | profile text,
> pattern text, OUT path text, OUT itemname text, OUT fieldname text, OUT
> fieldtype text | normal
> (1 row)
>
> And here is how the function looks like after updating the pg_proc catalog
> table:
>
> postgres=# \df pxf_get_item_fields
>
> List of functions
> Schema | Name | Result data type |
> Argument data types
> | Type
>
> ------------+---------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+--------
> pg_catalog | pxf_get_item_fields | SETOF record | profile text,
> pattern text, OUT path text, OUT itemname text, OUT fieldname text, OUT
> fieldtype text, OUT sourcefieldtype text | normal
> (1 row)
>
> The function is now working as expected.
>
> postgres=# \d hcatalog.default.hive_table
> PXF Hive Table "default.hive_table"
> Column | Type
> --------+------
> id | int4
> fname | text
> lname | text
>
> postgres=# SELECT * FROM pxf_get_item_fields('Hive','default.hive_table');
> path | itemname | fieldname | fieldtype | sourcefieldtype
> ---------+------------+-----------+-----------+-----------------
> default | hive_table | id | int4 | int
> default | hive_table | fname | text | string
> default | hive_table | lname | text | string
> (3 rows)
>
> I have created an issue HAWQ-1314 to either update the documentation to
> mention this step or improve the upgrade process which can take care of the
> catalog table update.
>
> https://issues.apache.org/jira/browse/HAWQ-1314
>
>
>
> Regards,
> Gagan Brahmi
>
>
> On Thu, Feb 2, 2017 at 9:42 PM, Gagan Brahmi <ga...@gmail.com>
> wrote:
>
> > Yes the PXF was upgraded as well. It is at 3.0.1 right now. The steps
> > followed are the same mentioned in the document.
> >
> >
> > Regards,
> > Gagan Brahmi
> >
> > On Thu, Feb 2, 2017 at 9:30 PM, Pratheesh Nair <pr...@pivotal.io>
> wrote:
> >
> >> Can you confirm pxf has also upgraded in the cluster ? Please have a
> look
> >> into the below document for the steps .
> >>
> >> http://hdb.docs.pivotal.io/201/hdb/install/HDB20to201Upgrade
> >> .html#20to201up_pxfup
> >>
> >> Thanks
> >> Pratheesh Nair
> >>
> >> On Thu, Feb 2, 2017 at 7:11 PM, Gagan Brahmi <ga...@gmail.com>
> >> wrote:
> >>
> >>> Hi All,
> >>>
> >>> I tried to upgrade from HDB 2.0.0.0 to HDB 2.0.1.0. The upgrade went
> >>> fine as per the mentioned documentation. However, I am seeing some odd
> >>> behavior around PXF Hive integration via HCatalog.
> >>>
> >>> When I try to describe the Hive table (or list the fields) using PXF
> >>> HCatalog integration it seems to run into issues. The expected fields
> out
> >>> of the function pxf_get_item_fields still seems to be 4. I know the
> newer
> >>> HDB 2.0.1.0 returns 5 fields from this function.
> >>>
> >>> However, the data access seems to be alright and no issues encountered
> >>> there.
> >>>
> >>> The following is how my testing looks like.
> >>>
> >>> ---------
> >>>
> >>> postgres=# \d hcatalog.default.hive_table
> >>> ERROR: function return row and query-specified return row do not match
> >>> DETAIL: Returned row contains 5 attributes, but query expects 4.
> >>>
> >>> postgres=# \d hcatalog.default.*
> >>> ERROR: function return row and query-specified return row do not match
> >>> DETAIL: Returned row contains 5 attributes, but query expects 4.
> >>>
> >>> postgres=# \d hcatalog.*.*
> >>> ERROR: function return row and query-specified return row do not match
> >>> DETAIL: Returned row contains 5 attributes, but query expects 4.
> >>>
> >>> postgres=# SELECT * FROM pxf_get_item_fields('Hive','de
> >>> fault.hive_table');
> >>> ERROR: function return row and query-specified return row do not match
> >>> DETAIL: Returned row contains 5 attributes, but query expects 4.
> >>>
> >>> ---------
> >>>
> >>> The following is the expected result from a HDB 2.0.1.0 cluster. This
> is
> >>> a clean install 2.0.1.0 cluster (not an upgrade).
> >>>
> >>> ---------
> >>>
> >>> postgres=# SELECT * FROM pxf_get_item_fields('Hive','de
> >>> fault.hive_table');
> >>> path | itemname | fieldname | fieldtype | sourcefieldtype
> >>> ---------+------------+-----------+-----------+-----------------
> >>> default | hive_table | id | int4 | int
> >>> default | hive_table | fname | text | string
> >>> default | hive_table | lname | text | string
> >>> (3 rows)
> >>>
> >>> ---------
> >>>
> >>> Is this a known issue? Am I missing something here?
> >>>
> >>>
> >>>
> >>> Regards,
> >>> Gagan Brahmi
> >>>
> >>
> >>
> >
>
Re: HAWQ Upgarde Issues
Posted by Gagan Brahmi <ga...@gmail.com>.
So finally I was able to figure out the problem.
The system catalog table for pg_proc needed to be updated. It looks like
the upgrade right now does not have the ability to update the pg_proc to
reflect the changed behavior in the OUT for the procedure/function
pxf_get_item_fields.
Here's how the function looked earlier.
postgres=# \df pxf_get_item_fields
List of functions
Schema | Name | Result data type |
Argument data types
| Type
------------+---------------------+------------------+------------------------------------------------------------------------------------------------------+--------
pg_catalog | pxf_get_item_fields | SETOF record | profile text,
pattern text, OUT path text, OUT itemname text, OUT fieldname text, OUT
fieldtype text | normal
(1 row)
And here is how the function looks like after updating the pg_proc catalog
table:
postgres=# \df pxf_get_item_fields
List of functions
Schema | Name | Result data type |
Argument data types
| Type
------------+---------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+--------
pg_catalog | pxf_get_item_fields | SETOF record | profile text,
pattern text, OUT path text, OUT itemname text, OUT fieldname text, OUT
fieldtype text, OUT sourcefieldtype text | normal
(1 row)
The function is now working as expected.
postgres=# \d hcatalog.default.hive_table
PXF Hive Table "default.hive_table"
Column | Type
--------+------
id | int4
fname | text
lname | text
postgres=# SELECT * FROM pxf_get_item_fields('Hive','default.hive_table');
path | itemname | fieldname | fieldtype | sourcefieldtype
---------+------------+-----------+-----------+-----------------
default | hive_table | id | int4 | int
default | hive_table | fname | text | string
default | hive_table | lname | text | string
(3 rows)
I have created an issue HAWQ-1314 to either update the documentation to
mention this step or improve the upgrade process which can take care of the
catalog table update.
https://issues.apache.org/jira/browse/HAWQ-1314
Regards,
Gagan Brahmi
On Thu, Feb 2, 2017 at 9:42 PM, Gagan Brahmi <ga...@gmail.com> wrote:
> Yes the PXF was upgraded as well. It is at 3.0.1 right now. The steps
> followed are the same mentioned in the document.
>
>
> Regards,
> Gagan Brahmi
>
> On Thu, Feb 2, 2017 at 9:30 PM, Pratheesh Nair <pr...@pivotal.io> wrote:
>
>> Can you confirm pxf has also upgraded in the cluster ? Please have a look
>> into the below document for the steps .
>>
>> http://hdb.docs.pivotal.io/201/hdb/install/HDB20to201Upgrade
>> .html#20to201up_pxfup
>>
>> Thanks
>> Pratheesh Nair
>>
>> On Thu, Feb 2, 2017 at 7:11 PM, Gagan Brahmi <ga...@gmail.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> I tried to upgrade from HDB 2.0.0.0 to HDB 2.0.1.0. The upgrade went
>>> fine as per the mentioned documentation. However, I am seeing some odd
>>> behavior around PXF Hive integration via HCatalog.
>>>
>>> When I try to describe the Hive table (or list the fields) using PXF
>>> HCatalog integration it seems to run into issues. The expected fields out
>>> of the function pxf_get_item_fields still seems to be 4. I know the newer
>>> HDB 2.0.1.0 returns 5 fields from this function.
>>>
>>> However, the data access seems to be alright and no issues encountered
>>> there.
>>>
>>> The following is how my testing looks like.
>>>
>>> ---------
>>>
>>> postgres=# \d hcatalog.default.hive_table
>>> ERROR: function return row and query-specified return row do not match
>>> DETAIL: Returned row contains 5 attributes, but query expects 4.
>>>
>>> postgres=# \d hcatalog.default.*
>>> ERROR: function return row and query-specified return row do not match
>>> DETAIL: Returned row contains 5 attributes, but query expects 4.
>>>
>>> postgres=# \d hcatalog.*.*
>>> ERROR: function return row and query-specified return row do not match
>>> DETAIL: Returned row contains 5 attributes, but query expects 4.
>>>
>>> postgres=# SELECT * FROM pxf_get_item_fields('Hive','de
>>> fault.hive_table');
>>> ERROR: function return row and query-specified return row do not match
>>> DETAIL: Returned row contains 5 attributes, but query expects 4.
>>>
>>> ---------
>>>
>>> The following is the expected result from a HDB 2.0.1.0 cluster. This is
>>> a clean install 2.0.1.0 cluster (not an upgrade).
>>>
>>> ---------
>>>
>>> postgres=# SELECT * FROM pxf_get_item_fields('Hive','de
>>> fault.hive_table');
>>> path | itemname | fieldname | fieldtype | sourcefieldtype
>>> ---------+------------+-----------+-----------+-----------------
>>> default | hive_table | id | int4 | int
>>> default | hive_table | fname | text | string
>>> default | hive_table | lname | text | string
>>> (3 rows)
>>>
>>> ---------
>>>
>>> Is this a known issue? Am I missing something here?
>>>
>>>
>>>
>>> Regards,
>>> Gagan Brahmi
>>>
>>
>>
>
Re: HAWQ Upgarde Issues
Posted by Gagan Brahmi <ga...@gmail.com>.
So finally I was able to figure out the problem.
The system catalog table for pg_proc needed to be updated. It looks like
the upgrade right now does not have the ability to update the pg_proc to
reflect the changed behavior in the OUT for the procedure/function
pxf_get_item_fields.
Here's how the function looked earlier.
postgres=# \df pxf_get_item_fields
List of functions
Schema | Name | Result data type |
Argument data types
| Type
------------+---------------------+------------------+------------------------------------------------------------------------------------------------------+--------
pg_catalog | pxf_get_item_fields | SETOF record | profile text,
pattern text, OUT path text, OUT itemname text, OUT fieldname text, OUT
fieldtype text | normal
(1 row)
And here is how the function looks like after updating the pg_proc catalog
table:
postgres=# \df pxf_get_item_fields
List of functions
Schema | Name | Result data type |
Argument data types
| Type
------------+---------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+--------
pg_catalog | pxf_get_item_fields | SETOF record | profile text,
pattern text, OUT path text, OUT itemname text, OUT fieldname text, OUT
fieldtype text, OUT sourcefieldtype text | normal
(1 row)
The function is now working as expected.
postgres=# \d hcatalog.default.hive_table
PXF Hive Table "default.hive_table"
Column | Type
--------+------
id | int4
fname | text
lname | text
postgres=# SELECT * FROM pxf_get_item_fields('Hive','default.hive_table');
path | itemname | fieldname | fieldtype | sourcefieldtype
---------+------------+-----------+-----------+-----------------
default | hive_table | id | int4 | int
default | hive_table | fname | text | string
default | hive_table | lname | text | string
(3 rows)
I have created an issue HAWQ-1314 to either update the documentation to
mention this step or improve the upgrade process which can take care of the
catalog table update.
https://issues.apache.org/jira/browse/HAWQ-1314
Regards,
Gagan Brahmi
On Thu, Feb 2, 2017 at 9:42 PM, Gagan Brahmi <ga...@gmail.com> wrote:
> Yes the PXF was upgraded as well. It is at 3.0.1 right now. The steps
> followed are the same mentioned in the document.
>
>
> Regards,
> Gagan Brahmi
>
> On Thu, Feb 2, 2017 at 9:30 PM, Pratheesh Nair <pr...@pivotal.io> wrote:
>
>> Can you confirm pxf has also upgraded in the cluster ? Please have a look
>> into the below document for the steps .
>>
>> http://hdb.docs.pivotal.io/201/hdb/install/HDB20to201Upgrade
>> .html#20to201up_pxfup
>>
>> Thanks
>> Pratheesh Nair
>>
>> On Thu, Feb 2, 2017 at 7:11 PM, Gagan Brahmi <ga...@gmail.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> I tried to upgrade from HDB 2.0.0.0 to HDB 2.0.1.0. The upgrade went
>>> fine as per the mentioned documentation. However, I am seeing some odd
>>> behavior around PXF Hive integration via HCatalog.
>>>
>>> When I try to describe the Hive table (or list the fields) using PXF
>>> HCatalog integration it seems to run into issues. The expected fields out
>>> of the function pxf_get_item_fields still seems to be 4. I know the newer
>>> HDB 2.0.1.0 returns 5 fields from this function.
>>>
>>> However, the data access seems to be alright and no issues encountered
>>> there.
>>>
>>> The following is how my testing looks like.
>>>
>>> ---------
>>>
>>> postgres=# \d hcatalog.default.hive_table
>>> ERROR: function return row and query-specified return row do not match
>>> DETAIL: Returned row contains 5 attributes, but query expects 4.
>>>
>>> postgres=# \d hcatalog.default.*
>>> ERROR: function return row and query-specified return row do not match
>>> DETAIL: Returned row contains 5 attributes, but query expects 4.
>>>
>>> postgres=# \d hcatalog.*.*
>>> ERROR: function return row and query-specified return row do not match
>>> DETAIL: Returned row contains 5 attributes, but query expects 4.
>>>
>>> postgres=# SELECT * FROM pxf_get_item_fields('Hive','de
>>> fault.hive_table');
>>> ERROR: function return row and query-specified return row do not match
>>> DETAIL: Returned row contains 5 attributes, but query expects 4.
>>>
>>> ---------
>>>
>>> The following is the expected result from a HDB 2.0.1.0 cluster. This is
>>> a clean install 2.0.1.0 cluster (not an upgrade).
>>>
>>> ---------
>>>
>>> postgres=# SELECT * FROM pxf_get_item_fields('Hive','de
>>> fault.hive_table');
>>> path | itemname | fieldname | fieldtype | sourcefieldtype
>>> ---------+------------+-----------+-----------+-----------------
>>> default | hive_table | id | int4 | int
>>> default | hive_table | fname | text | string
>>> default | hive_table | lname | text | string
>>> (3 rows)
>>>
>>> ---------
>>>
>>> Is this a known issue? Am I missing something here?
>>>
>>>
>>>
>>> Regards,
>>> Gagan Brahmi
>>>
>>
>>
>
Re: HAWQ Upgarde Issues
Posted by Gagan Brahmi <ga...@gmail.com>.
Yes the PXF was upgraded as well. It is at 3.0.1 right now. The steps
followed are the same mentioned in the document.
Regards,
Gagan Brahmi
On Thu, Feb 2, 2017 at 9:30 PM, Pratheesh Nair <pr...@pivotal.io> wrote:
> Can you confirm pxf has also upgraded in the cluster ? Please have a look
> into the below document for the steps .
>
> http://hdb.docs.pivotal.io/201/hdb/install/HDB20to201Upgrade.html#
> 20to201up_pxfup
>
> Thanks
> Pratheesh Nair
>
> On Thu, Feb 2, 2017 at 7:11 PM, Gagan Brahmi <ga...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> I tried to upgrade from HDB 2.0.0.0 to HDB 2.0.1.0. The upgrade went fine
>> as per the mentioned documentation. However, I am seeing some odd behavior
>> around PXF Hive integration via HCatalog.
>>
>> When I try to describe the Hive table (or list the fields) using PXF
>> HCatalog integration it seems to run into issues. The expected fields out
>> of the function pxf_get_item_fields still seems to be 4. I know the newer
>> HDB 2.0.1.0 returns 5 fields from this function.
>>
>> However, the data access seems to be alright and no issues encountered
>> there.
>>
>> The following is how my testing looks like.
>>
>> ---------
>>
>> postgres=# \d hcatalog.default.hive_table
>> ERROR: function return row and query-specified return row do not match
>> DETAIL: Returned row contains 5 attributes, but query expects 4.
>>
>> postgres=# \d hcatalog.default.*
>> ERROR: function return row and query-specified return row do not match
>> DETAIL: Returned row contains 5 attributes, but query expects 4.
>>
>> postgres=# \d hcatalog.*.*
>> ERROR: function return row and query-specified return row do not match
>> DETAIL: Returned row contains 5 attributes, but query expects 4.
>>
>> postgres=# SELECT * FROM pxf_get_item_fields('Hive','de
>> fault.hive_table');
>> ERROR: function return row and query-specified return row do not match
>> DETAIL: Returned row contains 5 attributes, but query expects 4.
>>
>> ---------
>>
>> The following is the expected result from a HDB 2.0.1.0 cluster. This is
>> a clean install 2.0.1.0 cluster (not an upgrade).
>>
>> ---------
>>
>> postgres=# SELECT * FROM pxf_get_item_fields('Hive','de
>> fault.hive_table');
>> path | itemname | fieldname | fieldtype | sourcefieldtype
>> ---------+------------+-----------+-----------+-----------------
>> default | hive_table | id | int4 | int
>> default | hive_table | fname | text | string
>> default | hive_table | lname | text | string
>> (3 rows)
>>
>> ---------
>>
>> Is this a known issue? Am I missing something here?
>>
>>
>>
>> Regards,
>> Gagan Brahmi
>>
>
>
Re: HAWQ Upgarde Issues
Posted by Gagan Brahmi <ga...@gmail.com>.
Yes the PXF was upgraded as well. It is at 3.0.1 right now. The steps
followed are the same mentioned in the document.
Regards,
Gagan Brahmi
On Thu, Feb 2, 2017 at 9:30 PM, Pratheesh Nair <pr...@pivotal.io> wrote:
> Can you confirm pxf has also upgraded in the cluster ? Please have a look
> into the below document for the steps .
>
> http://hdb.docs.pivotal.io/201/hdb/install/HDB20to201Upgrade.html#
> 20to201up_pxfup
>
> Thanks
> Pratheesh Nair
>
> On Thu, Feb 2, 2017 at 7:11 PM, Gagan Brahmi <ga...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> I tried to upgrade from HDB 2.0.0.0 to HDB 2.0.1.0. The upgrade went fine
>> as per the mentioned documentation. However, I am seeing some odd behavior
>> around PXF Hive integration via HCatalog.
>>
>> When I try to describe the Hive table (or list the fields) using PXF
>> HCatalog integration it seems to run into issues. The expected fields out
>> of the function pxf_get_item_fields still seems to be 4. I know the newer
>> HDB 2.0.1.0 returns 5 fields from this function.
>>
>> However, the data access seems to be alright and no issues encountered
>> there.
>>
>> The following is how my testing looks like.
>>
>> ---------
>>
>> postgres=# \d hcatalog.default.hive_table
>> ERROR: function return row and query-specified return row do not match
>> DETAIL: Returned row contains 5 attributes, but query expects 4.
>>
>> postgres=# \d hcatalog.default.*
>> ERROR: function return row and query-specified return row do not match
>> DETAIL: Returned row contains 5 attributes, but query expects 4.
>>
>> postgres=# \d hcatalog.*.*
>> ERROR: function return row and query-specified return row do not match
>> DETAIL: Returned row contains 5 attributes, but query expects 4.
>>
>> postgres=# SELECT * FROM pxf_get_item_fields('Hive','de
>> fault.hive_table');
>> ERROR: function return row and query-specified return row do not match
>> DETAIL: Returned row contains 5 attributes, but query expects 4.
>>
>> ---------
>>
>> The following is the expected result from a HDB 2.0.1.0 cluster. This is
>> a clean install 2.0.1.0 cluster (not an upgrade).
>>
>> ---------
>>
>> postgres=# SELECT * FROM pxf_get_item_fields('Hive','de
>> fault.hive_table');
>> path | itemname | fieldname | fieldtype | sourcefieldtype
>> ---------+------------+-----------+-----------+-----------------
>> default | hive_table | id | int4 | int
>> default | hive_table | fname | text | string
>> default | hive_table | lname | text | string
>> (3 rows)
>>
>> ---------
>>
>> Is this a known issue? Am I missing something here?
>>
>>
>>
>> Regards,
>> Gagan Brahmi
>>
>
>
Re: HAWQ Upgarde Issues
Posted by Pratheesh Nair <pr...@pivotal.io>.
Can you confirm pxf has also upgraded in the cluster ? Please have a look
into the below document for the steps .
http://hdb.docs.pivotal.io/201/hdb/install/HDB20to201Upgrade.html#20to201up_pxfup
Thanks
Pratheesh Nair
On Thu, Feb 2, 2017 at 7:11 PM, Gagan Brahmi <ga...@gmail.com> wrote:
> Hi All,
>
> I tried to upgrade from HDB 2.0.0.0 to HDB 2.0.1.0. The upgrade went fine
> as per the mentioned documentation. However, I am seeing some odd behavior
> around PXF Hive integration via HCatalog.
>
> When I try to describe the Hive table (or list the fields) using PXF
> HCatalog integration it seems to run into issues. The expected fields out
> of the function pxf_get_item_fields still seems to be 4. I know the newer
> HDB 2.0.1.0 returns 5 fields from this function.
>
> However, the data access seems to be alright and no issues encountered
> there.
>
> The following is how my testing looks like.
>
> ---------
>
> postgres=# \d hcatalog.default.hive_table
> ERROR: function return row and query-specified return row do not match
> DETAIL: Returned row contains 5 attributes, but query expects 4.
>
> postgres=# \d hcatalog.default.*
> ERROR: function return row and query-specified return row do not match
> DETAIL: Returned row contains 5 attributes, but query expects 4.
>
> postgres=# \d hcatalog.*.*
> ERROR: function return row and query-specified return row do not match
> DETAIL: Returned row contains 5 attributes, but query expects 4.
>
> postgres=# SELECT * FROM pxf_get_item_fields('Hive','default.hive_table');
> ERROR: function return row and query-specified return row do not match
> DETAIL: Returned row contains 5 attributes, but query expects 4.
>
> ---------
>
> The following is the expected result from a HDB 2.0.1.0 cluster. This is a
> clean install 2.0.1.0 cluster (not an upgrade).
>
> ---------
>
> postgres=# SELECT * FROM pxf_get_item_fields('Hive','default.hive_table');
> path | itemname | fieldname | fieldtype | sourcefieldtype
> ---------+------------+-----------+-----------+-----------------
> default | hive_table | id | int4 | int
> default | hive_table | fname | text | string
> default | hive_table | lname | text | string
> (3 rows)
>
> ---------
>
> Is this a known issue? Am I missing something here?
>
>
>
> Regards,
> Gagan Brahmi
>
Re: HAWQ Upgarde Issues
Posted by Pratheesh Nair <pr...@pivotal.io>.
Can you confirm pxf has also upgraded in the cluster ? Please have a look
into the below document for the steps .
http://hdb.docs.pivotal.io/201/hdb/install/HDB20to201Upgrade.html#20to201up_pxfup
Thanks
Pratheesh Nair
On Thu, Feb 2, 2017 at 7:11 PM, Gagan Brahmi <ga...@gmail.com> wrote:
> Hi All,
>
> I tried to upgrade from HDB 2.0.0.0 to HDB 2.0.1.0. The upgrade went fine
> as per the mentioned documentation. However, I am seeing some odd behavior
> around PXF Hive integration via HCatalog.
>
> When I try to describe the Hive table (or list the fields) using PXF
> HCatalog integration it seems to run into issues. The expected fields out
> of the function pxf_get_item_fields still seems to be 4. I know the newer
> HDB 2.0.1.0 returns 5 fields from this function.
>
> However, the data access seems to be alright and no issues encountered
> there.
>
> The following is how my testing looks like.
>
> ---------
>
> postgres=# \d hcatalog.default.hive_table
> ERROR: function return row and query-specified return row do not match
> DETAIL: Returned row contains 5 attributes, but query expects 4.
>
> postgres=# \d hcatalog.default.*
> ERROR: function return row and query-specified return row do not match
> DETAIL: Returned row contains 5 attributes, but query expects 4.
>
> postgres=# \d hcatalog.*.*
> ERROR: function return row and query-specified return row do not match
> DETAIL: Returned row contains 5 attributes, but query expects 4.
>
> postgres=# SELECT * FROM pxf_get_item_fields('Hive','default.hive_table');
> ERROR: function return row and query-specified return row do not match
> DETAIL: Returned row contains 5 attributes, but query expects 4.
>
> ---------
>
> The following is the expected result from a HDB 2.0.1.0 cluster. This is a
> clean install 2.0.1.0 cluster (not an upgrade).
>
> ---------
>
> postgres=# SELECT * FROM pxf_get_item_fields('Hive','default.hive_table');
> path | itemname | fieldname | fieldtype | sourcefieldtype
> ---------+------------+-----------+-----------+-----------------
> default | hive_table | id | int4 | int
> default | hive_table | fname | text | string
> default | hive_table | lname | text | string
> (3 rows)
>
> ---------
>
> Is this a known issue? Am I missing something here?
>
>
>
> Regards,
> Gagan Brahmi
>