You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by shashwat shriparv <dw...@gmail.com> on 2013/08/03 13:58:35 UTC
Error Accessing Mysql metastore
Hi Friends,
We have configured mysql as a metastore for hive, its just throwing
following error,
FAILED: Error in metadata: MetaException(message:Got exception:
javax.jdo.JDODataStoreException Exception thrown obtaining schema column
information from datastore
on what ever query i am trying, even show tables. i was working find lately
but suddenly it has started throwing this error, can any put some light on
this what may the problem.
i am able to connect to mysql meta store and can see all the tables for
hive database. using the same user name and password i am providing to hive
for jdbc connection.
i am able to to connect to the sql machine and db from the machine where i
am running hive.
*Thanks & Regards *
∞
Shashwat Shriparv
Re: Error Accessing Mysql metastore
Posted by shashwat shriparv <dw...@gmail.com>.
Thanx Nitin, we will let know once it works.. thanx alot for your
responses. :)
*Thanks & Regards *
∞
Shashwat Shriparv
On Mon, Aug 5, 2013 at 12:53 PM, Nitin Pawar <ni...@gmail.com>wrote:
> Yes.
> Please add these two to hive-site.xml
>
> stop your application
> start hive metastore service
> make sure hive is working on cli
> start thirft service
> start your application
> make sure your application is working fine
>
>
>
>
> On Mon, Aug 5, 2013 at 12:51 PM, varun kumar <va...@gmail.com> wrote:
>
>> Hi Nitin,
>>
>> Are these two tags need to mentioned in hive-site.xml.
>>
>> <property>
>> <name>datanucleus.autoCreateSchema</name>
>> <value>false</value>
>> </property>
>> <property>
>> <name>datanucleus.fixedDatastore</name>
>> <value>true</value>
>> </property>
>>
>> Regards,
>> Varun Kumar.P
>>
>>
>> On Mon, Aug 5, 2013 at 10:25 AM, Nitin Pawar <ni...@gmail.com>wrote:
>>
>>> shashwat,
>>>
>>> I don't if you have already read this or not
>>>
>>>
>>> http://mail-archives.apache.org/mod_mbox/hive-user/201107.mbox/%3C4F6B25AFFFCAFE44B6259A412D5F9B1033183876@ExchMBX104.netflix.com%3E
>>>
>>>
>>> On Sat, Aug 3, 2013 at 5:58 PM, shashwat shriparv <
>>> dwivedishashwat@gmail.com> wrote:
>>>
>>>> Actually this is a production server and i cant frequently change, i
>>>> was working perfectly in morning,
>>>>
>>>>
>>>> On Sat, Aug 3, 2013 at 5:55 PM, Nitin Pawar <ni...@gmail.com>wrote:
>>>>
>>>>> datanucleus.fixedDataStore=true
>>>>
>>>>
>>>>
>>>>
>>>> *Thanks & Regards *
>>>>
>>>> ∞
>>>> Shashwat Shriparv
>>>>
>>>>
>>>
>>>
>>> --
>>> Nitin Pawar
>>>
>>
>>
>>
>> --
>> Regards,
>> Varun Kumar.P
>>
>
>
>
> --
> Nitin Pawar
>
Re: Error Accessing Mysql metastore
Posted by Nitin Pawar <ni...@gmail.com>.
Yes.
Please add these two to hive-site.xml
stop your application
start hive metastore service
make sure hive is working on cli
start thirft service
start your application
make sure your application is working fine
On Mon, Aug 5, 2013 at 12:51 PM, varun kumar <va...@gmail.com> wrote:
> Hi Nitin,
>
> Are these two tags need to mentioned in hive-site.xml.
>
> <property>
> <name>datanucleus.autoCreateSchema</name>
> <value>false</value>
> </property>
> <property>
> <name>datanucleus.fixedDatastore</name>
> <value>true</value>
> </property>
>
> Regards,
> Varun Kumar.P
>
>
> On Mon, Aug 5, 2013 at 10:25 AM, Nitin Pawar <ni...@gmail.com>wrote:
>
>> shashwat,
>>
>> I don't if you have already read this or not
>>
>>
>> http://mail-archives.apache.org/mod_mbox/hive-user/201107.mbox/%3C4F6B25AFFFCAFE44B6259A412D5F9B1033183876@ExchMBX104.netflix.com%3E
>>
>>
>> On Sat, Aug 3, 2013 at 5:58 PM, shashwat shriparv <
>> dwivedishashwat@gmail.com> wrote:
>>
>>> Actually this is a production server and i cant frequently change, i was
>>> working perfectly in morning,
>>>
>>>
>>> On Sat, Aug 3, 2013 at 5:55 PM, Nitin Pawar <ni...@gmail.com>wrote:
>>>
>>>> datanucleus.fixedDataStore=true
>>>
>>>
>>>
>>>
>>> *Thanks & Regards *
>>>
>>> ∞
>>> Shashwat Shriparv
>>>
>>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
>
> --
> Regards,
> Varun Kumar.P
>
--
Nitin Pawar
Re: Error Accessing Mysql metastore
Posted by varun kumar <va...@gmail.com>.
Hi Nitin,
Are these two tags need to mentioned in hive-site.xml.
<property>
<name>datanucleus.autoCreateSchema</name>
<value>false</value>
</property>
<property>
<name>datanucleus.fixedDatastore</name>
<value>true</value>
</property>
Regards,
Varun Kumar.P
On Mon, Aug 5, 2013 at 10:25 AM, Nitin Pawar <ni...@gmail.com>wrote:
> shashwat,
>
> I don't if you have already read this or not
>
>
> http://mail-archives.apache.org/mod_mbox/hive-user/201107.mbox/%3C4F6B25AFFFCAFE44B6259A412D5F9B1033183876@ExchMBX104.netflix.com%3E
>
>
> On Sat, Aug 3, 2013 at 5:58 PM, shashwat shriparv <
> dwivedishashwat@gmail.com> wrote:
>
>> Actually this is a production server and i cant frequently change, i was
>> working perfectly in morning,
>>
>>
>> On Sat, Aug 3, 2013 at 5:55 PM, Nitin Pawar <ni...@gmail.com>wrote:
>>
>>> datanucleus.fixedDataStore=true
>>
>>
>>
>>
>> *Thanks & Regards *
>>
>> ∞
>> Shashwat Shriparv
>>
>>
>
>
> --
> Nitin Pawar
>
--
Regards,
Varun Kumar.P
Re: Error Accessing Mysql metastore
Posted by Nitin Pawar <ni...@gmail.com>.
shashwat,
I don't if you have already read this or not
http://mail-archives.apache.org/mod_mbox/hive-user/201107.mbox/%3C4F6B25AFFFCAFE44B6259A412D5F9B1033183876@ExchMBX104.netflix.com%3E
On Sat, Aug 3, 2013 at 5:58 PM, shashwat shriparv <dwivedishashwat@gmail.com
> wrote:
> Actually this is a production server and i cant frequently change, i was
> working perfectly in morning,
>
>
> On Sat, Aug 3, 2013 at 5:55 PM, Nitin Pawar <ni...@gmail.com>wrote:
>
>> datanucleus.fixedDataStore=true
>
>
>
>
> *Thanks & Regards *
>
> ∞
> Shashwat Shriparv
>
>
--
Nitin Pawar
Re: Error Accessing Mysql metastore
Posted by shashwat shriparv <dw...@gmail.com>.
Actually this is a production server and i cant frequently change, i was
working perfectly in morning,
On Sat, Aug 3, 2013 at 5:55 PM, Nitin Pawar <ni...@gmail.com> wrote:
> datanucleus.fixedDataStore=true
*Thanks & Regards *
∞
Shashwat Shriparv
Re: Error Accessing Mysql metastore
Posted by Nitin Pawar <ni...@gmail.com>.
can you try setting this datanucleus.fixedDataStore=true and see if it
solves the issues?
On Sat, Aug 3, 2013 at 5:28 PM, shashwat shriparv <dwivedishashwat@gmail.com
> wrote:
> Hi Friends,
>
> We have configured mysql as a metastore for hive, its just throwing
> following error,
>
> FAILED: Error in metadata: MetaException(message:Got exception:
> javax.jdo.JDODataStoreException Exception thrown obtaining schema column
> information from datastore
>
>
> on what ever query i am trying, even show tables. i was working find
> lately but suddenly it has started throwing this error, can any put some
> light on this what may the problem.
>
> i am able to connect to mysql meta store and can see all the tables for
> hive database. using the same user name and password i am providing to hive
> for jdbc connection.
>
> i am able to to connect to the sql machine and db from the machine where i
> am running hive.
>
>
> *Thanks & Regards *
>
> ∞
> Shashwat Shriparv
>
>
--
Nitin Pawar
Re: Hive UDAF extending UDAF class: iterate or evaluate method
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Please follow the guidance for UDF provided in the Hive Programming book by Wampler/Capriolo. That will work for u.
I can say with confidence that their book was mighty helpful to me in my project from start to production...
And I would recommend go ahead with a way, implement and then fine tune otherwise u will be in analysis paralysis mode…
We are all on a path of discovery here ...
Regards
sanjay
From: Ritesh Agrawal <ra...@netflix.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Wednesday, August 7, 2013 5:57 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Re: Hive UDAF extending UDAF class: iterate or evaluate method
Hi Sanjay, Lefty
Thanks for the help but none of above responses directly answering my question (probably I am not asking clear enough :-( ).
Below I have two different structure of a UDAF (aggregation function). My question is which one is the preferred/right approach
http://pastebin.com/QCgd4Hxc : This version is based on based on what I could understand from API docs about UDAF class.
http://pastebin.com/Uctamtek : This version is based on the book Hadoop The definitive guide. Notice the function names for different from the first one.
I hope this clarifies my question.
Thanks
Ritesh
On Wed, Aug 7, 2013 at 5:34 PM, Lefty Leverenz <le...@gmail.com>> wrote:
Sounds like the wikidoc needs some work. I'm open to suggestions. If Sanjay's simple UDF helps, I could put it in the wiki along with any advice you think would help.
Does anyone else have use cases to contribute?
-- Lefty
On Mon, Aug 5, 2013 at 2:45 PM, Sanjay Subramanian <Sa...@wizecommerce.com>> wrote:
Hi Ritesh
To help u get started , I am writing a simple HelloWorld-ish UDF that might help…If it doesn't please ask for more clarifications...
Good Luck
Thanks
sanjay
********************************************************************************
ToUpperCase.java
package com.sanjaysubramanian.utils.hive.udf;
import org.apache.hadoop.hive.ql.exec.UDF;
public finalclass ToUpperCase extends UDF{
protected final Log logger = LogFactory.getLog(toUpperCase.class);
publicString evaluate(final String inputString) {
if (inputString != null){
return inputString.toUpper;
}
else {
return inputString;
}
}
}
********************************************************************************
Usage in a Hive script
hive -e "
create temporary function toupper as 'com.sanjaysubramanian.utils.hive.udf.ToUpperCase';
SELECT
first_name,
toupper(first_name)
FROM
company_names
"
***********************************************************************************
From: Ritesh Agrawal <ra...@netflix.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Monday, August 5, 2013 9:41 AM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Re: Hive UDAF extending UDAF class: iterate or evaluate method
Hi Lefty,
I used the wiki you sent to write my first version of UDAF. However, I found it to be utterly complex, especially for storing partial results as I am not very familiar with hive API. Then I found another example of UDAF in the hadoop the definitive guide book and it had much simpler code but using different method. Instead of using iterate it was using evaluate method and so I am getting confused.
Ritesh
On Sun, Aug 4, 2013 at 2:18 PM, Lefty Leverenz <le...@gmail.com>> wrote:
You might find this wikidoc useful: GenericUDAFCaseStudy<https://cwiki.apache.org/confluence/display/Hive/GenericUDAFCaseStudy>.
The O'Reilly book "Programming Hive" also has a section called "User-Defined Aggregate Functions" in chapter 13 (Functions), pages 172 to 176.
-- Lefty
On Sun, Aug 4, 2013 at 7:12 AM, Ritesh Agrawal <ra...@netflix.com>> wrote:
Hi all,
I am trying to write a UDAF function. I found an example that shows how to implement a UDAF in "Hadoop The Definitive Guide" book. However I am little confused. In the book, the author extends UDAF class and implements init, iterate, terminatePartial, merge and terminate function. However looking at the hive docs (http://hive.apache.org/docs/r0.11.0/api/org/apache/hadoop/hive/ql/exec/UDAF.html), it seems I need to implement init, aggregate, evaluatePartial, aggregatePartial and evaluate function. Please let me know what are the write functions to implement.
Ritesh
--
Lefty
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
--
Lefty
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: Hive UDAF extending UDAF class: iterate or evaluate method
Posted by Ritesh Agrawal <ra...@netflix.com>.
Hi Sanjay, Lefty
Thanks for the help but none of above responses directly answering my
question (probably I am not asking clear enough :-( ).
Below I have two different structure of a UDAF (aggregation function). My
question is which one is the preferred/right approach
http://pastebin.com/QCgd4Hxc : This version is based on based on what I
could understand from API docs about UDAF class.
http://pastebin.com/Uctamtek : This version is based on the book Hadoop The
definitive guide. Notice the function names for different from the first
one.
I hope this clarifies my question.
Thanks
Ritesh
On Wed, Aug 7, 2013 at 5:34 PM, Lefty Leverenz <le...@gmail.com>wrote:
> Sounds like the wikidoc needs some work. I'm open to suggestions. If
> Sanjay's simple UDF helps, I could put it in the wiki along with any advice
> you think would help.
>
> Does anyone else have use cases to contribute?
>
> -- Lefty
>
>
> On Mon, Aug 5, 2013 at 2:45 PM, Sanjay Subramanian <
> Sanjay.Subramanian@wizecommerce.com> wrote:
>
>> Hi Ritesh
>>
>> To help u get started , I am writing a simple HelloWorld-ish UDF that
>> might help…If it doesn't please ask for more clarifications...
>>
>> Good Luck
>> Thanks
>>
>> sanjay
>>
>>
>> ********************************************************************************
>> *ToUpperCase.java*
>>
>> *package* com.sanjaysubramanian.utils.hive.udf;
>>
>> *import* org.apache.hadoop.hive.ql.exec.UDF;
>>
>>
>> *public* *final* *class* ToUpperCase *extends* UDF{
>>
>> *protected* *final* Log logger = LogFactory.*getLog*(toUpperCase.*
>> class*);
>>
>>
>> *public* *String* evaluate(*final* String inputString) {
>>
>> if (inputString != null){
>>
>> *return* inputString.toUpper;
>>
>> }
>>
>> else {
>>
>> *return* inputString;
>>
>> }
>>
>> }
>>
>> }
>>
>> ********************************************************************************
>>
>> *Usage in a Hive script*
>> *
>> *
>> hive -e "
>>
>> create temporary function toupper as
>> 'com.sanjaysubramanian.utils.hive.udf.ToUpperCase';
>> SELECT
>> first_name,
>> toupper(first_name)
>> FROM
>> company_names
>> "
>>
>>
>> ***********************************************************************************
>>
>> From: Ritesh Agrawal <ra...@netflix.com>
>> Reply-To: "user@hive.apache.org" <us...@hive.apache.org>
>> Date: Monday, August 5, 2013 9:41 AM
>> To: "user@hive.apache.org" <us...@hive.apache.org>
>> Subject: Re: Hive UDAF extending UDAF class: iterate or evaluate method
>>
>> Hi Lefty,
>>
>> I used the wiki you sent to write my first version of UDAF. However, I
>> found it to be utterly complex, especially for storing partial results as I
>> am not very familiar with hive API. Then I found another example of UDAF in
>> the hadoop the definitive guide book and it had much simpler code but using
>> different method. Instead of using iterate it was using evaluate method and
>> so I am getting confused.
>>
>> Ritesh
>>
>>
>> On Sun, Aug 4, 2013 at 2:18 PM, Lefty Leverenz <le...@gmail.com>wrote:
>>
>>> You might find this wikidoc useful: GenericUDAFCaseStudy<https://cwiki.apache.org/confluence/display/Hive/GenericUDAFCaseStudy>.
>>>
>>>
>>> The O'Reilly book "Programming Hive" also has a section called
>>> "User-Defined Aggregate Functions" in chapter 13 (Functions), pages 172 to
>>> 176.
>>>
>>> -- Lefty
>>>
>>>
>>> On Sun, Aug 4, 2013 at 7:12 AM, Ritesh Agrawal <ra...@netflix.com>wrote:
>>>
>>>> Hi all,
>>>>
>>>> I am trying to write a UDAF function. I found an example that shows
>>>> how to implement a UDAF in "Hadoop The Definitive Guide" book. However I am
>>>> little confused. In the book, the author extends UDAF class and implements
>>>> init, iterate, terminatePartial, merge and terminate function. However
>>>> looking at the hive docs (
>>>> http://hive.apache.org/docs/r0.11.0/api/org/apache/hadoop/hive/ql/exec/UDAF.html),
>>>> it seems I need to implement init, aggregate, evaluatePartial,
>>>> aggregatePartial and evaluate function. Please let me know what are the
>>>> write functions to implement.
>>>>
>>>> Ritesh
>>>>
>>>
>>>
>>>
>>> --
>>> Lefty
>>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> ======================
>> This email message and any attachments are for the exclusive use of the
>> intended recipient(s) and may contain confidential and privileged
>> information. Any unauthorized review, use, disclosure or distribution is
>> prohibited. If you are not the intended recipient, please contact the
>> sender by reply email and destroy all copies of the original message along
>> with any attachments, from your computer system. If you are the intended
>> recipient, please be advised that the content of this message is subject to
>> access, review and disclosure by the sender's Email System Administrator.
>>
>
>
>
> --
> Lefty
>
Re: Hive UDAF extending UDAF class: iterate or evaluate method
Posted by Lefty Leverenz <le...@gmail.com>.
Sounds like the wikidoc needs some work. I'm open to suggestions. If
Sanjay's simple UDF helps, I could put it in the wiki along with any advice
you think would help.
Does anyone else have use cases to contribute?
-- Lefty
On Mon, Aug 5, 2013 at 2:45 PM, Sanjay Subramanian <
Sanjay.Subramanian@wizecommerce.com> wrote:
> Hi Ritesh
>
> To help u get started , I am writing a simple HelloWorld-ish UDF that
> might help…If it doesn't please ask for more clarifications...
>
> Good Luck
> Thanks
>
> sanjay
>
>
> ********************************************************************************
> *ToUpperCase.java*
>
> *package* com.sanjaysubramanian.utils.hive.udf;
>
> *import* org.apache.hadoop.hive.ql.exec.UDF;
>
>
> *public* *final* *class* ToUpperCase *extends* UDF{
>
> *protected* *final* Log logger = LogFactory.*getLog*(toUpperCase.*
> class*);
>
>
> *public* *String* evaluate(*final* String inputString) {
>
> if (inputString != null){
>
> *return* inputString.toUpper;
>
> }
>
> else {
>
> *return* inputString;
>
> }
>
> }
>
> }
>
> ********************************************************************************
>
> *Usage in a Hive script*
> *
> *
> hive -e "
>
> create temporary function toupper as
> 'com.sanjaysubramanian.utils.hive.udf.ToUpperCase';
> SELECT
> first_name,
> toupper(first_name)
> FROM
> company_names
> "
>
>
> ***********************************************************************************
>
> From: Ritesh Agrawal <ra...@netflix.com>
> Reply-To: "user@hive.apache.org" <us...@hive.apache.org>
> Date: Monday, August 5, 2013 9:41 AM
> To: "user@hive.apache.org" <us...@hive.apache.org>
> Subject: Re: Hive UDAF extending UDAF class: iterate or evaluate method
>
> Hi Lefty,
>
> I used the wiki you sent to write my first version of UDAF. However, I
> found it to be utterly complex, especially for storing partial results as I
> am not very familiar with hive API. Then I found another example of UDAF in
> the hadoop the definitive guide book and it had much simpler code but using
> different method. Instead of using iterate it was using evaluate method and
> so I am getting confused.
>
> Ritesh
>
>
> On Sun, Aug 4, 2013 at 2:18 PM, Lefty Leverenz <le...@gmail.com>wrote:
>
>> You might find this wikidoc useful: GenericUDAFCaseStudy<https://cwiki.apache.org/confluence/display/Hive/GenericUDAFCaseStudy>.
>>
>>
>> The O'Reilly book "Programming Hive" also has a section called
>> "User-Defined Aggregate Functions" in chapter 13 (Functions), pages 172 to
>> 176.
>>
>> -- Lefty
>>
>>
>> On Sun, Aug 4, 2013 at 7:12 AM, Ritesh Agrawal <ra...@netflix.com>wrote:
>>
>>> Hi all,
>>>
>>> I am trying to write a UDAF function. I found an example that shows
>>> how to implement a UDAF in "Hadoop The Definitive Guide" book. However I am
>>> little confused. In the book, the author extends UDAF class and implements
>>> init, iterate, terminatePartial, merge and terminate function. However
>>> looking at the hive docs (
>>> http://hive.apache.org/docs/r0.11.0/api/org/apache/hadoop/hive/ql/exec/UDAF.html),
>>> it seems I need to implement init, aggregate, evaluatePartial,
>>> aggregatePartial and evaluate function. Please let me know what are the
>>> write functions to implement.
>>>
>>> Ritesh
>>>
>>
>>
>>
>> --
>> Lefty
>>
>
>
> CONFIDENTIALITY NOTICE
> ======================
> This email message and any attachments are for the exclusive use of the
> intended recipient(s) and may contain confidential and privileged
> information. Any unauthorized review, use, disclosure or distribution is
> prohibited. If you are not the intended recipient, please contact the
> sender by reply email and destroy all copies of the original message along
> with any attachments, from your computer system. If you are the intended
> recipient, please be advised that the content of this message is subject to
> access, review and disclosure by the sender's Email System Administrator.
>
--
Lefty
Re: Hive UDAF extending UDAF class: iterate or evaluate method
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Hi Ritesh
To help u get started , I am writing a simple HelloWorld-ish UDF that might help…If it doesn't please ask for more clarifications...
Good Luck
Thanks
sanjay
********************************************************************************
ToUpperCase.java
package com.sanjaysubramanian.utils.hive.udf;
import org.apache.hadoop.hive.ql.exec.UDF;
public final class ToUpperCase extends UDF{
protected final Log logger = LogFactory.getLog(toUpperCase.class);
public String evaluate(final String inputString) {
if (inputString != null){
return inputString.toUpper;
}
else {
return inputString;
}
}
}
********************************************************************************
Usage in a Hive script
hive -e "
create temporary function toupper as 'com.sanjaysubramanian.utils.hive.udf.ToUpperCase';
SELECT
first_name,
toupper(first_name)
FROM
company_names
"
***********************************************************************************
From: Ritesh Agrawal <ra...@netflix.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Monday, August 5, 2013 9:41 AM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Re: Hive UDAF extending UDAF class: iterate or evaluate method
Hi Lefty,
I used the wiki you sent to write my first version of UDAF. However, I found it to be utterly complex, especially for storing partial results as I am not very familiar with hive API. Then I found another example of UDAF in the hadoop the definitive guide book and it had much simpler code but using different method. Instead of using iterate it was using evaluate method and so I am getting confused.
Ritesh
On Sun, Aug 4, 2013 at 2:18 PM, Lefty Leverenz <le...@gmail.com>> wrote:
You might find this wikidoc useful: GenericUDAFCaseStudy<https://cwiki.apache.org/confluence/display/Hive/GenericUDAFCaseStudy>.
The O'Reilly book "Programming Hive" also has a section called "User-Defined Aggregate Functions" in chapter 13 (Functions), pages 172 to 176.
-- Lefty
On Sun, Aug 4, 2013 at 7:12 AM, Ritesh Agrawal <ra...@netflix.com>> wrote:
Hi all,
I am trying to write a UDAF function. I found an example that shows how to implement a UDAF in "Hadoop The Definitive Guide" book. However I am little confused. In the book, the author extends UDAF class and implements init, iterate, terminatePartial, merge and terminate function. However looking at the hive docs (http://hive.apache.org/docs/r0.11.0/api/org/apache/hadoop/hive/ql/exec/UDAF.html), it seems I need to implement init, aggregate, evaluatePartial, aggregatePartial and evaluate function. Please let me know what are the write functions to implement.
Ritesh
--
Lefty
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: Hive UDAF extending UDAF class: iterate or evaluate method
Posted by Ritesh Agrawal <ra...@netflix.com>.
Hi Lefty,
I used the wiki you sent to write my first version of UDAF. However, I
found it to be utterly complex, especially for storing partial results as I
am not very familiar with hive API. Then I found another example of UDAF in
the hadoop the definitive guide book and it had much simpler code but using
different method. Instead of using iterate it was using evaluate method and
so I am getting confused.
Ritesh
On Sun, Aug 4, 2013 at 2:18 PM, Lefty Leverenz <le...@gmail.com>wrote:
> You might find this wikidoc useful: GenericUDAFCaseStudy<https://cwiki.apache.org/confluence/display/Hive/GenericUDAFCaseStudy>
> .
>
> The O'Reilly book "Programming Hive" also has a section called
> "User-Defined Aggregate Functions" in chapter 13 (Functions), pages 172 to
> 176.
>
> -- Lefty
>
>
> On Sun, Aug 4, 2013 at 7:12 AM, Ritesh Agrawal <ra...@netflix.com>wrote:
>
>> Hi all,
>>
>> I am trying to write a UDAF function. I found an example that shows how
>> to implement a UDAF in "Hadoop The Definitive Guide" book. However I am
>> little confused. In the book, the author extends UDAF class and implements
>> init, iterate, terminatePartial, merge and terminate function. However
>> looking at the hive docs (
>> http://hive.apache.org/docs/r0.11.0/api/org/apache/hadoop/hive/ql/exec/UDAF.html),
>> it seems I need to implement init, aggregate, evaluatePartial,
>> aggregatePartial and evaluate function. Please let me know what are the
>> write functions to implement.
>>
>> Ritesh
>>
>
>
>
> --
> Lefty
>
Re: Hive UDAF extending UDAF class: iterate or evaluate method
Posted by Lefty Leverenz <le...@gmail.com>.
You might find this wikidoc useful:
GenericUDAFCaseStudy<https://cwiki.apache.org/confluence/display/Hive/GenericUDAFCaseStudy>
.
The O'Reilly book "Programming Hive" also has a section called
"User-Defined Aggregate Functions" in chapter 13 (Functions), pages 172 to
176.
-- Lefty
On Sun, Aug 4, 2013 at 7:12 AM, Ritesh Agrawal <ra...@netflix.com> wrote:
> Hi all,
>
> I am trying to write a UDAF function. I found an example that shows how to
> implement a UDAF in "Hadoop The Definitive Guide" book. However I am little
> confused. In the book, the author extends UDAF class and implements init,
> iterate, terminatePartial, merge and terminate function. However looking
> at the hive docs (
> http://hive.apache.org/docs/r0.11.0/api/org/apache/hadoop/hive/ql/exec/UDAF.html),
> it seems I need to implement init, aggregate, evaluatePartial,
> aggregatePartial and evaluate function. Please let me know what are the
> write functions to implement.
>
> Ritesh
>
--
Lefty
Hive UDAF extending UDAF class: iterate or evaluate method
Posted by Ritesh Agrawal <ra...@netflix.com>.
Hi all,
I am trying to write a UDAF function. I found an example that shows how to implement a UDAF in "Hadoop The Definitive Guide" book. However I am little confused. In the book, the author extends UDAF class and implements init, iterate, terminatePartial, merge and terminate function. However looking at the hive docs (http://hive.apache.org/docs/r0.11.0/api/org/apache/hadoop/hive/ql/exec/UDAF.html), it seems I need to implement init, aggregate, evaluatePartial, aggregatePartial and evaluate function. Please let me know what are the write functions to implement.
Ritesh