You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Viraj Bhat <vi...@yahoo-inc.com> on 2010/06/22 01:03:17 UTC
Using Distributed Cache in Hive UDF's??
Hi all,
I have a lookup function in hive which looks if a certain pattern is
present in a large text file. I upload this text file to HDFS. I hope to
use this text file in my UDF evaluate() method.
Is there some documentation I can look up?
Distributed Cache relies on
lookupFiles = DistributedCache.getLocalCacheFiles(job);
job is of type JobConf.
Where do I get the JobConf object from within the UDF?
Thanks
Viraj
RE: Using Distributed Cache in Hive UDF's??
Posted by Viraj Bhat <vi...@yahoo-inc.com>.
Hi Edward,
I was able to use the distributed cache, using the set
mapred.cache.files option. I could read the files locally using standard
java api's.
Thanks
Viraj
________________________________
From: Edward Capriolo [mailto:edlinuxguru@gmail.com]
Sent: Tuesday, June 22, 2010 7:44 AM
To: hive-user@hadoop.apache.org
Subject: Re: Using Distributed Cache in Hive UDF's??
Shameless plug.
IF you put a file in the distributed cache it is in the working
directory of the UDF so you do not need fancy hadoop isms to access it.
Shameless plug:
My geo-ip-udf does exactly this.
http://www.jointhegrid.com/hive-udf-geo-ip-jtg/index.jsp
http://www.jointhegrid.com/svn/hive-udf-geo-ip-jtg/
Edward
On Mon, Jun 21, 2010 at 7:03 PM, Viraj Bhat <vi...@yahoo-inc.com> wrote:
Hi all,
I have a lookup function in hive which looks if a certain pattern is
present in a large text file. I upload this text file to HDFS. I hope to
use this text file in my UDF evaluate() method.
Is there some documentation I can look up?
Distributed Cache relies on
lookupFiles = DistributedCache.getLocalCacheFiles(job);
job is of type JobConf.
Where do I get the JobConf object from within the UDF?
Thanks
Viraj
Re: Using Distributed Cache in Hive UDF's??
Posted by Edward Capriolo <ed...@gmail.com>.
Shameless plug.
IF you put a file in the distributed cache it is in the working directory of
the UDF so you do not need fancy hadoop isms to access it.
Shameless plug:
My geo-ip-udf does exactly this.
http://www.jointhegrid.com/hive-udf-geo-ip-jtg/index.jsp
http://www.jointhegrid.com/svn/hive-udf-geo-ip-jtg/
Edward
On Mon, Jun 21, 2010 at 7:03 PM, Viraj Bhat <vi...@yahoo-inc.com> wrote:
> Hi all,
>
> I have a lookup function in hive which looks if a certain pattern is
> present in a large text file. I upload this text file to HDFS. I hope to use
> this text file in my UDF evaluate() method.
>
> Is there some documentation I can look up?
>
> Distributed Cache relies on
>
> lookupFiles = DistributedCache.getLocalCacheFiles(job);
>
> job is of type JobConf.
>
> Where do I get the JobConf object from within the UDF?
>
>
>
> Thanks
>
> Viraj
>