You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Johannes Rußek <jo...@io-consulting.net> on 2010/03/17 18:34:15 UTC
GeoIP UDF
Hello Everybody,
i've been searching the web trying to find a nice to way to use GeoIP
from pig / hadoop.
So far the only things i've been able to find is a perlscript to use
with streaming and some suggestions to use the maxmind geoip java api
from within hadoop (by packaging the geoip.dat file in the jar). I'd
like to avoid streaming though.
question is, how can i access that from within pig? do i have to write a
udf "wrapper" to use it? or can i use any function that is available in
hadoop from pig?
Thanks,
Johannes
Re: GeoIP UDF
Posted by Johannes Rußek <jo...@io-consulting.net>.
Hello Dmitriy,
so i've implemented the wrapper using a public static LookupService to
store the GeoIP object. Is this what you though of when you said "static
service"?
Attached you can find the UDF, it's very simple so far and only resolved
to the country.
I'm not sure if I'm handling the errors correctly so far.
I was able to build and run it with pig 0.6.0 on a few sample datasets
and it did work. i'm running it on 40G of input data now, let's see.
Thanks and regards,
Johannes
Re: GeoIP UDF
Posted by Kevin Weil <ke...@gmail.com>.
Your UDF is getting excecuted on an arbitrary datanode, and the java process
is trying to load the *local* file ./GeoIP.dat. You could use
FileSystem.open to get an inputstream to the HDFS version you have, but then
all datanodes will be trying to access that one (or three with replication)
file, which may not be efficient. The way we handle this is to have our
automated deploy/machine setup put GeoIP.dat in a specified location on all
datanodes. That is, don't put it in HDFS, put it in a specified location on
the local filesystem, and then your code will work.
Kevin
On Thu, Mar 18, 2010 at 11:58 AM, Johannes Rußek <
johannes.russek@io-consulting.net> wrote:
> Hello Everybody,
> i've written a wrapper class for the GeoIP api, but now i'm trying to
> access the GeoIP.dat file which i've added to hdfs via hadoop dfs -put
> GeoIP.dat GeoIP.dat and added to the cache in pig.properties via
> mapred.cache.files=hdfs://localhost:8020/user/root/GeoIP.dat
> however, it seems the geoip api is unable to open the file with
> './GeoIP.dat' as path. What should i use for this?
> Regards,
> Johannes
>
Re: GeoIP UDF
Posted by Johannes Rußek <jo...@io-consulting.net>.
Hello Everybody,
i've written a wrapper class for the GeoIP api, but now i'm trying to
access the GeoIP.dat file which i've added to hdfs via hadoop dfs -put
GeoIP.dat GeoIP.dat and added to the cache in pig.properties via
mapred.cache.files=hdfs://localhost:8020/user/root/GeoIP.dat
however, it seems the geoip api is unable to open the file with
'./GeoIP.dat' as path. What should i use for this?
Regards,
Johannes
Re: GeoIP UDF
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
Yes on both counts.
On Wed, Mar 17, 2010 at 3:51 PM, Johannes Rußek <
johannes.russek@io-consulting.net> wrote:
> Hi Dmitriy!
>
>
> Am 17.03.2010 18:46, schrieb Dmitriy Ryaboy:
>
> The recipe on the cloudera blog works. It's made a little extra complex
>> for
>> educational purposes -- you should just put the geoip db on all the nodes
>> so
>> that you don't have to package it up and ship every time.
>>
>>
>
> I just want to clarify, you do mean this:
> http://www.cloudera.com/blog/2009/06/analyzing-apache-logs-with-pig/right?
> unfortunately this does use streaming to solve this.
> However, i found this:
> http://www.data-miners.com/blog/2009/12/hadoop-and-mapreduce-what-country-is-ip.html
> which seems interesting. Getting this into a UDF is just a matter of
> wrapping it?
>
> Thanks!
> Johannes
>
Re: GeoIP UDF
Posted by Johannes Rußek <jo...@io-consulting.net>.
Hi Dmitriy!
Am 17.03.2010 18:46, schrieb Dmitriy Ryaboy:
> The recipe on the cloudera blog works. It's made a little extra complex for
> educational purposes -- you should just put the geoip db on all the nodes so
> that you don't have to package it up and ship every time.
>
I just want to clarify, you do mean this:
http://www.cloudera.com/blog/2009/06/analyzing-apache-logs-with-pig/ right?
unfortunately this does use streaming to solve this.
However, i found this:
http://www.data-miners.com/blog/2009/12/hadoop-and-mapreduce-what-country-is-ip.html
which seems interesting. Getting this into a UDF is just a matter of
wrapping it?
Thanks!
Johannes
Re: GeoIP UDF
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
The recipe on the cloudera blog works. It's made a little extra complex for
educational purposes -- you should just put the geoip db on all the nodes so
that you don't have to package it up and ship every time.
To be more efficient, you need to wrap the java api in a UDF, and preload
the geoip database into memory (make it a static service, and you can take
advantage of JVM reuse, too!).
-D
On Wed, Mar 17, 2010 at 10:34 AM, Johannes Rußek <
johannes.russek@io-consulting.net> wrote:
> Hello Everybody,
> i've been searching the web trying to find a nice to way to use GeoIP from
> pig / hadoop.
> So far the only things i've been able to find is a perlscript to use with
> streaming and some suggestions to use the maxmind geoip java api from within
> hadoop (by packaging the geoip.dat file in the jar). I'd like to avoid
> streaming though.
> question is, how can i access that from within pig? do i have to write a
> udf "wrapper" to use it? or can i use any function that is available in
> hadoop from pig?
> Thanks,
> Johannes
>