You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Olga Natkovich <ol...@yahoo-inc.com> on 2009/03/12 02:01:05 UTC

How are you using Pig?

Hi,
 
If you have done some interesting work using Pig, we would like to know!
Could you please, send a brief description of the kind of work you are
doing with Pig and what have you learned from working woth Pig: things
that worked well, things that you would like to be improved.
 
Thanks,
 
Olga

Re: How are you using Pig?

Posted by prasenjit mukherjee <pr...@gmail.com>.
Hi Olga,
   I have already mentioned about it my earlier post.

   I have implemented T. Hoffmann's PLSI ( Probabilistic Latent Semantic
Indexing ) based on EM ( Expectation-Maximization ) algorithm in pig. The
E/M login was implemented in pig in ~ 30-35 lines of pig-latin statements.
The implementation is available in the open-source apache project mahout  as
a part of the following patch :
https://issues.apache.org/jira/browse/MAHOUT-106.

Though the code works fine, I feel it will be lot easier for developers to
use pig if you allow some framework where I can write my UDFs in some
interpreted language like jgroovy,python,perl,. It could even be pig as
well.

-Prasen


On Thu, Mar 12, 2009 at 6:31 AM, Olga Natkovich <ol...@yahoo-inc.com> wrote:

> Hi,
>
> If you have done some interesting work using Pig, we would like to know!
> Could you please, send a brief description of the kind of work you are
> doing with Pig and what have you learned from working woth Pig: things
> that worked well, things that you would like to be improved.
>
> Thanks,
>
> Olga
>

RE: How are you using Pig?

Posted by Pradeep Kamath <pr...@yahoo-inc.com>.
Have you tried the latest code from svn? There have been some fixes for
union related issues in the recent past. If you still see issues, you
should file a jira with details.

-Pradeep

-----Original Message-----
From: Dhruv Matani [mailto:dhruv.m@directi.com] 
Sent: Tuesday, March 31, 2009 11:45 PM
To: pig-user@hadoop.apache.org
Subject: Re: How are you using Pig?

Hello all,
  We are using pig to perform transformation on a lot of data. This
includes doing join on some relations. However, we have been facing a
lot of problems with using UNION, and it is causing a lot of data loss,
because of which we have had to do certain steps manually. However, all
in all, pig has considerably reduced the amount of work we would have
had to put in to perform ad-hoc queries and transformations on data.

Regards,
-Dhruv.


zhang jianfeng wrote:
> I am using pig to parse site logs, generate some stat like unique
users,
> download count and so on.  Currently we are in the initial stage, so
all the
> work is done in local. Later we will move the pig to EC2, and all the
data
> will be put on S3.
>
> My suggestion: sometimes the exception message is not so informative.
>
>
>
> On Wed, Apr 1, 2009 at 10:49 AM, Earl Cahill <ca...@yahoo.com>
wrote:
>
>   
>> I am using pig to parse apache logs and generate aggregation files,
which
>> data I put into mysql.  From mysql I generate stats for the sites on
my free
>> hosting site.  I used to work for a web hosting company and we spent
quite a
>> bit of effort to get to my current level.  My pig script is fewer
than I
>> think 50 lines and it does the work of a whole processing / parsing
>> framework we built.
>>
>> Earl
>>
>>  http://blog.spack.net
>> http://holaservers.com
>>
>>
>>
>>
>> ________________________________
>> From: Olga Natkovich <ol...@yahoo-inc.com>
>> To: pig-user@hadoop.apache.org
>> Sent: Wednesday, March 11, 2009 7:01:05 PM
>> Subject: How are you using Pig?
>>
>> Hi,
>>
>> If you have done some interesting work using Pig, we would like to
know!
>> Could you please, send a brief description of the kind of work you
are
>> doing with Pig and what have you learned from working woth Pig:
things
>> that worked well, things that you would like to be improved.
>>
>> Thanks,
>>
>> Olga
>>
>>
>>
>>
>>     
>
>   


Re: How are you using Pig?

Posted by Chris Olston <ol...@yahoo-inc.com>.
> we have been facing a
> lot of problems with using UNION, and it is causing a lot of data loss,
> because of which we have had to do certain steps manually.

Sorry to hear that. Did you file a bug report? If not it's unlikely to get
fixed :)

http://issues.apache.org/jira/browse/PIG

Cheers,

Chris

 
> zhang jianfeng wrote:
>> I am using pig to parse site logs, generate some stat like unique users,
>> download count and so on.  Currently we are in the initial stage, so all the
>> work is done in local. Later we will move the pig to EC2, and all the data
>> will be put on S3.
>> 
>> My suggestion: sometimes the exception message is not so informative.
>> 
>> 
>> 
>> On Wed, Apr 1, 2009 at 10:49 AM, Earl Cahill <ca...@yahoo.com> wrote:
>> 
>>   
>>> I am using pig to parse apache logs and generate aggregation files, which
>>> data I put into mysql.  From mysql I generate stats for the sites on my free
>>> hosting site.  I used to work for a web hosting company and we spent quite a
>>> bit of effort to get to my current level.  My pig script is fewer than I
>>> think 50 lines and it does the work of a whole processing / parsing
>>> framework we built.
>>> 
>>> Earl
>>> 
>>>  http://blog.spack.net
>>> http://holaservers.com
>>> 
>>> 
>>> 
>>> 
>>> ________________________________
>>> From: Olga Natkovich <ol...@yahoo-inc.com>
>>> To: pig-user@hadoop.apache.org
>>> Sent: Wednesday, March 11, 2009 7:01:05 PM
>>> Subject: How are you using Pig?
>>> 
>>> Hi,
>>> 
>>> If you have done some interesting work using Pig, we would like to know!
>>> Could you please, send a brief description of the kind of work you are
>>> doing with Pig and what have you learned from working woth Pig: things
>>> that worked well, things that you would like to be improved.
>>> 
>>> Thanks,
>>> 
>>> Olga
>>> 
>>> 
>>> 
>>> 
>>>     
>> 
>>   
> 

--
Christopher Olston, Ph.D.
Sr. Research Scientist
Yahoo! Research





Re: How are you using Pig?

Posted by Dhruv Matani <dh...@directi.com>.
Hello all,
  We are using pig to perform transformation on a lot of data. This
includes doing join on some relations. However, we have been facing a
lot of problems with using UNION, and it is causing a lot of data loss,
because of which we have had to do certain steps manually. However, all
in all, pig has considerably reduced the amount of work we would have
had to put in to perform ad-hoc queries and transformations on data.

Regards,
-Dhruv.


zhang jianfeng wrote:
> I am using pig to parse site logs, generate some stat like unique users,
> download count and so on.  Currently we are in the initial stage, so all the
> work is done in local. Later we will move the pig to EC2, and all the data
> will be put on S3.
>
> My suggestion: sometimes the exception message is not so informative.
>
>
>
> On Wed, Apr 1, 2009 at 10:49 AM, Earl Cahill <ca...@yahoo.com> wrote:
>
>   
>> I am using pig to parse apache logs and generate aggregation files, which
>> data I put into mysql.  From mysql I generate stats for the sites on my free
>> hosting site.  I used to work for a web hosting company and we spent quite a
>> bit of effort to get to my current level.  My pig script is fewer than I
>> think 50 lines and it does the work of a whole processing / parsing
>> framework we built.
>>
>> Earl
>>
>>  http://blog.spack.net
>> http://holaservers.com
>>
>>
>>
>>
>> ________________________________
>> From: Olga Natkovich <ol...@yahoo-inc.com>
>> To: pig-user@hadoop.apache.org
>> Sent: Wednesday, March 11, 2009 7:01:05 PM
>> Subject: How are you using Pig?
>>
>> Hi,
>>
>> If you have done some interesting work using Pig, we would like to know!
>> Could you please, send a brief description of the kind of work you are
>> doing with Pig and what have you learned from working woth Pig: things
>> that worked well, things that you would like to be improved.
>>
>> Thanks,
>>
>> Olga
>>
>>
>>
>>
>>     
>
>   


Re: How are you using Pig?

Posted by zhang jianfeng <zj...@gmail.com>.
I am using pig to parse site logs, generate some stat like unique users,
download count and so on.  Currently we are in the initial stage, so all the
work is done in local. Later we will move the pig to EC2, and all the data
will be put on S3.

My suggestion: sometimes the exception message is not so informative.



On Wed, Apr 1, 2009 at 10:49 AM, Earl Cahill <ca...@yahoo.com> wrote:

> I am using pig to parse apache logs and generate aggregation files, which
> data I put into mysql.  From mysql I generate stats for the sites on my free
> hosting site.  I used to work for a web hosting company and we spent quite a
> bit of effort to get to my current level.  My pig script is fewer than I
> think 50 lines and it does the work of a whole processing / parsing
> framework we built.
>
> Earl
>
>  http://blog.spack.net
> http://holaservers.com
>
>
>
>
> ________________________________
> From: Olga Natkovich <ol...@yahoo-inc.com>
> To: pig-user@hadoop.apache.org
> Sent: Wednesday, March 11, 2009 7:01:05 PM
> Subject: How are you using Pig?
>
> Hi,
>
> If you have done some interesting work using Pig, we would like to know!
> Could you please, send a brief description of the kind of work you are
> doing with Pig and what have you learned from working woth Pig: things
> that worked well, things that you would like to be improved.
>
> Thanks,
>
> Olga
>
>
>
>

Re: How are you using Pig?

Posted by Earl Cahill <ca...@yahoo.com>.
I am using pig to parse apache logs and generate aggregation files, which data I put into mysql.  From mysql I generate stats for the sites on my free hosting site.  I used to work for a web hosting company and we spent quite a bit of effort to get to my current level.  My pig script is fewer than I think 50 lines and it does the work of a whole processing / parsing framework we built.

Earl

 http://blog.spack.net
http://holaservers.com




________________________________
From: Olga Natkovich <ol...@yahoo-inc.com>
To: pig-user@hadoop.apache.org
Sent: Wednesday, March 11, 2009 7:01:05 PM
Subject: How are you using Pig?

Hi,

If you have done some interesting work using Pig, we would like to know!
Could you please, send a brief description of the kind of work you are
doing with Pig and what have you learned from working woth Pig: things
that worked well, things that you would like to be improved.

Thanks,

Olga



      

Re: How are you using Pig?

Posted by nitesh bhatia <ni...@gmail.com>.
Hi
I am working on adaptive query optimization part. Currently the
problem I am facing is cost-estimation of query. Current pig system
builds physical plan and feeds it to hadoop. Final query gets executed
over hadoop, so as such their is no way to compute query cost before
execution. For multiquery the optimization idea is to compute cost if
various plans by executing them for small amount of data (say 1% of
number of lowest data table) and get the cost. At the run time the
system can choose the best plan based on statistics generated by this
sample data. This approach is however doesn't appears to be perfect
but it can work out as a solution. I am looking for a perfect solution
which I will let you know shortly.

Other ideas that will increase functionality of Pig can be
1. Unix Mail Method - User can directly mail the output - file to some
remote addresss
2. Web Based interface - User can submit scripts remotely through a
web based interface. A functionaly to view the results can be added.
Also the above mentioned mail method can be incorporated to e-mail the
file in any zip ( eg. .bzip, gunzip, .Z) format.

I have few more ideas in pipeline. I will add in the list as soon as I
get some clarity in thoughts.

--nitesh

On Thu, Mar 12, 2009 at 6:31 AM, Olga Natkovich <ol...@yahoo-inc.com> wrote:
> Hi,
>
> If you have done some interesting work using Pig, we would like to know!
> Could you please, send a brief description of the kind of work you are
> doing with Pig and what have you learned from working woth Pig: things
> that worked well, things that you would like to be improved.
>
> Thanks,
>
> Olga
>



-- 
Nitesh Bhatia
Dhirubhai Ambani Institute of Information & Communication Technology
Gandhinagar
Gujarat

"Life is never perfect. It just depends where you draw the line."

visit:
http://www.awaaaz.com - connecting through music
http://www.volstreet.com - lets volunteer for better tomorrow
http://www.instibuzz.com - Voice opinions, Transact easily, Have fun

RE: How are you using Pig?

Posted by Olga Natkovich <ol...@yahoo-inc.com>.
Hi Charles,

Thanks for the information. My reply to your asks is inline. 

> -----Original Message-----
> From: charles du [mailto:taiping.du@gmail.com] 
> Sent: Friday, March 13, 2009 8:15 PM
> To: pig-user@hadoop.apache.org
> Subject: Re: How are you using Pig?
> 
> Hi Olga:
> 
> We are using to pig latin programs to process search engine 
> query logs. To us, Pig programs are easier to maintain, and 
> less error-prone than native java programs. It is an 
> excellent piece of work.
> 
> A few things could make our life even easier:
> 
> 1. support functions/command line parameters. Many of our pig 
> programs diff only in their inputs/outputs, and a few 
> filtering conditions in some cases.
> So We got a lot of duplicate code among pig programs. It will 
> be helpful to us if we can change program behaviors based 
> command line parameters, or build a pig function and call it 
> with different arguments.

Pig supports parameter substitution. Would that serve your needs?

http://wiki.apache.org/pig-data/attachments/FrontPage/attachments/plrm.h
tm#_Parameter_Substitution
> 
> 2. support outputing multiple records for each input record. 
> Very often, we need produce multiple records for each input 
> record. Right now, we use hadoop streaming /or java programs to do it.

You can use streaming support in Pig as well:
http://wiki.apache.org/pig-data/attachments/FrontPage/attachments/plrm.h
tm#_STREAM_
> 
> 3. Allow specifying job priority in a pig program. We can set 
> job name in a pig program, but cannot set its priority.

We have not had people asking for this feature yet. Please, open a JIRA
to track the request. Also, feel free to contribute! :).

Olga

> 
> Regards
> 
> charles.
> 
> 
> On Wed, Mar 11, 2009 at 6:01 PM, Olga Natkovich 
> <ol...@yahoo-inc.com> wrote:
> 
> > Hi,
> >
> > If you have done some interesting work using Pig, we would 
> like to know!
> > Could you please, send a brief description of the kind of 
> work you are 
> > doing with Pig and what have you learned from working woth 
> Pig: things 
> > that worked well, things that you would like to be improved.
> >
> > Thanks,
> >
> > Olga
> >
> 
> 
> 
> --
> tp
> 

Re: How are you using Pig?

Posted by "Ivanka M. Li" <di...@gmail.com>.
Hi Olga

I am curious too. Is there interesting projects going on using Pig? We
expect more replies.

Ivanka

Re: How are you using Pig?

Posted by charles du <ta...@gmail.com>.
Hi Olga:

We are using to pig latin programs to process search engine query logs. To
us, Pig programs are easier to maintain, and less error-prone than native
java programs. It is an excellent piece of work.

A few things could make our life even easier:

1. support functions/command line parameters. Many of our pig programs diff
only in their inputs/outputs, and a few filtering conditions in some cases.
So We got a lot of duplicate code among pig programs. It will be helpful to
us if we can change program behaviors based command line parameters, or
build a pig function and call it with different arguments.

2. support outputing multiple records for each input record. Very often, we
need produce multiple records for each input record. Right now, we use
hadoop streaming /or java programs to do it.

3. Allow specifying job priority in a pig program. We can set job name in a
pig program, but cannot set its priority.

Regards

charles.


On Wed, Mar 11, 2009 at 6:01 PM, Olga Natkovich <ol...@yahoo-inc.com> wrote:

> Hi,
>
> If you have done some interesting work using Pig, we would like to know!
> Could you please, send a brief description of the kind of work you are
> doing with Pig and what have you learned from working woth Pig: things
> that worked well, things that you would like to be improved.
>
> Thanks,
>
> Olga
>



-- 
tp

RE: How are you using Pig?

Posted by Olga Natkovich <ol...@yahoo-inc.com>.
Hi Baldo,

Thanks for detailed information. My comments inline. 

> -----Original Message-----
> From: Baldo Faieta [mailto:baldofaieta@yahoo.com] 
> Sent: Thursday, March 12, 2009 5:35 AM
> To: pig-user@hadoop.apache.org
> Subject: Re: How are you using Pig?
> 
> Hi Olga,
> 
> I have used pig to implement the Adsorption Algorithm described in the
> paper:
> Video Suggestion and Discovery for YouTube: Taking Random 
> Walks Through the View Graph, Baluja et al. 2008
> 
> We're thinking on using the algorithm for implementing 
> implicit recommendations.  I need to wrap it up, but when it 
> is in a decent state I'll post it in case others are interested on it.
> 
> Using pig was great, but I would have wanted to have more 
> parameterization of the scripts and some way to do loops.  I 
> guess being able to do some kind of inline scripting might be 
> interesting as now I am forced to do all of the loops outside 
> the scripts with shell scripts and it is a bit clunky.

I assume you already tried  Pig's parameter substitution feature. What
was missing there?

As for loops, we are thinking along the lines of embeding Pig into a
scripting language like Python or Perl. Would this serve your needs?

> More specific problems I encountered was that I started 
> filtering 27,000 log files, which resulted in 27,000 results, 
> even though all of these result files could have been 
> collapsed in a few files.  Then, when I ran additional steps, 
> it would be very slow because pig launched a map job for each 
> result file.  I had to collapse them manually and then it 
> would be very fast.

By default, Hadoop does not give multiple files to a map. You will need
a custom slicer to support this. If you do write one, please, contribute
it back to Pig.

> Also, at some point I needed to put a unique index number per 
> record, and I couldn't find a way to do it, so I had to 
> result to use a random GUID, which works ok, though I would 
> have preferred the index number.


There is no easy way to provide index without going through a single
reducer. I think GUID is probably the right way to go here.

Olga

> 
> Anyway, that's all for now,
> 
> Baldo
> 
> On 12 Mar 2009, at 02:01, Olga Natkovich wrote:
> 
> > Hi,
> >
> > If you have done some interesting work using Pig, we would like to 
> > know!
> > Could you please, send a brief description of the kind of 
> work you are 
> > doing with Pig and what have you learned from working woth 
> Pig: things 
> > that worked well, things that you would like to be improved.
> >
> > Thanks,
> >
> > Olga
> 
> 

Re: How are you using Pig?

Posted by Vadim Zaliva <kr...@gmail.com>.
I am using PIG (along with some custom Java task) to calculate Image
recommendations
at Imageshack.us. I am using it as a rapid-prototyping language to
test some algorithms on
huge amounts of data.

Vadim

On Thu, Mar 12, 2009 at 05:34, Baldo Faieta <ba...@yahoo.com> wrote:
> Hi Olga,
>
> I have used pig to implement the Adsorption Algorithm described in the
> paper:
> Video Suggestion and Discovery for YouTube: Taking Random Walks Through the
> View Graph, Baluja et al. 2008
>
> We're thinking on using the algorithm for implementing implicit
> recommendations.  I need to wrap it up, but when it is in a decent state
> I'll post it in case others are interested on it.
>
> Using pig was great, but I would have wanted to have more parameterization
> of the scripts and some way to do loops.  I guess being able to do some kind
> of inline scripting might be interesting as now I am forced to do all of the
> loops outside the scripts with shell scripts and it is a bit clunky.
>
> More specific problems I encountered was that I started filtering 27,000 log
> files, which resulted in 27,000 results, even though all of these result
> files could have been collapsed in a few files.  Then, when I ran additional
> steps, it would be very slow because pig launched a map job for each result
> file.  I had to collapse them manually and then it would be very fast.
>
> Also, at some point I needed to put a unique index number per record, and I
> couldn't find a way to do it, so I had to result to use a random GUID, which
> works ok, though I would have preferred the index number.
>
> Anyway, that's all for now,
>
> Baldo
>
> On 12 Mar 2009, at 02:01, Olga Natkovich wrote:
>
>> Hi,
>>
>> If you have done some interesting work using Pig, we would like to know!
>> Could you please, send a brief description of the kind of work you are
>> doing with Pig and what have you learned from working woth Pig: things
>> that worked well, things that you would like to be improved.
>>
>> Thanks,
>>
>> Olga
>
>

Re: How are you using Pig?

Posted by Baldo Faieta <ba...@yahoo.com>.
Hi Olga,

I have used pig to implement the Adsorption Algorithm described in the  
paper:
Video Suggestion and Discovery for YouTube: Taking Random Walks  
Through the View Graph, Baluja et al. 2008

We're thinking on using the algorithm for implementing implicit  
recommendations.  I need to wrap it up, but when it is in a decent  
state I'll post it in case others are interested on it.

Using pig was great, but I would have wanted to have more  
parameterization of the scripts and some way to do loops.  I guess  
being able to do some kind of inline scripting might be interesting as  
now I am forced to do all of the loops outside the scripts with shell  
scripts and it is a bit clunky.

More specific problems I encountered was that I started filtering  
27,000 log files, which resulted in 27,000 results, even though all of  
these result files could have been collapsed in a few files.  Then,  
when I ran additional steps, it would be very slow because pig  
launched a map job for each result file.  I had to collapse them  
manually and then it would be very fast.

Also, at some point I needed to put a unique index number per record,  
and I couldn't find a way to do it, so I had to result to use a random  
GUID, which works ok, though I would have preferred the index number.

Anyway, that's all for now,

Baldo

On 12 Mar 2009, at 02:01, Olga Natkovich wrote:

> Hi,
>
> If you have done some interesting work using Pig, we would like to  
> know!
> Could you please, send a brief description of the kind of work you are
> doing with Pig and what have you learned from working woth Pig: things
> that worked well, things that you would like to be improved.
>
> Thanks,
>
> Olga