You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@pig.apache.org by V Satish Kumar <sa...@mkhoj.com> on 2009/11/17 07:17:51 UTC

Storing Pig Output in a Database

I would like the output from my pig task to be stored in a mysql database.
Is there a way of storing this pig output directly into a mysql database?


Thanks,
Satish

Re: Storing Pig Output in a Database

Posted by Pratyush Banerjee <pr...@aol.com>.

Hi Satish,

There are two ways to do your job.

1.  Dump your results into a file in HDFS as the records in the table 
would look like, and then do a load in file directly from the file to 
the MySql table
2. Create a UDF to dump your results directly into a database. For 
writing this UIDF you would need to implement the StoreFunc class.

Refer to the UDF 
Manual(http://hadoop.apache.org/pig/docs/r0.3.0/udf.html) in the apache 
Pig website for further details on how to implement the same.

Thanks and regards,
Pratyush Banerjee

V Satish Kumar wrote:
>
> I would like the output from my pig task to be stored in a mysql 
> database.
> Is there a way of storing this pig output directly into a mysql database?
>
>
> Thanks,
> Satish

Re: Storing Pig Output in a Database

Posted by V Satish Kumar <sa...@mkhoj.com>.

Thanks Dmitriy & Pratyush.

I also figured out that there is a file 
'contrib/zebra/src/java/org/apache/hadoop/zebra/pig/TableStorer.java', 
which seems to have an example of storing output.


Satish


Dmitriy Ryaboy wrote:
> Hi Satish,
> You can write a StoreFunc that will do whatever you need to do to your output.
> Here is an earlier message with some thoughts on writing Pig job
> outputs to a database:
> http://markmail.org/message/qo7oz3lhywnf43mq
>
> On Mon, Nov 16, 2009 at 10:17 PM, V Satish Kumar <sa...@mkhoj.com> wrote:
>   
>> I would like the output from my pig task to be stored in a mysql database.
>> Is there a way of storing this pig output directly into a mysql database?
>>
>>
>> Thanks,
>> Satish
>>
>>     
> .
>
>

Re: Storing Pig Output in a Database

Posted by Dmitriy Ryaboy <dv...@gmail.com>.

Hi Satish,
You can write a StoreFunc that will do whatever you need to do to your output.
Here is an earlier message with some thoughts on writing Pig job
outputs to a database:
http://markmail.org/message/qo7oz3lhywnf43mq

On Mon, Nov 16, 2009 at 10:17 PM, V Satish Kumar <sa...@mkhoj.com> wrote:
>
> I would like the output from my pig task to be stored in a mysql database.
> Is there a way of storing this pig output directly into a mysql database?
>
>
> Thanks,
> Satish
>