You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@pig.apache.org by Mance Rylan <ma...@gmail.com> on 2011/07/26 12:40:10 UTC

RDBMS and Pig

Hi,

I have googled a lot about if I can have Pig interact with an RDBMS.
Is there any way to have Pig load data from an rdbms? perform some
operations and then store data on Hadoop?

Thanks,
Mance

Re: RDBMS and Pig

Posted by Mance Rylan <ma...@gmail.com>.

Thanks guys, though SQLLOADER is part of piggybank, but its a start for me.

--Mance

On Tue, Jul 26, 2011 at 9:59 PM, Dmitriy Ryaboy <dv...@gmail.com> wrote:

> I don't think there ever was a general-purpose SQLLOADER. It's just
> explaining the concept -- that if you want to, you can write one, and you
> aren't limited to reading from HDFS.
>
> D
>
>
> On Tue, Jul 26, 2011 at 7:21 AM, Juan Martin Pampliega <
> jpampliega@gmail.com
> > wrote:
>
> > I found about in the PigLatin manual.
> >
> http://pig.apache.org/docs/r0.8.1/piglatin_ref1.html#Backward+Compatibility
> >
> > But maybe it is deprecated as there is no reference in the Java Doc.
> >
> > On Tue, Jul 26, 2011 at 10:11 AM, Jacob Perkins
> > <ja...@gmail.com>wrote:
> >
> > > You might also take a look at
> > >
> > >
> > >
> >
> http://pig.apache.org/docs/r0.8.1/api/index.html?org/apache/pig/piggybank/storage/DBStorage.html
> > >
> > > which is going to require that you 'register' the piggybank jar. I'm
> not
> > > entirely sure that's what you're looking for either though.
> > >
> > > I'm curious, in the same java doc I see no reference to SQLLOADER,
> where
> > > did you find that?
> > >
> > > --jacob
> > > @thedatachef
> > >
> > > On Tue, 2011-07-26 at 10:00 -0300, Juan Martin Pampliega wrote:
> > > > From what I read, data from a RDBMS is read by the command:
> > > > LOAD 'sql://mytable' USING SQLLOADER();
> > > >
> > > > I couldn't find any info on how the connection parameters to the
> > database
> > > > are passed and which databases are supported.
> > > >
> > > > On Tue, Jul 26, 2011 at 7:40 AM, Mance Rylan <ma...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I have googled a lot about if I can have Pig interact with an
> RDBMS.
> > > > > Is there any way to have Pig load data from an rdbms? perform some
> > > > > operations and then store data on Hadoop?
> > > > >
> > > > > Thanks,
> > > > > Mance
> > > > >
> > >
> > >
> > >
> >
>

Re: RDBMS and Pig

Posted by Dmitriy Ryaboy <dv...@gmail.com>.

I don't think there ever was a general-purpose SQLLOADER. It's just
explaining the concept -- that if you want to, you can write one, and you
aren't limited to reading from HDFS.

D


On Tue, Jul 26, 2011 at 7:21 AM, Juan Martin Pampliega <jpampliega@gmail.com
> wrote:

> I found about in the PigLatin manual.
> http://pig.apache.org/docs/r0.8.1/piglatin_ref1.html#Backward+Compatibility
>
> But maybe it is deprecated as there is no reference in the Java Doc.
>
> On Tue, Jul 26, 2011 at 10:11 AM, Jacob Perkins
> <ja...@gmail.com>wrote:
>
> > You might also take a look at
> >
> >
> >
> http://pig.apache.org/docs/r0.8.1/api/index.html?org/apache/pig/piggybank/storage/DBStorage.html
> >
> > which is going to require that you 'register' the piggybank jar. I'm not
> > entirely sure that's what you're looking for either though.
> >
> > I'm curious, in the same java doc I see no reference to SQLLOADER, where
> > did you find that?
> >
> > --jacob
> > @thedatachef
> >
> > On Tue, 2011-07-26 at 10:00 -0300, Juan Martin Pampliega wrote:
> > > From what I read, data from a RDBMS is read by the command:
> > > LOAD 'sql://mytable' USING SQLLOADER();
> > >
> > > I couldn't find any info on how the connection parameters to the
> database
> > > are passed and which databases are supported.
> > >
> > > On Tue, Jul 26, 2011 at 7:40 AM, Mance Rylan <ma...@gmail.com>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > I have googled a lot about if I can have Pig interact with an RDBMS.
> > > > Is there any way to have Pig load data from an rdbms? perform some
> > > > operations and then store data on Hadoop?
> > > >
> > > > Thanks,
> > > > Mance
> > > >
> >
> >
> >
>

Re: RDBMS and Pig

Posted by Juan Martin Pampliega <jp...@gmail.com>.

I found about in the PigLatin manual.
http://pig.apache.org/docs/r0.8.1/piglatin_ref1.html#Backward+Compatibility

But maybe it is deprecated as there is no reference in the Java Doc.

On Tue, Jul 26, 2011 at 10:11 AM, Jacob Perkins
<ja...@gmail.com>wrote:

> You might also take a look at
>
>
> http://pig.apache.org/docs/r0.8.1/api/index.html?org/apache/pig/piggybank/storage/DBStorage.html
>
> which is going to require that you 'register' the piggybank jar. I'm not
> entirely sure that's what you're looking for either though.
>
> I'm curious, in the same java doc I see no reference to SQLLOADER, where
> did you find that?
>
> --jacob
> @thedatachef
>
> On Tue, 2011-07-26 at 10:00 -0300, Juan Martin Pampliega wrote:
> > From what I read, data from a RDBMS is read by the command:
> > LOAD 'sql://mytable' USING SQLLOADER();
> >
> > I couldn't find any info on how the connection parameters to the database
> > are passed and which databases are supported.
> >
> > On Tue, Jul 26, 2011 at 7:40 AM, Mance Rylan <ma...@gmail.com>
> wrote:
> >
> > > Hi,
> > >
> > > I have googled a lot about if I can have Pig interact with an RDBMS.
> > > Is there any way to have Pig load data from an rdbms? perform some
> > > operations and then store data on Hadoop?
> > >
> > > Thanks,
> > > Mance
> > >
>
>
>

Re: RDBMS and Pig

Posted by Jacob Perkins <ja...@gmail.com>.

You might also take a look at

http://pig.apache.org/docs/r0.8.1/api/index.html?org/apache/pig/piggybank/storage/DBStorage.html

which is going to require that you 'register' the piggybank jar. I'm not
entirely sure that's what you're looking for either though.

I'm curious, in the same java doc I see no reference to SQLLOADER, where
did you find that?

--jacob
@thedatachef

On Tue, 2011-07-26 at 10:00 -0300, Juan Martin Pampliega wrote:
> From what I read, data from a RDBMS is read by the command:
> LOAD 'sql://mytable' USING SQLLOADER();
> 
> I couldn't find any info on how the connection parameters to the database
> are passed and which databases are supported.
> 
> On Tue, Jul 26, 2011 at 7:40 AM, Mance Rylan <ma...@gmail.com> wrote:
> 
> > Hi,
> >
> > I have googled a lot about if I can have Pig interact with an RDBMS.
> > Is there any way to have Pig load data from an rdbms? perform some
> > operations and then store data on Hadoop?
> >
> > Thanks,
> > Mance
> >

Re: RDBMS and Pig

Posted by Juan Martin Pampliega <jp...@gmail.com>.

>From what I read, data from a RDBMS is read by the command:
LOAD 'sql://mytable' USING SQLLOADER();

I couldn't find any info on how the connection parameters to the database
are passed and which databases are supported.

On Tue, Jul 26, 2011 at 7:40 AM, Mance Rylan <ma...@gmail.com> wrote:

> Hi,
>
> I have googled a lot about if I can have Pig interact with an RDBMS.
> Is there any way to have Pig load data from an rdbms? perform some
> operations and then store data on Hadoop?
>
> Thanks,
> Mance
>