You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by rakesh sharma <ra...@hotmail.com> on 2012/01/10 22:38:43 UTC
Oracle as Pig data store
Hi All,
What is the best way of pushing pig MR output to oracle DB? I would appreciate if a working example can be provided.
Thanks,RS
RE: Oracle as Pig data store
Posted by rakesh sharma <ra...@hotmail.com>.
Thanks! I will give it a try.
> Date: Sat, 14 Jan 2012 15:50:29 +0200
> Subject: Re: Oracle as Pig data store
> From: hakanilter@gmail.com
> To: user@pig.apache.org
>
> Hi,
>
> I haven't used DBStorage yet but I have developed my own database
> export/import classes. You can find source codes and binaries in here:
> http://static.jurfa.com/piql.tar.gz
>
> You can also use these classes with their main methods from console. I hope
> it helped you.
>
>
> On Sat, Jan 14, 2012 at 11:35 AM, rakesh sharma <rakesh_sharma66@hotmail.com
> > wrote:
>
> >
> > It works fine when there is only one store statement in pig script.
> > However, no data gets loaded in the database when there is more than one
> > store statement (loading two different sets of data in different tables).
> > Any ideas?
> >
> > > Date: Tue, 10 Jan 2012 15:38:23 -0800
> > > Subject: Re: Oracle as Pig data store
> > > From: dvryaboy@gmail.com
> > > To: user@pig.apache.org
> > >
> > >
> > http://pig.apache.org/docs/r0.8.1/api/org/apache/pig/piggybank/storage/DBStorage.html
> > >
> > > Warning: this might effectively DDOS your database.
> > >
> > > On Tue, Jan 10, 2012 at 2:12 PM, Russell Jurney
> > > <ru...@gmail.com> wrote:
> > > > I'd be interested in how to do this for MySQL or another JDBC
> > connection.
> > > >
> > > > Russell Jurney
> > > > twitter.com/rjurney
> > > > russell.jurney@gmail.com
> > > > datasyndrome.com
> > > >
> > > > On Jan 10, 2012, at 1:39 PM, rakesh sharma <
> > rakesh_sharma66@hotmail.com> wrote:
> > > >
> > > >>
> > > >> Hi All,
> > > >> What is the best way of pushing pig MR output to oracle DB? I would
> > appreciate if a working example can be provided.
> > > >> Thanks,RS
> >
> >
Re: Oracle as Pig data store
Posted by Hakan İlter <ha...@gmail.com>.
Hi,
I haven't used DBStorage yet but I have developed my own database
export/import classes. You can find source codes and binaries in here:
http://static.jurfa.com/piql.tar.gz
You can also use these classes with their main methods from console. I hope
it helped you.
On Sat, Jan 14, 2012 at 11:35 AM, rakesh sharma <rakesh_sharma66@hotmail.com
> wrote:
>
> It works fine when there is only one store statement in pig script.
> However, no data gets loaded in the database when there is more than one
> store statement (loading two different sets of data in different tables).
> Any ideas?
>
> > Date: Tue, 10 Jan 2012 15:38:23 -0800
> > Subject: Re: Oracle as Pig data store
> > From: dvryaboy@gmail.com
> > To: user@pig.apache.org
> >
> >
> http://pig.apache.org/docs/r0.8.1/api/org/apache/pig/piggybank/storage/DBStorage.html
> >
> > Warning: this might effectively DDOS your database.
> >
> > On Tue, Jan 10, 2012 at 2:12 PM, Russell Jurney
> > <ru...@gmail.com> wrote:
> > > I'd be interested in how to do this for MySQL or another JDBC
> connection.
> > >
> > > Russell Jurney
> > > twitter.com/rjurney
> > > russell.jurney@gmail.com
> > > datasyndrome.com
> > >
> > > On Jan 10, 2012, at 1:39 PM, rakesh sharma <
> rakesh_sharma66@hotmail.com> wrote:
> > >
> > >>
> > >> Hi All,
> > >> What is the best way of pushing pig MR output to oracle DB? I would
> appreciate if a working example can be provided.
> > >> Thanks,RS
>
>
RE: Oracle as Pig data store
Posted by rakesh sharma <ra...@hotmail.com>.
It works fine when there is only one store statement in pig script. However, no data gets loaded in the database when there is more than one store statement (loading two different sets of data in different tables). Any ideas?
> Date: Tue, 10 Jan 2012 15:38:23 -0800
> Subject: Re: Oracle as Pig data store
> From: dvryaboy@gmail.com
> To: user@pig.apache.org
>
> http://pig.apache.org/docs/r0.8.1/api/org/apache/pig/piggybank/storage/DBStorage.html
>
> Warning: this might effectively DDOS your database.
>
> On Tue, Jan 10, 2012 at 2:12 PM, Russell Jurney
> <ru...@gmail.com> wrote:
> > I'd be interested in how to do this for MySQL or another JDBC connection.
> >
> > Russell Jurney
> > twitter.com/rjurney
> > russell.jurney@gmail.com
> > datasyndrome.com
> >
> > On Jan 10, 2012, at 1:39 PM, rakesh sharma <ra...@hotmail.com> wrote:
> >
> >>
> >> Hi All,
> >> What is the best way of pushing pig MR output to oracle DB? I would appreciate if a working example can be provided.
> >> Thanks,RS
Re: Oracle as Pig data store
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
http://pig.apache.org/docs/r0.8.1/api/org/apache/pig/piggybank/storage/DBStorage.html
Warning: this might effectively DDOS your database.
On Tue, Jan 10, 2012 at 2:12 PM, Russell Jurney
<ru...@gmail.com> wrote:
> I'd be interested in how to do this for MySQL or another JDBC connection.
>
> Russell Jurney
> twitter.com/rjurney
> russell.jurney@gmail.com
> datasyndrome.com
>
> On Jan 10, 2012, at 1:39 PM, rakesh sharma <ra...@hotmail.com> wrote:
>
>>
>> Hi All,
>> What is the best way of pushing pig MR output to oracle DB? I would appreciate if a working example can be provided.
>> Thanks,RS
Re: Oracle as Pig data store
Posted by Russell Jurney <ru...@gmail.com>.
I'd be interested in how to do this for MySQL or another JDBC connection.
Russell Jurney
twitter.com/rjurney
russell.jurney@gmail.com
datasyndrome.com
On Jan 10, 2012, at 1:39 PM, rakesh sharma <ra...@hotmail.com> wrote:
>
> Hi All,
> What is the best way of pushing pig MR output to oracle DB? I would appreciate if a working example can be provided.
> Thanks,RS
RE: Invoking pig script from java
Posted by rakesh sharma <ra...@hotmail.com>.
Thanks Prashant, It serves my purpose.
> Date: Wed, 18 Jan 2012 12:27:03 -0800
> Subject: Re: Invoking pig script from java
> From: prash1784@gmail.com
> To: user@pig.apache.org
>
> Hi Rakesh,
>
> Take a look at http://pig.apache.org/docs/r0.9.1/cont.html#embed-java
> You could use PigServer.registerScript
> http://pig.apache.org/docs/r0.9.1/api/org/apache/pig/PigServer.html#registerScript%28java.io.InputStream%29
>
> Thanks,
> Prashant
>
> On Wed, Jan 18, 2012 at 12:12 PM, rakesh sharma <rakesh_sharma66@hotmail.com
> > wrote:
>
> >
> > Hi All,
> > I want to invoke a pig script (stored in a separate file) from java. What
> > are the options for doing it?
> > Thanks,Rakesh
Re: Issue with thrift dependency and cloudera CDH3U2
Posted by Scott Carey <sc...@richrelevance.com>.
Rakesh,
I suggest you try:
'mvn dependency:tree' to see where it is coming from.
You may also prune your dependencies by using maven <exclusion>
declarations if needed.
On 1/18/12 4:03 PM, "Dmitriy Ryaboy" <dv...@gmail.com> wrote:
>Pig does not depend on thrift, something else is pulling it in.
>
>-D
>
>On Wed, Jan 18, 2012 at 2:00 PM, rakesh sharma
><ra...@hotmail.com>wrote:
>
>>
>> Hi All,
>> I am using following dependency in my maven pom:
>> <dependency> <groupId>org.apache.pig</groupId>
>> <artifactId>pig</artifactId> <version>0.8.1-cdh3u2</version>
>> </dependency> <dependency>
>><groupId>org.apache.hadoop</groupId>
>> <artifactId>hadoop-core</artifactId>
>> <version>0.20.2-cdh3u2</version> </dependency>
>> The java code in my project invokes pig script using PigServer class.
>> While compiling, I get following error: "Could not find artifact
>> org.apache.thrift:thrift:jar:0.2.0 in nexus". I am wondering why so old
>> version of thrift is being used by pig here. Also, is it the right
>>artifact
>> to use when I need to use only PigServer class?
>> Thanks,Rakesh
>>
Re: Issue with thrift dependency and cloudera CDH3U2
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
Pig does not depend on thrift, something else is pulling it in.
-D
On Wed, Jan 18, 2012 at 2:00 PM, rakesh sharma
<ra...@hotmail.com>wrote:
>
> Hi All,
> I am using following dependency in my maven pom:
> <dependency> <groupId>org.apache.pig</groupId>
> <artifactId>pig</artifactId> <version>0.8.1-cdh3u2</version>
> </dependency> <dependency> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-core</artifactId>
> <version>0.20.2-cdh3u2</version> </dependency>
> The java code in my project invokes pig script using PigServer class.
> While compiling, I get following error: "Could not find artifact
> org.apache.thrift:thrift:jar:0.2.0 in nexus". I am wondering why so old
> version of thrift is being used by pig here. Also, is it the right artifact
> to use when I need to use only PigServer class?
> Thanks,Rakesh
>
Issue with thrift dependency and cloudera CDH3U2
Posted by rakesh sharma <ra...@hotmail.com>.
Hi All,
I am using following dependency in my maven pom:
<dependency> <groupId>org.apache.pig</groupId> <artifactId>pig</artifactId> <version>0.8.1-cdh3u2</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-core</artifactId> <version>0.20.2-cdh3u2</version> </dependency>
The java code in my project invokes pig script using PigServer class. While compiling, I get following error: "Could not find artifact org.apache.thrift:thrift:jar:0.2.0 in nexus". I am wondering why so old version of thrift is being used by pig here. Also, is it the right artifact to use when I need to use only PigServer class?
Thanks,Rakesh
Re: Invoking pig script from java
Posted by Prashant Kommireddi <pr...@gmail.com>.
Hi Rakesh,
Take a look at http://pig.apache.org/docs/r0.9.1/cont.html#embed-java
You could use PigServer.registerScript
http://pig.apache.org/docs/r0.9.1/api/org/apache/pig/PigServer.html#registerScript%28java.io.InputStream%29
Thanks,
Prashant
On Wed, Jan 18, 2012 at 12:12 PM, rakesh sharma <rakesh_sharma66@hotmail.com
> wrote:
>
> Hi All,
> I want to invoke a pig script (stored in a separate file) from java. What
> are the options for doing it?
> Thanks,Rakesh
Invoking pig script from java
Posted by rakesh sharma <ra...@hotmail.com>.
Hi All,
I want to invoke a pig script (stored in a separate file) from java. What are the options for doing it?
Thanks,Rakesh