You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Wenhao Xu <xu...@gmail.com> on 2010/08/05 05:56:46 UTC
Call Pig from Java
Hi all,
I am new to pig. I am wondering is there any recommended way to call Pig
code from Java?
Is there any Java interface which can be called directly from Java and
makes them work smoothly? It seems each keyword (filter, group, cogrape,
generate) and data types in Pig can have a counterpart in Java by using
Class, interface and data type. Is these Java interface available to Java
programmers to use? If not, why not?
Thanks very much for help!
regards,
Wenhao
--
~_~
Re: Call Pig from Java
Posted by Harsh J <qw...@gmail.com>.
Sorry, forgot the API link:
http://hadoop.apache.org/pig/docs/r0.7.0/api/org/apache/pig/PigServer.html
On Thu, Aug 5, 2010 at 9:31 AM, Harsh J <qw...@gmail.com> wrote:
> You need to use the class PigServer.
>
> PigServer pigServer = new PigServer("mapreduce"); // Or "local" for local mode
> pigServer.registerQuery("A = LOAD ...");
> (...) // Your statements here.
> pigServer.store("A", "filename");
>
> On Thu, Aug 5, 2010 at 9:26 AM, Wenhao Xu <xu...@gmail.com> wrote:
>> Hi all,
>> I am new to pig. I am wondering is there any recommended way to call Pig
>> code from Java?
>> Is there any Java interface which can be called directly from Java and
>> makes them work smoothly? It seems each keyword (filter, group, cogrape,
>> generate) and data types in Pig can have a counterpart in Java by using
>> Class, interface and data type. Is these Java interface available to Java
>> programmers to use? If not, why not?
>> Thanks very much for help!
>>
>> regards,
>> Wenhao
>>
>> --
>> ~_~
>>
>
>
>
> --
> Harsh J
> www.harshj.com
>
--
Harsh J
www.harshj.com
Re: Call Pig from Java
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
No, pig (or any MR stuff) is not really useful for real time queries. Not
unless you can wait at least a couple of minutes.
It would seem you need to look towards HBase, Cassandra and the likes going
under 'NoSQL' umbrella.
On Wed, Aug 4, 2010 at 9:13 PM, Wenhao Xu <xu...@gmail.com> wrote:
> btw, I am considering using it to speedup (parallel) online queries over
> large dataset. Is pig suitable for this, or just suitable for offline large
> data analysis? Will it be a better choice than distributed(parallel)
> database in terms of scalability and latency?
>
> I really like the pig's programming interface. So I want to try to use it
> instead of using parallel database.
>
> Thanks!
>
> cheers,
> W.
>
> On Wed, Aug 4, 2010 at 9:08 PM, Wenhao Xu <xu...@gmail.com> wrote:
>
> > Thanks!
> > Can PigServer handle concurrent requests? Because the store is a
> > synchronous interface, is there any asynchronous one?
> >
> > cheers,
> > W.
> >
> >
> > On Wed, Aug 4, 2010 at 9:01 PM, Harsh J <qw...@gmail.com> wrote:
> >
> >> You need to use the class PigServer.
> >>
> >> PigServer pigServer = new PigServer("mapreduce"); // Or "local" for
> local
> >> mode
> >> pigServer.registerQuery("A = LOAD ...");
> >> (...) // Your statements here.
> >> pigServer.store("A", "filename");
> >>
> >> On Thu, Aug 5, 2010 at 9:26 AM, Wenhao Xu <xu...@gmail.com>
> wrote:
> >> > Hi all,
> >> > I am new to pig. I am wondering is there any recommended way to call
> >> Pig
> >> > code from Java?
> >> > Is there any Java interface which can be called directly from Java
> and
> >> > makes them work smoothly? It seems each keyword (filter, group,
> cogrape,
> >> > generate) and data types in Pig can have a counterpart in Java by
> using
> >> > Class, interface and data type. Is these Java interface available to
> >> Java
> >> > programmers to use? If not, why not?
> >> > Thanks very much for help!
> >> >
> >> > regards,
> >> > Wenhao
> >> >
> >> > --
> >> > ~_~
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
> >> www.harshj.com
> >>
> >
> >
> >
> > --
> > ~_~
> >
>
>
>
> --
> ~_~
>
Re: Call Pig from Java
Posted by Wenhao Xu <xu...@gmail.com>.
btw, I am considering using it to speedup (parallel) online queries over
large dataset. Is pig suitable for this, or just suitable for offline large
data analysis? Will it be a better choice than distributed(parallel)
database in terms of scalability and latency?
I really like the pig's programming interface. So I want to try to use it
instead of using parallel database.
Thanks!
cheers,
W.
On Wed, Aug 4, 2010 at 9:08 PM, Wenhao Xu <xu...@gmail.com> wrote:
> Thanks!
> Can PigServer handle concurrent requests? Because the store is a
> synchronous interface, is there any asynchronous one?
>
> cheers,
> W.
>
>
> On Wed, Aug 4, 2010 at 9:01 PM, Harsh J <qw...@gmail.com> wrote:
>
>> You need to use the class PigServer.
>>
>> PigServer pigServer = new PigServer("mapreduce"); // Or "local" for local
>> mode
>> pigServer.registerQuery("A = LOAD ...");
>> (...) // Your statements here.
>> pigServer.store("A", "filename");
>>
>> On Thu, Aug 5, 2010 at 9:26 AM, Wenhao Xu <xu...@gmail.com> wrote:
>> > Hi all,
>> > I am new to pig. I am wondering is there any recommended way to call
>> Pig
>> > code from Java?
>> > Is there any Java interface which can be called directly from Java and
>> > makes them work smoothly? It seems each keyword (filter, group, cogrape,
>> > generate) and data types in Pig can have a counterpart in Java by using
>> > Class, interface and data type. Is these Java interface available to
>> Java
>> > programmers to use? If not, why not?
>> > Thanks very much for help!
>> >
>> > regards,
>> > Wenhao
>> >
>> > --
>> > ~_~
>> >
>>
>>
>>
>> --
>> Harsh J
>> www.harshj.com
>>
>
>
>
> --
> ~_~
>
--
~_~
Re: Call Pig from Java
Posted by Jeff Zhang <zj...@gmail.com>.
Currently, PigServer is not thread-safe. You can try patches in
http://issues.apache.org/jira/browse/PIG-240
On Thu, Aug 5, 2010 at 12:08 PM, Wenhao Xu <xu...@gmail.com> wrote:
> Thanks!
> Can PigServer handle concurrent requests? Because the store is a
> synchronous interface, is there any asynchronous one?
>
> cheers,
> W.
>
> On Wed, Aug 4, 2010 at 9:01 PM, Harsh J <qw...@gmail.com> wrote:
>
> > You need to use the class PigServer.
> >
> > PigServer pigServer = new PigServer("mapreduce"); // Or "local" for local
> > mode
> > pigServer.registerQuery("A = LOAD ...");
> > (...) // Your statements here.
> > pigServer.store("A", "filename");
> >
> > On Thu, Aug 5, 2010 at 9:26 AM, Wenhao Xu <xu...@gmail.com>
> wrote:
> > > Hi all,
> > > I am new to pig. I am wondering is there any recommended way to call
> > Pig
> > > code from Java?
> > > Is there any Java interface which can be called directly from Java
> and
> > > makes them work smoothly? It seems each keyword (filter, group,
> cogrape,
> > > generate) and data types in Pig can have a counterpart in Java by using
> > > Class, interface and data type. Is these Java interface available to
> Java
> > > programmers to use? If not, why not?
> > > Thanks very much for help!
> > >
> > > regards,
> > > Wenhao
> > >
> > > --
> > > ~_~
> > >
> >
> >
> >
> > --
> > Harsh J
> > www.harshj.com
> >
>
>
>
> --
> ~_~
>
--
Best Regards
Jeff Zhang
Re: Call Pig from Java
Posted by Wenhao Xu <xu...@gmail.com>.
Thanks!
Can PigServer handle concurrent requests? Because the store is a
synchronous interface, is there any asynchronous one?
cheers,
W.
On Wed, Aug 4, 2010 at 9:01 PM, Harsh J <qw...@gmail.com> wrote:
> You need to use the class PigServer.
>
> PigServer pigServer = new PigServer("mapreduce"); // Or "local" for local
> mode
> pigServer.registerQuery("A = LOAD ...");
> (...) // Your statements here.
> pigServer.store("A", "filename");
>
> On Thu, Aug 5, 2010 at 9:26 AM, Wenhao Xu <xu...@gmail.com> wrote:
> > Hi all,
> > I am new to pig. I am wondering is there any recommended way to call
> Pig
> > code from Java?
> > Is there any Java interface which can be called directly from Java and
> > makes them work smoothly? It seems each keyword (filter, group, cogrape,
> > generate) and data types in Pig can have a counterpart in Java by using
> > Class, interface and data type. Is these Java interface available to Java
> > programmers to use? If not, why not?
> > Thanks very much for help!
> >
> > regards,
> > Wenhao
> >
> > --
> > ~_~
> >
>
>
>
> --
> Harsh J
> www.harshj.com
>
--
~_~
Re: Call Pig from Java
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
In my personal and very not so long lived opinion PigServer is not very
useful as it doesn't run directly pig scripts.
I actually integrated Grunt setup in a spring bean and been able to run pig
scripts that way initialized as a resource. it also takes care of project
classpath on the hadoop side so no registration of any jars is necessary,
anything in our project classpath (as built by maven) is automatically added
to the backend classpaths. Also feeding in script parameters thru spring
injections is also pretty consistent with our spring use and useful.
This requires some work (couple of days) to dig up various grant parameters
and PigContext parameters but i think it pays off with convenience of using
regular grunt script and ease of UDF access.
-Dmitriy
On Wed, Aug 4, 2010 at 9:01 PM, Harsh J <qw...@gmail.com> wrote:
> You need to use the class PigServer.
>
> PigServer pigServer = new PigServer("mapreduce"); // Or "local" for local
> mode
> pigServer.registerQuery("A = LOAD ...");
> (...) // Your statements here.
> pigServer.store("A", "filename");
>
> On Thu, Aug 5, 2010 at 9:26 AM, Wenhao Xu <xu...@gmail.com> wrote:
> > Hi all,
> > I am new to pig. I am wondering is there any recommended way to call
> Pig
> > code from Java?
> > Is there any Java interface which can be called directly from Java and
> > makes them work smoothly? It seems each keyword (filter, group, cogrape,
> > generate) and data types in Pig can have a counterpart in Java by using
> > Class, interface and data type. Is these Java interface available to Java
> > programmers to use? If not, why not?
> > Thanks very much for help!
> >
> > regards,
> > Wenhao
> >
> > --
> > ~_~
> >
>
>
>
> --
> Harsh J
> www.harshj.com
>
Re: Call Pig from Java
Posted by Harsh J <qw...@gmail.com>.
You need to use the class PigServer.
PigServer pigServer = new PigServer("mapreduce"); // Or "local" for local mode
pigServer.registerQuery("A = LOAD ...");
(...) // Your statements here.
pigServer.store("A", "filename");
On Thu, Aug 5, 2010 at 9:26 AM, Wenhao Xu <xu...@gmail.com> wrote:
> Hi all,
> I am new to pig. I am wondering is there any recommended way to call Pig
> code from Java?
> Is there any Java interface which can be called directly from Java and
> makes them work smoothly? It seems each keyword (filter, group, cogrape,
> generate) and data types in Pig can have a counterpart in Java by using
> Class, interface and data type. Is these Java interface available to Java
> programmers to use? If not, why not?
> Thanks very much for help!
>
> regards,
> Wenhao
>
> --
> ~_~
>
--
Harsh J
www.harshj.com