You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Olga Natkovich <ol...@yahoo-inc.com> on 2009/05/05 18:12:53 UTC
Multiquery support
Hi,
Yesterday we committed support for multiquery into Pig trunk. This
functionality allows to share scans and other computations across
multiple stores within the same pig script. The details are outlined in
http://wiki.apache.org/pig/PigMultiQueryPerformanceSpecification. This
functionality will go into Pig 0.3.0 to be release in late May or early
June but please give it a try and see whether it improves performance of
your queries. We also made some changes to commands like describe and
explain to work with multiquery - the changes are outlined in the same
document.
Olga
Re: Multiquery support
Posted by Tamir Kamara <ta...@gmail.com>.
You're right - with ant 1.6.5 the build fails but with 1.7.1 it's builds
just fine.
Thanks.
On Tue, May 5, 2009 at 10:33 PM, Olga Natkovich <ol...@yahoo-inc.com> wrote:
> What version of ant do you have on your system? I don't have problems
> building and have ant 1.7.0 and 1.7.1 installed on my machines.
>
> Also, this is not related to multiquery commit but looks like
> integration with ivy that was also done recently.
>
> Olga
>
> > -----Original Message-----
> > From: Tamir Kamara [mailto:tamirkamara@gmail.com]
> > Sent: Tuesday, May 05, 2009 10:40 AM
> > To: pig-user@hadoop.apache.org
> > Subject: Re: Multiquery support
> >
> > Hi Olga,
> >
> > Recent changes cause the build to fail. ant output is below.
> >
> > Thanks,
> > Tamir
> >
> > Buildfile: build.xml
> >
> > init:
> > [mkdir] Created dir:
> > /app/home/tkamara/pig1/src-gen/org/apache/pig/impl/logicalLayer/parser
> > [mkdir] Created dir:
> > /app/home/tkamara/pig1/src-gen/org/apache/pig/tools/pigscript/parser
> > [mkdir] Created dir:
> > /app/home/tkamara/pig1/src-gen/org/apache/pig/tools/parameters
> > [mkdir] Created dir: /app/home/tkamara/pig1/build/pig-0.3.0-dev
> > [mkdir] Created dir: /app/home/tkamara/pig1/build/classes
> > [mkdir] Created dir: /app/home/tkamara/pig1/build/test/classes
> > [mkdir] Created dir:
> > /app/home/tkamara/pig1/test/org/apache/pig/test/utils/dotGraph/parser
> > [mkdir] Created dir:
> > /app/home/tkamara/pig1/src-gen/org/apache/pig/data/parser
> >
> > ivy-download:
> > [get] Getting:
> > http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy
> > -2.0.0-rc2.jar
> > [get] To: /app/home/tkamara/pig1/ivy/ivy-2.0.0-rc2.jar
> >
> > ivy-init-dirs:
> > [mkdir] Created dir: /app/home/tkamara/pig1/build/ivy
> > [mkdir] Created dir: /app/home/tkamara/pig1/build/ivy/lib
> > [mkdir] Created dir: /app/home/tkamara/pig1/build/ivy/report
> > [mkdir] Created dir: /app/home/tkamara/pig1/build/ivy/maven
> >
> > ivy-probe-antlib:
> >
> > BUILD FAILED
> > /app/home/tkamara/pig1/build.xml:727: Class
> > org.apache.tools.ant.taskdefs.ConditionTask doesn't support
> > the nested "typefound" element.
> >
> >
> >
> > On Tue, May 5, 2009 at 7:12 PM, Olga Natkovich
> > <ol...@yahoo-inc.com> wrote:
> >
> > > Hi,
> > >
> > > Yesterday we committed support for multiquery into Pig trunk. This
> > > functionality allows to share scans and other computations across
> > > multiple stores within the same pig script. The details are
> > outlined
> > > in
> > http://wiki.apache.org/pig/PigMultiQueryPerformanceSpecification.
> > > This functionality will go into Pig 0.3.0 to be release in
> > late May or
> > > early June but please give it a try and see whether it improves
> > > performance of your queries. We also made some changes to commands
> > > like describe and explain to work with multiquery - the changes are
> > > outlined in the same document.
> > >
> > > Olga
> > >
> >
>
RE: Multiquery support
Posted by Olga Natkovich <ol...@yahoo-inc.com>.
What version of ant do you have on your system? I don't have problems
building and have ant 1.7.0 and 1.7.1 installed on my machines.
Also, this is not related to multiquery commit but looks like
integration with ivy that was also done recently.
Olga
> -----Original Message-----
> From: Tamir Kamara [mailto:tamirkamara@gmail.com]
> Sent: Tuesday, May 05, 2009 10:40 AM
> To: pig-user@hadoop.apache.org
> Subject: Re: Multiquery support
>
> Hi Olga,
>
> Recent changes cause the build to fail. ant output is below.
>
> Thanks,
> Tamir
>
> Buildfile: build.xml
>
> init:
> [mkdir] Created dir:
> /app/home/tkamara/pig1/src-gen/org/apache/pig/impl/logicalLayer/parser
> [mkdir] Created dir:
> /app/home/tkamara/pig1/src-gen/org/apache/pig/tools/pigscript/parser
> [mkdir] Created dir:
> /app/home/tkamara/pig1/src-gen/org/apache/pig/tools/parameters
> [mkdir] Created dir: /app/home/tkamara/pig1/build/pig-0.3.0-dev
> [mkdir] Created dir: /app/home/tkamara/pig1/build/classes
> [mkdir] Created dir: /app/home/tkamara/pig1/build/test/classes
> [mkdir] Created dir:
> /app/home/tkamara/pig1/test/org/apache/pig/test/utils/dotGraph/parser
> [mkdir] Created dir:
> /app/home/tkamara/pig1/src-gen/org/apache/pig/data/parser
>
> ivy-download:
> [get] Getting:
> http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy
> -2.0.0-rc2.jar
> [get] To: /app/home/tkamara/pig1/ivy/ivy-2.0.0-rc2.jar
>
> ivy-init-dirs:
> [mkdir] Created dir: /app/home/tkamara/pig1/build/ivy
> [mkdir] Created dir: /app/home/tkamara/pig1/build/ivy/lib
> [mkdir] Created dir: /app/home/tkamara/pig1/build/ivy/report
> [mkdir] Created dir: /app/home/tkamara/pig1/build/ivy/maven
>
> ivy-probe-antlib:
>
> BUILD FAILED
> /app/home/tkamara/pig1/build.xml:727: Class
> org.apache.tools.ant.taskdefs.ConditionTask doesn't support
> the nested "typefound" element.
>
>
>
> On Tue, May 5, 2009 at 7:12 PM, Olga Natkovich
> <ol...@yahoo-inc.com> wrote:
>
> > Hi,
> >
> > Yesterday we committed support for multiquery into Pig trunk. This
> > functionality allows to share scans and other computations across
> > multiple stores within the same pig script. The details are
> outlined
> > in
> http://wiki.apache.org/pig/PigMultiQueryPerformanceSpecification.
> > This functionality will go into Pig 0.3.0 to be release in
> late May or
> > early June but please give it a try and see whether it improves
> > performance of your queries. We also made some changes to commands
> > like describe and explain to work with multiquery - the changes are
> > outlined in the same document.
> >
> > Olga
> >
>
Re: Multiquery support
Posted by Tamir Kamara <ta...@gmail.com>.
Hi Olga,
Recent changes cause the build to fail. ant output is below.
Thanks,
Tamir
Buildfile: build.xml
init:
[mkdir] Created dir:
/app/home/tkamara/pig1/src-gen/org/apache/pig/impl/logicalLayer/parser
[mkdir] Created dir:
/app/home/tkamara/pig1/src-gen/org/apache/pig/tools/pigscript/parser
[mkdir] Created dir:
/app/home/tkamara/pig1/src-gen/org/apache/pig/tools/parameters
[mkdir] Created dir: /app/home/tkamara/pig1/build/pig-0.3.0-dev
[mkdir] Created dir: /app/home/tkamara/pig1/build/classes
[mkdir] Created dir: /app/home/tkamara/pig1/build/test/classes
[mkdir] Created dir:
/app/home/tkamara/pig1/test/org/apache/pig/test/utils/dotGraph/parser
[mkdir] Created dir:
/app/home/tkamara/pig1/src-gen/org/apache/pig/data/parser
ivy-download:
[get] Getting:
http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy-2.0.0-rc2.jar
[get] To: /app/home/tkamara/pig1/ivy/ivy-2.0.0-rc2.jar
ivy-init-dirs:
[mkdir] Created dir: /app/home/tkamara/pig1/build/ivy
[mkdir] Created dir: /app/home/tkamara/pig1/build/ivy/lib
[mkdir] Created dir: /app/home/tkamara/pig1/build/ivy/report
[mkdir] Created dir: /app/home/tkamara/pig1/build/ivy/maven
ivy-probe-antlib:
BUILD FAILED
/app/home/tkamara/pig1/build.xml:727: Class
org.apache.tools.ant.taskdefs.ConditionTask doesn't support the nested
"typefound" element.
On Tue, May 5, 2009 at 7:12 PM, Olga Natkovich <ol...@yahoo-inc.com> wrote:
> Hi,
>
> Yesterday we committed support for multiquery into Pig trunk. This
> functionality allows to share scans and other computations across
> multiple stores within the same pig script. The details are outlined in
> http://wiki.apache.org/pig/PigMultiQueryPerformanceSpecification. This
> functionality will go into Pig 0.3.0 to be release in late May or early
> June but please give it a try and see whether it improves performance of
> your queries. We also made some changes to commands like describe and
> explain to work with multiquery - the changes are outlined in the same
> document.
>
> Olga
>