You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Shai Harel <sh...@mythings.com> on 2011/05/05 15:46:04 UTC
TOP function isn't working
i'm trying to run this code on Amazon Elastic MapReduce
...
score = FOREACH a GENERATE A, B, C,D,
1*v + 20*c + 30*a + 100*p AS score;
gg = group score by (A, B, C);
topResults = FOREACH gg {
result = TOP(20, 4, score);
GENERATE FLATTEN(result);
};
dump topResults;
and i get this error
2011-05-05 13:41:18,665 [main] ERROR org.apache.pig.tools.grunt.Grunt -
ERROR 1070: Could not resolve TOP using imports: [, org.apache.pig.builtin.,
org.apache.pig.impl.builtin.]
does anyone know where the TOP function is defined?
(what do i need to register)
Re: TOP function isn't working
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
ant jar-withouthadoop
(really, that's what it's called).
On Sun, May 8, 2011 at 8:28 AM, Jonathan Coveney <jc...@gmail.com> wrote:
> Alex,
>
> Do you know what is involved in compiling pig without the Hadoop jar so that
> you can point it to your own? Or compiling it with your own version of
> Hadoop?
>
> 2011/5/8 Alex Rovner <al...@gmail.com>
>
>> It's not that simple. Pig jar is bundled with hadoop libraries that will be
>> incompatible with the amazon version of hadoop. You need to compile the pig
>> jar that does not have the hadoop libraries bundles with it. There might be
>> one available for download somewhere as well.
>>
>> Alex
>>
>> Sent from my iPhone
>>
>> On May 8, 2011, at 5:33 AM, Shai Harel <sh...@mythings.com> wrote:
>>
>> > i tried to replace
>> > /home/hadoop/.versions/pig-0.6/lib/pig/pig-0.6-amzn.jar
>> > with pig-0.8.1-core.jar
>> >
>> > and i get the following error:
>> >
>> > 11/05/08 09:28:47 ERROR pig.Main: ERROR 2999: Unexpected internal error.
>> > java.net.URISyntaxException: Illegal character in scheme name at index
>> 19:
>> > pig.initial.fs.name=hdfs:///
>> > 11/05/08 09:28:47 WARN pig.Main: There is no log file to write to.
>> > 11/05/08 09:28:47 ERROR pig.Main: java.lang.RuntimeException:
>> > java.net.URISyntaxException: Illegal character in scheme name at index
>> 19:
>> > pig.initial.fs.name=hdfs:///
>> > at
>> > org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:742)
>> > at org.apache.pig.Main.run(Main.java:299)
>> > at org.apache.pig.Main.main(Main.java:107)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>> > Caused by: java.net.URISyntaxException: Illegal character in scheme name
>> at
>> > index 19: pig.initial.fs.name=hdfs:///
>> > at java.net.URI$Parser.fail(URI.java:2809)
>> > at java.net.URI$Parser.checkChars(URI.java:2982)
>> > at java.net.URI$Parser.parse(URI.java:3009)
>> > at java.net.URI.<init>(URI.java:578)
>> > at
>> > org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:740)
>> > ... 7 more
>> >
>> >
>> >
>> > On Thu, May 5, 2011 at 7:47 PM, Renato Marroquín Mogrovejo <
>> > renatoj.marroquin@gmail.com> wrote:
>> >
>> >> The only way to use built in TOP function would be if you manually
>> >> install pig 0.8.1 because Amazon only supports pig 0.6 [1]
>> >> So things in 7 and 8 are still out of Amazon EMR.
>> >>
>> >> [1] http://aws.amazon.com/elasticmapreduce/faqs/#pig-7
>> >>
>> >>
>> >> Renato M.
>> >>
>> >> 2011/5/5 Gianmarco <gi...@gmail.com>:
>> >>> You can use the builtin TOP() function.
>> >>>
>> >>> http://pig.apache.org/docs/r0.8.1/api/org/apache/pig/builtin/TOP.html
>> >>>
>> >>>
>> >>> Cheers,
>> >>> --
>> >>> Gianmarco De Francisci Morales
>> >>>
>> >>>
>> >>> On Thu, May 5, 2011 at 15:49, Shai Harel <sh...@mythings.com>
>> >> wrote:
>> >>>
>> >>>> OK, i'v included piggybank
>> >>>>
>> >>>> register s3://my_path/piggybank.jar;
>> >>>> DEFINE TOP org.apache.pig.piggybank.evaluation.util.Top();
>> >>>>
>> >>>> and now i get this error
>> >>>>
>> >>>> Pig Stack Trace
>> >>>> ---------------
>> >>>> ERROR 1045: Could not infer the matching function for
>> >>>> org.apache.pig.piggybank.evaluation.util.Top as multiple or none of
>> them
>> >>>> fit. Please use an explicit cast.
>> >>>>
>> >>>> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable
>> >> to
>> >>>> store alias 78
>> >>>> at
>> >> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1004)
>> >>>> at org.apache.pig.PigServer.registerQuery(PigServer.java:386)
>> >>>> at
>> >>>>
>> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:739)
>> >>>>
>> >>>>
>> >>>> On Thu, May 5, 2011 at 4:46 PM, Shai Harel <sh...@mythings.com>
>> >>>> wrote:
>> >>>>
>> >>>>> i'm trying to run this code on Amazon Elastic MapReduce
>> >>>>>
>> >>>>> ...
>> >>>>> score = FOREACH a GENERATE A, B, C,D,
>> >>>>> 1*v + 20*c + 30*a + 100*p AS score;
>> >>>>> gg = group score by (A, B, C);
>> >>>>>
>> >>>>> topResults = FOREACH gg {
>> >>>>> result = TOP(20, 4, score);
>> >>>>> GENERATE FLATTEN(result);
>> >>>>> };
>> >>>>> dump topResults;
>> >>>>>
>> >>>>> and i get this error
>> >>>>>
>> >>>>> 2011-05-05 13:41:18,665 [main] ERROR org.apache.pig.tools.grunt.Grunt
>> >> -
>> >>>>> ERROR 1070: Could not resolve TOP using imports: [,
>> >>>> org.apache.pig.builtin.,
>> >>>>> org.apache.pig.impl.builtin.]
>> >>>>>
>> >>>>> does anyone know where the TOP function is defined?
>> >>>>> (what do i need to register)
>> >>>>>
>> >>>>
>> >>>
>> >>
>>
>
Re: TOP function isn't working
Posted by Jonathan Coveney <jc...@gmail.com>.
Alex,
Do you know what is involved in compiling pig without the Hadoop jar so that
you can point it to your own? Or compiling it with your own version of
Hadoop?
2011/5/8 Alex Rovner <al...@gmail.com>
> It's not that simple. Pig jar is bundled with hadoop libraries that will be
> incompatible with the amazon version of hadoop. You need to compile the pig
> jar that does not have the hadoop libraries bundles with it. There might be
> one available for download somewhere as well.
>
> Alex
>
> Sent from my iPhone
>
> On May 8, 2011, at 5:33 AM, Shai Harel <sh...@mythings.com> wrote:
>
> > i tried to replace
> > /home/hadoop/.versions/pig-0.6/lib/pig/pig-0.6-amzn.jar
> > with pig-0.8.1-core.jar
> >
> > and i get the following error:
> >
> > 11/05/08 09:28:47 ERROR pig.Main: ERROR 2999: Unexpected internal error.
> > java.net.URISyntaxException: Illegal character in scheme name at index
> 19:
> > pig.initial.fs.name=hdfs:///
> > 11/05/08 09:28:47 WARN pig.Main: There is no log file to write to.
> > 11/05/08 09:28:47 ERROR pig.Main: java.lang.RuntimeException:
> > java.net.URISyntaxException: Illegal character in scheme name at index
> 19:
> > pig.initial.fs.name=hdfs:///
> > at
> > org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:742)
> > at org.apache.pig.Main.run(Main.java:299)
> > at org.apache.pig.Main.main(Main.java:107)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> > Caused by: java.net.URISyntaxException: Illegal character in scheme name
> at
> > index 19: pig.initial.fs.name=hdfs:///
> > at java.net.URI$Parser.fail(URI.java:2809)
> > at java.net.URI$Parser.checkChars(URI.java:2982)
> > at java.net.URI$Parser.parse(URI.java:3009)
> > at java.net.URI.<init>(URI.java:578)
> > at
> > org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:740)
> > ... 7 more
> >
> >
> >
> > On Thu, May 5, 2011 at 7:47 PM, Renato Marroquín Mogrovejo <
> > renatoj.marroquin@gmail.com> wrote:
> >
> >> The only way to use built in TOP function would be if you manually
> >> install pig 0.8.1 because Amazon only supports pig 0.6 [1]
> >> So things in 7 and 8 are still out of Amazon EMR.
> >>
> >> [1] http://aws.amazon.com/elasticmapreduce/faqs/#pig-7
> >>
> >>
> >> Renato M.
> >>
> >> 2011/5/5 Gianmarco <gi...@gmail.com>:
> >>> You can use the builtin TOP() function.
> >>>
> >>> http://pig.apache.org/docs/r0.8.1/api/org/apache/pig/builtin/TOP.html
> >>>
> >>>
> >>> Cheers,
> >>> --
> >>> Gianmarco De Francisci Morales
> >>>
> >>>
> >>> On Thu, May 5, 2011 at 15:49, Shai Harel <sh...@mythings.com>
> >> wrote:
> >>>
> >>>> OK, i'v included piggybank
> >>>>
> >>>> register s3://my_path/piggybank.jar;
> >>>> DEFINE TOP org.apache.pig.piggybank.evaluation.util.Top();
> >>>>
> >>>> and now i get this error
> >>>>
> >>>> Pig Stack Trace
> >>>> ---------------
> >>>> ERROR 1045: Could not infer the matching function for
> >>>> org.apache.pig.piggybank.evaluation.util.Top as multiple or none of
> them
> >>>> fit. Please use an explicit cast.
> >>>>
> >>>> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable
> >> to
> >>>> store alias 78
> >>>> at
> >> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1004)
> >>>> at org.apache.pig.PigServer.registerQuery(PigServer.java:386)
> >>>> at
> >>>>
> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:739)
> >>>>
> >>>>
> >>>> On Thu, May 5, 2011 at 4:46 PM, Shai Harel <sh...@mythings.com>
> >>>> wrote:
> >>>>
> >>>>> i'm trying to run this code on Amazon Elastic MapReduce
> >>>>>
> >>>>> ...
> >>>>> score = FOREACH a GENERATE A, B, C,D,
> >>>>> 1*v + 20*c + 30*a + 100*p AS score;
> >>>>> gg = group score by (A, B, C);
> >>>>>
> >>>>> topResults = FOREACH gg {
> >>>>> result = TOP(20, 4, score);
> >>>>> GENERATE FLATTEN(result);
> >>>>> };
> >>>>> dump topResults;
> >>>>>
> >>>>> and i get this error
> >>>>>
> >>>>> 2011-05-05 13:41:18,665 [main] ERROR org.apache.pig.tools.grunt.Grunt
> >> -
> >>>>> ERROR 1070: Could not resolve TOP using imports: [,
> >>>> org.apache.pig.builtin.,
> >>>>> org.apache.pig.impl.builtin.]
> >>>>>
> >>>>> does anyone know where the TOP function is defined?
> >>>>> (what do i need to register)
> >>>>>
> >>>>
> >>>
> >>
>
Re: TOP function isn't working
Posted by Shai Harel <sh...@mythings.com>.
ok, so i gave up with the TOP function and converted this statement
score = FOREACH a GENERATE A, B, C,D,
1*v + 20*c + 30*a + 100*p AS score;
gg = group score by (A, B, C);
topResults = FOREACH gg {
result = TOP(20, 4, score);
GENERATE FLATTEN(result);
};
To
score = FOREACH a GENERATE A, B, C, D,
1*v + 20*c + 30*a + 100*p AS score;
gg = group score by (A, B, C);
topResults = FOREACH gg {
s = ORDER score BY score DESC;
l = limit s 10;
generate FLATTEN(l);
}
I think it gives the same result...
On Sun, May 8, 2011 at 3:43 PM, Alex Rovner <al...@gmail.com> wrote:
> It's not that simple. Pig jar is bundled with hadoop libraries that will be
> incompatible with the amazon version of hadoop. You need to compile the pig
> jar that does not have the hadoop libraries bundles with it. There might be
> one available for download somewhere as well.
>
> Alex
>
> Sent from my iPhone
>
> On May 8, 2011, at 5:33 AM, Shai Harel <sh...@mythings.com> wrote:
>
> > i tried to replace
> > /home/hadoop/.versions/pig-0.6/lib/pig/pig-0.6-amzn.jar
> > with pig-0.8.1-core.jar
> >
> > and i get the following error:
> >
> > 11/05/08 09:28:47 ERROR pig.Main: ERROR 2999: Unexpected internal error.
> > java.net.URISyntaxException: Illegal character in scheme name at index
> 19:
> > pig.initial.fs.name=hdfs:///
> > 11/05/08 09:28:47 WARN pig.Main: There is no log file to write to.
> > 11/05/08 09:28:47 ERROR pig.Main: java.lang.RuntimeException:
> > java.net.URISyntaxException: Illegal character in scheme name at index
> 19:
> > pig.initial.fs.name=hdfs:///
> > at
> > org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:742)
> > at org.apache.pig.Main.run(Main.java:299)
> > at org.apache.pig.Main.main(Main.java:107)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> > Caused by: java.net.URISyntaxException: Illegal character in scheme name
> at
> > index 19: pig.initial.fs.name=hdfs:///
> > at java.net.URI$Parser.fail(URI.java:2809)
> > at java.net.URI$Parser.checkChars(URI.java:2982)
> > at java.net.URI$Parser.parse(URI.java:3009)
> > at java.net.URI.<init>(URI.java:578)
> > at
> > org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:740)
> > ... 7 more
> >
> >
> >
> > On Thu, May 5, 2011 at 7:47 PM, Renato Marroquín Mogrovejo <
> > renatoj.marroquin@gmail.com> wrote:
> >
> >> The only way to use built in TOP function would be if you manually
> >> install pig 0.8.1 because Amazon only supports pig 0.6 [1]
> >> So things in 7 and 8 are still out of Amazon EMR.
> >>
> >> [1] http://aws.amazon.com/elasticmapreduce/faqs/#pig-7
> >>
> >>
> >> Renato M.
> >>
> >> 2011/5/5 Gianmarco <gi...@gmail.com>:
> >>> You can use the builtin TOP() function.
> >>>
> >>> http://pig.apache.org/docs/r0.8.1/api/org/apache/pig/builtin/TOP.html
> >>>
> >>>
> >>> Cheers,
> >>> --
> >>> Gianmarco De Francisci Morales
> >>>
> >>>
> >>> On Thu, May 5, 2011 at 15:49, Shai Harel <sh...@mythings.com>
> >> wrote:
> >>>
> >>>> OK, i'v included piggybank
> >>>>
> >>>> register s3://my_path/piggybank.jar;
> >>>> DEFINE TOP org.apache.pig.piggybank.evaluation.util.Top();
> >>>>
> >>>> and now i get this error
> >>>>
> >>>> Pig Stack Trace
> >>>> ---------------
> >>>> ERROR 1045: Could not infer the matching function for
> >>>> org.apache.pig.piggybank.evaluation.util.Top as multiple or none of
> them
> >>>> fit. Please use an explicit cast.
> >>>>
> >>>> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable
> >> to
> >>>> store alias 78
> >>>> at
> >> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1004)
> >>>> at org.apache.pig.PigServer.registerQuery(PigServer.java:386)
> >>>> at
> >>>>
> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:739)
> >>>>
> >>>>
> >>>> On Thu, May 5, 2011 at 4:46 PM, Shai Harel <sh...@mythings.com>
> >>>> wrote:
> >>>>
> >>>>> i'm trying to run this code on Amazon Elastic MapReduce
> >>>>>
> >>>>> ...
> >>>>> score = FOREACH a GENERATE A, B, C,D,
> >>>>> 1*v + 20*c + 30*a + 100*p AS score;
> >>>>> gg = group score by (A, B, C);
> >>>>>
> >>>>> topResults = FOREACH gg {
> >>>>> result = TOP(20, 4, score);
> >>>>> GENERATE FLATTEN(result);
> >>>>> };
> >>>>> dump topResults;
> >>>>>
> >>>>> and i get this error
> >>>>>
> >>>>> 2011-05-05 13:41:18,665 [main] ERROR org.apache.pig.tools.grunt.Grunt
> >> -
> >>>>> ERROR 1070: Could not resolve TOP using imports: [,
> >>>> org.apache.pig.builtin.,
> >>>>> org.apache.pig.impl.builtin.]
> >>>>>
> >>>>> does anyone know where the TOP function is defined?
> >>>>> (what do i need to register)
> >>>>>
> >>>>
> >>>
> >>
>
Re: TOP function isn't working
Posted by Alex Rovner <al...@gmail.com>.
It's not that simple. Pig jar is bundled with hadoop libraries that will be incompatible with the amazon version of hadoop. You need to compile the pig jar that does not have the hadoop libraries bundles with it. There might be one available for download somewhere as well.
Alex
Sent from my iPhone
On May 8, 2011, at 5:33 AM, Shai Harel <sh...@mythings.com> wrote:
> i tried to replace
> /home/hadoop/.versions/pig-0.6/lib/pig/pig-0.6-amzn.jar
> with pig-0.8.1-core.jar
>
> and i get the following error:
>
> 11/05/08 09:28:47 ERROR pig.Main: ERROR 2999: Unexpected internal error.
> java.net.URISyntaxException: Illegal character in scheme name at index 19:
> pig.initial.fs.name=hdfs:///
> 11/05/08 09:28:47 WARN pig.Main: There is no log file to write to.
> 11/05/08 09:28:47 ERROR pig.Main: java.lang.RuntimeException:
> java.net.URISyntaxException: Illegal character in scheme name at index 19:
> pig.initial.fs.name=hdfs:///
> at
> org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:742)
> at org.apache.pig.Main.run(Main.java:299)
> at org.apache.pig.Main.main(Main.java:107)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.net.URISyntaxException: Illegal character in scheme name at
> index 19: pig.initial.fs.name=hdfs:///
> at java.net.URI$Parser.fail(URI.java:2809)
> at java.net.URI$Parser.checkChars(URI.java:2982)
> at java.net.URI$Parser.parse(URI.java:3009)
> at java.net.URI.<init>(URI.java:578)
> at
> org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:740)
> ... 7 more
>
>
>
> On Thu, May 5, 2011 at 7:47 PM, Renato Marroquín Mogrovejo <
> renatoj.marroquin@gmail.com> wrote:
>
>> The only way to use built in TOP function would be if you manually
>> install pig 0.8.1 because Amazon only supports pig 0.6 [1]
>> So things in 7 and 8 are still out of Amazon EMR.
>>
>> [1] http://aws.amazon.com/elasticmapreduce/faqs/#pig-7
>>
>>
>> Renato M.
>>
>> 2011/5/5 Gianmarco <gi...@gmail.com>:
>>> You can use the builtin TOP() function.
>>>
>>> http://pig.apache.org/docs/r0.8.1/api/org/apache/pig/builtin/TOP.html
>>>
>>>
>>> Cheers,
>>> --
>>> Gianmarco De Francisci Morales
>>>
>>>
>>> On Thu, May 5, 2011 at 15:49, Shai Harel <sh...@mythings.com>
>> wrote:
>>>
>>>> OK, i'v included piggybank
>>>>
>>>> register s3://my_path/piggybank.jar;
>>>> DEFINE TOP org.apache.pig.piggybank.evaluation.util.Top();
>>>>
>>>> and now i get this error
>>>>
>>>> Pig Stack Trace
>>>> ---------------
>>>> ERROR 1045: Could not infer the matching function for
>>>> org.apache.pig.piggybank.evaluation.util.Top as multiple or none of them
>>>> fit. Please use an explicit cast.
>>>>
>>>> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable
>> to
>>>> store alias 78
>>>> at
>> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1004)
>>>> at org.apache.pig.PigServer.registerQuery(PigServer.java:386)
>>>> at
>>>> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:739)
>>>>
>>>>
>>>> On Thu, May 5, 2011 at 4:46 PM, Shai Harel <sh...@mythings.com>
>>>> wrote:
>>>>
>>>>> i'm trying to run this code on Amazon Elastic MapReduce
>>>>>
>>>>> ...
>>>>> score = FOREACH a GENERATE A, B, C,D,
>>>>> 1*v + 20*c + 30*a + 100*p AS score;
>>>>> gg = group score by (A, B, C);
>>>>>
>>>>> topResults = FOREACH gg {
>>>>> result = TOP(20, 4, score);
>>>>> GENERATE FLATTEN(result);
>>>>> };
>>>>> dump topResults;
>>>>>
>>>>> and i get this error
>>>>>
>>>>> 2011-05-05 13:41:18,665 [main] ERROR org.apache.pig.tools.grunt.Grunt
>> -
>>>>> ERROR 1070: Could not resolve TOP using imports: [,
>>>> org.apache.pig.builtin.,
>>>>> org.apache.pig.impl.builtin.]
>>>>>
>>>>> does anyone know where the TOP function is defined?
>>>>> (what do i need to register)
>>>>>
>>>>
>>>
>>
Re: TOP function isn't working
Posted by Shai Harel <sh...@mythings.com>.
i tried to replace
/home/hadoop/.versions/pig-0.6/lib/pig/pig-0.6-amzn.jar
with pig-0.8.1-core.jar
and i get the following error:
11/05/08 09:28:47 ERROR pig.Main: ERROR 2999: Unexpected internal error.
java.net.URISyntaxException: Illegal character in scheme name at index 19:
pig.initial.fs.name=hdfs:///
11/05/08 09:28:47 WARN pig.Main: There is no log file to write to.
11/05/08 09:28:47 ERROR pig.Main: java.lang.RuntimeException:
java.net.URISyntaxException: Illegal character in scheme name at index 19:
pig.initial.fs.name=hdfs:///
at
org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:742)
at org.apache.pig.Main.run(Main.java:299)
at org.apache.pig.Main.main(Main.java:107)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.net.URISyntaxException: Illegal character in scheme name at
index 19: pig.initial.fs.name=hdfs:///
at java.net.URI$Parser.fail(URI.java:2809)
at java.net.URI$Parser.checkChars(URI.java:2982)
at java.net.URI$Parser.parse(URI.java:3009)
at java.net.URI.<init>(URI.java:578)
at
org.apache.pig.impl.io.FileLocalizer.fetchFile(FileLocalizer.java:740)
... 7 more
On Thu, May 5, 2011 at 7:47 PM, Renato Marroquín Mogrovejo <
renatoj.marroquin@gmail.com> wrote:
> The only way to use built in TOP function would be if you manually
> install pig 0.8.1 because Amazon only supports pig 0.6 [1]
> So things in 7 and 8 are still out of Amazon EMR.
>
> [1] http://aws.amazon.com/elasticmapreduce/faqs/#pig-7
>
>
> Renato M.
>
> 2011/5/5 Gianmarco <gi...@gmail.com>:
> > You can use the builtin TOP() function.
> >
> > http://pig.apache.org/docs/r0.8.1/api/org/apache/pig/builtin/TOP.html
> >
> >
> > Cheers,
> > --
> > Gianmarco De Francisci Morales
> >
> >
> > On Thu, May 5, 2011 at 15:49, Shai Harel <sh...@mythings.com>
> wrote:
> >
> >> OK, i'v included piggybank
> >>
> >> register s3://my_path/piggybank.jar;
> >> DEFINE TOP org.apache.pig.piggybank.evaluation.util.Top();
> >>
> >> and now i get this error
> >>
> >> Pig Stack Trace
> >> ---------------
> >> ERROR 1045: Could not infer the matching function for
> >> org.apache.pig.piggybank.evaluation.util.Top as multiple or none of them
> >> fit. Please use an explicit cast.
> >>
> >> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable
> to
> >> store alias 78
> >> at
> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1004)
> >> at org.apache.pig.PigServer.registerQuery(PigServer.java:386)
> >> at
> >> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:739)
> >>
> >>
> >> On Thu, May 5, 2011 at 4:46 PM, Shai Harel <sh...@mythings.com>
> >> wrote:
> >>
> >> > i'm trying to run this code on Amazon Elastic MapReduce
> >> >
> >> > ...
> >> > score = FOREACH a GENERATE A, B, C,D,
> >> > 1*v + 20*c + 30*a + 100*p AS score;
> >> > gg = group score by (A, B, C);
> >> >
> >> > topResults = FOREACH gg {
> >> > result = TOP(20, 4, score);
> >> > GENERATE FLATTEN(result);
> >> > };
> >> > dump topResults;
> >> >
> >> > and i get this error
> >> >
> >> > 2011-05-05 13:41:18,665 [main] ERROR org.apache.pig.tools.grunt.Grunt
> -
> >> > ERROR 1070: Could not resolve TOP using imports: [,
> >> org.apache.pig.builtin.,
> >> > org.apache.pig.impl.builtin.]
> >> >
> >> > does anyone know where the TOP function is defined?
> >> > (what do i need to register)
> >> >
> >>
> >
>
Re: TOP function isn't working
Posted by Renato Marroquín Mogrovejo <re...@gmail.com>.
The only way to use built in TOP function would be if you manually
install pig 0.8.1 because Amazon only supports pig 0.6 [1]
So things in 7 and 8 are still out of Amazon EMR.
[1] http://aws.amazon.com/elasticmapreduce/faqs/#pig-7
Renato M.
2011/5/5 Gianmarco <gi...@gmail.com>:
> You can use the builtin TOP() function.
>
> http://pig.apache.org/docs/r0.8.1/api/org/apache/pig/builtin/TOP.html
>
>
> Cheers,
> --
> Gianmarco De Francisci Morales
>
>
> On Thu, May 5, 2011 at 15:49, Shai Harel <sh...@mythings.com> wrote:
>
>> OK, i'v included piggybank
>>
>> register s3://my_path/piggybank.jar;
>> DEFINE TOP org.apache.pig.piggybank.evaluation.util.Top();
>>
>> and now i get this error
>>
>> Pig Stack Trace
>> ---------------
>> ERROR 1045: Could not infer the matching function for
>> org.apache.pig.piggybank.evaluation.util.Top as multiple or none of them
>> fit. Please use an explicit cast.
>>
>> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to
>> store alias 78
>> at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1004)
>> at org.apache.pig.PigServer.registerQuery(PigServer.java:386)
>> at
>> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:739)
>>
>>
>> On Thu, May 5, 2011 at 4:46 PM, Shai Harel <sh...@mythings.com>
>> wrote:
>>
>> > i'm trying to run this code on Amazon Elastic MapReduce
>> >
>> > ...
>> > score = FOREACH a GENERATE A, B, C,D,
>> > 1*v + 20*c + 30*a + 100*p AS score;
>> > gg = group score by (A, B, C);
>> >
>> > topResults = FOREACH gg {
>> > result = TOP(20, 4, score);
>> > GENERATE FLATTEN(result);
>> > };
>> > dump topResults;
>> >
>> > and i get this error
>> >
>> > 2011-05-05 13:41:18,665 [main] ERROR org.apache.pig.tools.grunt.Grunt -
>> > ERROR 1070: Could not resolve TOP using imports: [,
>> org.apache.pig.builtin.,
>> > org.apache.pig.impl.builtin.]
>> >
>> > does anyone know where the TOP function is defined?
>> > (what do i need to register)
>> >
>>
>
Re: TOP function isn't working
Posted by Gianmarco <gi...@gmail.com>.
You can use the builtin TOP() function.
http://pig.apache.org/docs/r0.8.1/api/org/apache/pig/builtin/TOP.html
Cheers,
--
Gianmarco De Francisci Morales
On Thu, May 5, 2011 at 15:49, Shai Harel <sh...@mythings.com> wrote:
> OK, i'v included piggybank
>
> register s3://my_path/piggybank.jar;
> DEFINE TOP org.apache.pig.piggybank.evaluation.util.Top();
>
> and now i get this error
>
> Pig Stack Trace
> ---------------
> ERROR 1045: Could not infer the matching function for
> org.apache.pig.piggybank.evaluation.util.Top as multiple or none of them
> fit. Please use an explicit cast.
>
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to
> store alias 78
> at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1004)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:386)
> at
> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:739)
>
>
> On Thu, May 5, 2011 at 4:46 PM, Shai Harel <sh...@mythings.com>
> wrote:
>
> > i'm trying to run this code on Amazon Elastic MapReduce
> >
> > ...
> > score = FOREACH a GENERATE A, B, C,D,
> > 1*v + 20*c + 30*a + 100*p AS score;
> > gg = group score by (A, B, C);
> >
> > topResults = FOREACH gg {
> > result = TOP(20, 4, score);
> > GENERATE FLATTEN(result);
> > };
> > dump topResults;
> >
> > and i get this error
> >
> > 2011-05-05 13:41:18,665 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> > ERROR 1070: Could not resolve TOP using imports: [,
> org.apache.pig.builtin.,
> > org.apache.pig.impl.builtin.]
> >
> > does anyone know where the TOP function is defined?
> > (what do i need to register)
> >
>
Re: TOP function isn't working
Posted by Shai Harel <sh...@mythings.com>.
OK, i'v included piggybank
register s3://my_path/piggybank.jar;
DEFINE TOP org.apache.pig.piggybank.evaluation.util.Top();
and now i get this error
Pig Stack Trace
---------------
ERROR 1045: Could not infer the matching function for
org.apache.pig.piggybank.evaluation.util.Top as multiple or none of them
fit. Please use an explicit cast.
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to
store alias 78
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1004)
at org.apache.pig.PigServer.registerQuery(PigServer.java:386)
at
org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:739)
On Thu, May 5, 2011 at 4:46 PM, Shai Harel <sh...@mythings.com> wrote:
> i'm trying to run this code on Amazon Elastic MapReduce
>
> ...
> score = FOREACH a GENERATE A, B, C,D,
> 1*v + 20*c + 30*a + 100*p AS score;
> gg = group score by (A, B, C);
>
> topResults = FOREACH gg {
> result = TOP(20, 4, score);
> GENERATE FLATTEN(result);
> };
> dump topResults;
>
> and i get this error
>
> 2011-05-05 13:41:18,665 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> ERROR 1070: Could not resolve TOP using imports: [, org.apache.pig.builtin.,
> org.apache.pig.impl.builtin.]
>
> does anyone know where the TOP function is defined?
> (what do i need to register)
>