You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Johannes Schwenk <jo...@adition.com> on 2012/06/20 13:38:56 UTC
Pig 0.10.0 and Hadoop 0.23
Hi list,
I have written two UDFs that run fine with Hadoop 0.20.2 and Pig 0.10.0.
I am trying to switch from Hadoop 0.20.2 to 0.23 and am encountering
some errors on running my tests:
http://pastebin.com/JwHRQ9RQ
Might this be a simple error in configuration?
Johannes Schwenk
--
Softwareentwickler (Reporting)
________________________________________________________
ADITION technologies AG
Schwarzwaldstraße 78b
79117 Freiburg
http://www.adition.com
T +49 / (0)761 / 88147 - 30
F +49 / (0)761 / 88147 - 77
SUPPORT +49 / (0)1805 - ADITION
(Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min)
Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076
Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus Schlüter
Aufsichtsratsvorsitzender: Rechtsanwalt Daniel Raimer
UStIDNr.: DE 218 858 434
Re: Pig 0.10.0 and Hadoop 0.23
Posted by Johannes Schwenk <jo...@adition.com>.
Thanks.
Well, I did that and rebuild all pig jars with -Dhadoopversion=23 but am
still getting errors - this time around it's errors with opening an
iterator on alias b again, sigh...
I have singled out one of the similarly failing tests and pasted the output:
http://pastebin.com/reKjcX0p
The test case failing looks like this:
http://pastebin.com/L5seh0ka
And INPUT_FILE gets created correctly through @BeforeClass
http://pastebin.com/wKrH7i50
If I run the script in the grunt shell invoked like this
pig -x local -propertyFile
/home/schwenk/dev/hadoop/trunk/pig/conf/adition.properties
everything works fine and I can execute the script
grunt> a = LOAD 'input.dat' AS (id:int, grp:int, additional:int,
referer:chararray);
b = FILTER a BY com.adition.pig.filtering.string.CONTAINS(referer, 'obama');
grunt> dump b;
...
2012-07-03 18:11:23,032 [main] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total
input paths to process : 1
(4,323,242,http://www.google.com/url&url=http%3A%2F%2Fwww.tagesschau.de&q=barack%20obama)
(5,423,342,http://www.google.com/url&url=http%3A%2F%2Fwww.bild.de&q=obama)
and verify that the output is correct.
Any further advice on this?
Thanks,
Johannes
Am 27.06.2012 22:24, schrieb Daniel Dai:
> Do "ant clean".
>
> On Thu, Jun 21, 2012 at 1:16 AM, Johannes Schwenk <
> johannes.schwenk@adition.com> wrote:
>
>> Am 20.06.2012 20:21, schrieb Daniel Dai:
>>> Have you recompiled Pig using -Dhadoopversion=23?
>>
>> No, thanks that solved the issue at hand!
>>
>> I have other errors though. My test cases return errors like
>>
>> http://pastebin.com/XpszYeu2
>>
>> So I am still doing something wrong it seems?!
>>
>> Thanks,
>> Johannes
>>
>>
>>> On Wed, Jun 20, 2012 at 4:38 AM, Johannes Schwenk <
>>> johannes.schwenk@adition.com> wrote:
>>>
>>>> Hi list,
>>>>
>>>> I have written two UDFs that run fine with Hadoop 0.20.2 and Pig 0.10.0.
>>>> I am trying to switch from Hadoop 0.20.2 to 0.23 and am encountering
>>>> some errors on running my tests:
>>>>
>>>> http://pastebin.com/JwHRQ9RQ
>>>>
>>>> Might this be a simple error in configuration?
>>>>
>>>> Johannes Schwenk
>>>>
>>>> --
>>>> Softwareentwickler (Reporting)
>>>> ________________________________________________________
>>>>
>>>> ADITION technologies AG
>>>> Schwarzwaldstraße 78b
>>>> 79117 Freiburg
>>>>
>>>> http://www.adition.com
>>>>
>>>> T +49 / (0)761 / 88147 - 30
>>>> F +49 / (0)761 / 88147 - 77
>>>> SUPPORT +49 / (0)1805 - ADITION
>>>>
>>>> (Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min)
>>>>
>>>> Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076
>>>> Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus
>> Schlüter
>>>> Aufsichtsratsvorsitzender: Rechtsanwalt Daniel Raimer
>>>> UStIDNr.: DE 218 858 434
>>>>
>>>>
>>>
>>
>>
>>
>> Johannes Schwenk
>>
>> --
>> Softwareentwickler (Reporting)
>> ________________________________________________________
>>
>> ADITION technologies AG
>> Schwarzwaldstraße 78b
>> 79117 Freiburg
>>
>> http://www.adition.com
>>
>> T +49 / (0)761 / 88147 - 30
>> F +49 / (0)761 / 88147 - 77
>> SUPPORT +49 / (0)1805 - ADITION
>>
>> (Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min)
>>
>> Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076
>> Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus Schlüter
>> Aufsichtsratsvorsitzender: Rechtsanwalt Daniel Raimer
>> UStIDNr.: DE 218 858 434
>>
>>
>
Johannes Schwenk
--
Softwareentwickler (Reporting)
________________________________________________________
ADITION technologies AG
Schwarzwaldstraße 78b
79117 Freiburg
http://www.adition.com
T +49 / (0)761 / 88147 - 30
F +49 / (0)761 / 88147 - 77
SUPPORT +49 / (0)1805 - ADITION
(Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min)
Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076
Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus Schlüter
Aufsichtsratsvorsitzender: Rechtsanwalt Daniel Raimer
UStIDNr.: DE 218 858 434
Re: Pig 0.10.0 and Hadoop 0.23
Posted by Daniel Dai <da...@hortonworks.com>.
Do "ant clean".
On Thu, Jun 21, 2012 at 1:16 AM, Johannes Schwenk <
johannes.schwenk@adition.com> wrote:
> Am 20.06.2012 20:21, schrieb Daniel Dai:
> > Have you recompiled Pig using -Dhadoopversion=23?
>
> No, thanks that solved the issue at hand!
>
> I have other errors though. My test cases return errors like
>
> http://pastebin.com/XpszYeu2
>
> So I am still doing something wrong it seems?!
>
> Thanks,
> Johannes
>
>
> > On Wed, Jun 20, 2012 at 4:38 AM, Johannes Schwenk <
> > johannes.schwenk@adition.com> wrote:
> >
> >> Hi list,
> >>
> >> I have written two UDFs that run fine with Hadoop 0.20.2 and Pig 0.10.0.
> >> I am trying to switch from Hadoop 0.20.2 to 0.23 and am encountering
> >> some errors on running my tests:
> >>
> >> http://pastebin.com/JwHRQ9RQ
> >>
> >> Might this be a simple error in configuration?
> >>
> >> Johannes Schwenk
> >>
> >> --
> >> Softwareentwickler (Reporting)
> >> ________________________________________________________
> >>
> >> ADITION technologies AG
> >> Schwarzwaldstraße 78b
> >> 79117 Freiburg
> >>
> >> http://www.adition.com
> >>
> >> T +49 / (0)761 / 88147 - 30
> >> F +49 / (0)761 / 88147 - 77
> >> SUPPORT +49 / (0)1805 - ADITION
> >>
> >> (Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min)
> >>
> >> Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076
> >> Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus
> Schlüter
> >> Aufsichtsratsvorsitzender: Rechtsanwalt Daniel Raimer
> >> UStIDNr.: DE 218 858 434
> >>
> >>
> >
>
>
>
> Johannes Schwenk
>
> --
> Softwareentwickler (Reporting)
> ________________________________________________________
>
> ADITION technologies AG
> Schwarzwaldstraße 78b
> 79117 Freiburg
>
> http://www.adition.com
>
> T +49 / (0)761 / 88147 - 30
> F +49 / (0)761 / 88147 - 77
> SUPPORT +49 / (0)1805 - ADITION
>
> (Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min)
>
> Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076
> Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus Schlüter
> Aufsichtsratsvorsitzender: Rechtsanwalt Daniel Raimer
> UStIDNr.: DE 218 858 434
>
>
Re: Pig 0.10.0 and Hadoop 0.23
Posted by Johannes Schwenk <jo...@adition.com>.
Am 20.06.2012 20:21, schrieb Daniel Dai:
> Have you recompiled Pig using -Dhadoopversion=23?
No, thanks that solved the issue at hand!
I have other errors though. My test cases return errors like
http://pastebin.com/XpszYeu2
So I am still doing something wrong it seems?!
Thanks,
Johannes
> On Wed, Jun 20, 2012 at 4:38 AM, Johannes Schwenk <
> johannes.schwenk@adition.com> wrote:
>
>> Hi list,
>>
>> I have written two UDFs that run fine with Hadoop 0.20.2 and Pig 0.10.0.
>> I am trying to switch from Hadoop 0.20.2 to 0.23 and am encountering
>> some errors on running my tests:
>>
>> http://pastebin.com/JwHRQ9RQ
>>
>> Might this be a simple error in configuration?
>>
>> Johannes Schwenk
>>
>> --
>> Softwareentwickler (Reporting)
>> ________________________________________________________
>>
>> ADITION technologies AG
>> Schwarzwaldstraße 78b
>> 79117 Freiburg
>>
>> http://www.adition.com
>>
>> T +49 / (0)761 / 88147 - 30
>> F +49 / (0)761 / 88147 - 77
>> SUPPORT +49 / (0)1805 - ADITION
>>
>> (Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min)
>>
>> Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076
>> Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus Schlüter
>> Aufsichtsratsvorsitzender: Rechtsanwalt Daniel Raimer
>> UStIDNr.: DE 218 858 434
>>
>>
>
Johannes Schwenk
--
Softwareentwickler (Reporting)
________________________________________________________
ADITION technologies AG
Schwarzwaldstraße 78b
79117 Freiburg
http://www.adition.com
T +49 / (0)761 / 88147 - 30
F +49 / (0)761 / 88147 - 77
SUPPORT +49 / (0)1805 - ADITION
(Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min)
Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076
Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus Schlüter
Aufsichtsratsvorsitzender: Rechtsanwalt Daniel Raimer
UStIDNr.: DE 218 858 434
Re: Pig 0.10.0 and Hadoop 0.23
Posted by Daniel Dai <da...@hortonworks.com>.
Have you recompiled Pig using -Dhadoopversion=23?
On Wed, Jun 20, 2012 at 4:38 AM, Johannes Schwenk <
johannes.schwenk@adition.com> wrote:
> Hi list,
>
> I have written two UDFs that run fine with Hadoop 0.20.2 and Pig 0.10.0.
> I am trying to switch from Hadoop 0.20.2 to 0.23 and am encountering
> some errors on running my tests:
>
> http://pastebin.com/JwHRQ9RQ
>
> Might this be a simple error in configuration?
>
> Johannes Schwenk
>
> --
> Softwareentwickler (Reporting)
> ________________________________________________________
>
> ADITION technologies AG
> Schwarzwaldstraße 78b
> 79117 Freiburg
>
> http://www.adition.com
>
> T +49 / (0)761 / 88147 - 30
> F +49 / (0)761 / 88147 - 77
> SUPPORT +49 / (0)1805 - ADITION
>
> (Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min)
>
> Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076
> Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus Schlüter
> Aufsichtsratsvorsitzender: Rechtsanwalt Daniel Raimer
> UStIDNr.: DE 218 858 434
>
>