You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Andrew Clegg <an...@gmail.com> on 2011/10/31 20:20:56 UTC
Two-level access in Pig 0.8.1
Hi,
If I create two schemas like this:
Schema s1 = Utils.getSchemaFromString("b: {t: (f0: [])}");
Schema s2 = SchemaUtil.newBagSchema(new Byte[]{DataType.MAP});
then compare them like this:
boolean match = Schema.equals(expected, output, false, true);
they don't match, even though the toString output of the two is identical:
{b: {t: (f0: map[ ])}}
This turns out to be because in s1, t.setTwoLevelAccessRequired is
true, but in s2, it's false.
Which is correct? If I'm writing a UDF that returns a bag of maps,
what should I make the outputSchema method return?
Thanks,
Andrew.
--
http://tinyurl.com/andrew-clegg-linkedin | http://twitter.com/andrew_clegg
Re: Two-level access in Pig 0.8.1
Posted by Andrew Clegg <an...@gmail.com>.
Sorry, scratch that question, clearly it doesn't, because I'm using it
in 0.8.1 and it's not setting two-level access. Oh well.
On 1 November 2011 11:03, Andrew Clegg <an...@gmail.com> wrote:
> Would it be fair to assume that SchemaUtil.newBagSchema(...) etc. will
> do the right thing for whichever version you're using, without
> requiring additional manual intervention?
>
> On 31 October 2011 23:20, Dmitriy Ryaboy <dv...@gmail.com> wrote:
>> Our 0.8-compatible schemas for nested structures stopped working on
>> 0.9 (it wasn't just two-level access, it's also how deeply nested
>> structures are treated).
>> You can see what the problem is from EB's code that uses this method:
>>
>> https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/util/ThriftToPig.java
>>
>> (read singleFieldToTupleSchema method)
>>
>> D
>>
>>
>> On Mon, Oct 31, 2011 at 3:05 PM, Daniel Dai <da...@hortonworks.com> wrote:
>>> I guess this is what Dmitriy mean: since it is marked as deprecated, Pig
>>> could drop it anytime in the future. So you'd better make your UDF work
>>> even if twoLevelAccess is dropped.
>>>
>>> Daniel
>>>
>>> On Mon, Oct 31, 2011 at 2:48 PM, Andrew Clegg <andrew.clegg+mahout@gmail.com
>>>> wrote:
>>>
>>>> Thanks chaps. Just to clarify something though -- Daniel's answer
>>>> suggests twoLevelAccess is redundant but harmless in 0.9, but Dmitry's
>>>> advice suggests that it should actively be avoided. Unless I'm
>>>> misreading what one of you means?
>>>>
>>>> On 31 October 2011 20:34, Dmitriy Ryaboy <dv...@gmail.com> wrote:
>>>> > When implementing Daniel's suggestion, future-proof yourself:
>>>> >
>>>> >
>>>> https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/util/PigUtil.java
>>>> >
>>>> > (see the Pig9orNewer() method)
>>>> >
>>>> > D
>>>> >
>>>> > On Mon, Oct 31, 2011 at 12:59 PM, Daniel Dai <da...@hortonworks.com>
>>>> wrote:
>>>> >> Always set twoLevelAccess to true in 0.8. From 0.9, don't worry about it
>>>> >> any more.
>>>> >>
>>>> >> Daniel
>>>> >>
>>>> >> On Mon, Oct 31, 2011 at 12:20 PM, Andrew Clegg <
>>>> >> andrew.clegg+mahout@gmail.com> wrote:
>>>> >>
>>>> >>> Hi,
>>>> >>>
>>>> >>> If I create two schemas like this:
>>>> >>>
>>>> >>> Schema s1 = Utils.getSchemaFromString("b: {t: (f0: [])}");
>>>> >>> Schema s2 = SchemaUtil.newBagSchema(new Byte[]{DataType.MAP});
>>>> >>>
>>>> >>> then compare them like this:
>>>> >>>
>>>> >>> boolean match = Schema.equals(expected, output, false, true);
>>>> >>>
>>>> >>> they don't match, even though the toString output of the two is
>>>> identical:
>>>> >>>
>>>> >>> {b: {t: (f0: map[ ])}}
>>>> >>>
>>>> >>> This turns out to be because in s1, t.setTwoLevelAccessRequired is
>>>> >>> true, but in s2, it's false.
>>>> >>>
>>>> >>> Which is correct? If I'm writing a UDF that returns a bag of maps,
>>>> >>> what should I make the outputSchema method return?
>>>> >>>
>>>> >>> Thanks,
>>>> >>>
>>>> >>> Andrew.
>>>> >>>
>>>> >>> --
>>>> >>>
>>>> >>> http://tinyurl.com/andrew-clegg-linkedin |
>>>> http://twitter.com/andrew_clegg
>>>> >>>
>>>> >>
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> http://tinyurl.com/andrew-clegg-linkedin | http://twitter.com/andrew_clegg
>>>>
>>>
>>
>
>
>
> --
>
> http://tinyurl.com/andrew-clegg-linkedin | http://twitter.com/andrew_clegg
>
--
http://tinyurl.com/andrew-clegg-linkedin | http://twitter.com/andrew_clegg
Re: Two-level access in Pig 0.8.1
Posted by Andrew Clegg <an...@gmail.com>.
Would it be fair to assume that SchemaUtil.newBagSchema(...) etc. will
do the right thing for whichever version you're using, without
requiring additional manual intervention?
On 31 October 2011 23:20, Dmitriy Ryaboy <dv...@gmail.com> wrote:
> Our 0.8-compatible schemas for nested structures stopped working on
> 0.9 (it wasn't just two-level access, it's also how deeply nested
> structures are treated).
> You can see what the problem is from EB's code that uses this method:
>
> https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/util/ThriftToPig.java
>
> (read singleFieldToTupleSchema method)
>
> D
>
>
> On Mon, Oct 31, 2011 at 3:05 PM, Daniel Dai <da...@hortonworks.com> wrote:
>> I guess this is what Dmitriy mean: since it is marked as deprecated, Pig
>> could drop it anytime in the future. So you'd better make your UDF work
>> even if twoLevelAccess is dropped.
>>
>> Daniel
>>
>> On Mon, Oct 31, 2011 at 2:48 PM, Andrew Clegg <andrew.clegg+mahout@gmail.com
>>> wrote:
>>
>>> Thanks chaps. Just to clarify something though -- Daniel's answer
>>> suggests twoLevelAccess is redundant but harmless in 0.9, but Dmitry's
>>> advice suggests that it should actively be avoided. Unless I'm
>>> misreading what one of you means?
>>>
>>> On 31 October 2011 20:34, Dmitriy Ryaboy <dv...@gmail.com> wrote:
>>> > When implementing Daniel's suggestion, future-proof yourself:
>>> >
>>> >
>>> https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/util/PigUtil.java
>>> >
>>> > (see the Pig9orNewer() method)
>>> >
>>> > D
>>> >
>>> > On Mon, Oct 31, 2011 at 12:59 PM, Daniel Dai <da...@hortonworks.com>
>>> wrote:
>>> >> Always set twoLevelAccess to true in 0.8. From 0.9, don't worry about it
>>> >> any more.
>>> >>
>>> >> Daniel
>>> >>
>>> >> On Mon, Oct 31, 2011 at 12:20 PM, Andrew Clegg <
>>> >> andrew.clegg+mahout@gmail.com> wrote:
>>> >>
>>> >>> Hi,
>>> >>>
>>> >>> If I create two schemas like this:
>>> >>>
>>> >>> Schema s1 = Utils.getSchemaFromString("b: {t: (f0: [])}");
>>> >>> Schema s2 = SchemaUtil.newBagSchema(new Byte[]{DataType.MAP});
>>> >>>
>>> >>> then compare them like this:
>>> >>>
>>> >>> boolean match = Schema.equals(expected, output, false, true);
>>> >>>
>>> >>> they don't match, even though the toString output of the two is
>>> identical:
>>> >>>
>>> >>> {b: {t: (f0: map[ ])}}
>>> >>>
>>> >>> This turns out to be because in s1, t.setTwoLevelAccessRequired is
>>> >>> true, but in s2, it's false.
>>> >>>
>>> >>> Which is correct? If I'm writing a UDF that returns a bag of maps,
>>> >>> what should I make the outputSchema method return?
>>> >>>
>>> >>> Thanks,
>>> >>>
>>> >>> Andrew.
>>> >>>
>>> >>> --
>>> >>>
>>> >>> http://tinyurl.com/andrew-clegg-linkedin |
>>> http://twitter.com/andrew_clegg
>>> >>>
>>> >>
>>> >
>>>
>>>
>>>
>>> --
>>>
>>> http://tinyurl.com/andrew-clegg-linkedin | http://twitter.com/andrew_clegg
>>>
>>
>
--
http://tinyurl.com/andrew-clegg-linkedin | http://twitter.com/andrew_clegg
Re: Two-level access in Pig 0.8.1
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
Our 0.8-compatible schemas for nested structures stopped working on
0.9 (it wasn't just two-level access, it's also how deeply nested
structures are treated).
You can see what the problem is from EB's code that uses this method:
https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/util/ThriftToPig.java
(read singleFieldToTupleSchema method)
D
On Mon, Oct 31, 2011 at 3:05 PM, Daniel Dai <da...@hortonworks.com> wrote:
> I guess this is what Dmitriy mean: since it is marked as deprecated, Pig
> could drop it anytime in the future. So you'd better make your UDF work
> even if twoLevelAccess is dropped.
>
> Daniel
>
> On Mon, Oct 31, 2011 at 2:48 PM, Andrew Clegg <andrew.clegg+mahout@gmail.com
>> wrote:
>
>> Thanks chaps. Just to clarify something though -- Daniel's answer
>> suggests twoLevelAccess is redundant but harmless in 0.9, but Dmitry's
>> advice suggests that it should actively be avoided. Unless I'm
>> misreading what one of you means?
>>
>> On 31 October 2011 20:34, Dmitriy Ryaboy <dv...@gmail.com> wrote:
>> > When implementing Daniel's suggestion, future-proof yourself:
>> >
>> >
>> https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/util/PigUtil.java
>> >
>> > (see the Pig9orNewer() method)
>> >
>> > D
>> >
>> > On Mon, Oct 31, 2011 at 12:59 PM, Daniel Dai <da...@hortonworks.com>
>> wrote:
>> >> Always set twoLevelAccess to true in 0.8. From 0.9, don't worry about it
>> >> any more.
>> >>
>> >> Daniel
>> >>
>> >> On Mon, Oct 31, 2011 at 12:20 PM, Andrew Clegg <
>> >> andrew.clegg+mahout@gmail.com> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> If I create two schemas like this:
>> >>>
>> >>> Schema s1 = Utils.getSchemaFromString("b: {t: (f0: [])}");
>> >>> Schema s2 = SchemaUtil.newBagSchema(new Byte[]{DataType.MAP});
>> >>>
>> >>> then compare them like this:
>> >>>
>> >>> boolean match = Schema.equals(expected, output, false, true);
>> >>>
>> >>> they don't match, even though the toString output of the two is
>> identical:
>> >>>
>> >>> {b: {t: (f0: map[ ])}}
>> >>>
>> >>> This turns out to be because in s1, t.setTwoLevelAccessRequired is
>> >>> true, but in s2, it's false.
>> >>>
>> >>> Which is correct? If I'm writing a UDF that returns a bag of maps,
>> >>> what should I make the outputSchema method return?
>> >>>
>> >>> Thanks,
>> >>>
>> >>> Andrew.
>> >>>
>> >>> --
>> >>>
>> >>> http://tinyurl.com/andrew-clegg-linkedin |
>> http://twitter.com/andrew_clegg
>> >>>
>> >>
>> >
>>
>>
>>
>> --
>>
>> http://tinyurl.com/andrew-clegg-linkedin | http://twitter.com/andrew_clegg
>>
>
Re: Two-level access in Pig 0.8.1
Posted by Daniel Dai <da...@hortonworks.com>.
I guess this is what Dmitriy mean: since it is marked as deprecated, Pig
could drop it anytime in the future. So you'd better make your UDF work
even if twoLevelAccess is dropped.
Daniel
On Mon, Oct 31, 2011 at 2:48 PM, Andrew Clegg <andrew.clegg+mahout@gmail.com
> wrote:
> Thanks chaps. Just to clarify something though -- Daniel's answer
> suggests twoLevelAccess is redundant but harmless in 0.9, but Dmitry's
> advice suggests that it should actively be avoided. Unless I'm
> misreading what one of you means?
>
> On 31 October 2011 20:34, Dmitriy Ryaboy <dv...@gmail.com> wrote:
> > When implementing Daniel's suggestion, future-proof yourself:
> >
> >
> https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/util/PigUtil.java
> >
> > (see the Pig9orNewer() method)
> >
> > D
> >
> > On Mon, Oct 31, 2011 at 12:59 PM, Daniel Dai <da...@hortonworks.com>
> wrote:
> >> Always set twoLevelAccess to true in 0.8. From 0.9, don't worry about it
> >> any more.
> >>
> >> Daniel
> >>
> >> On Mon, Oct 31, 2011 at 12:20 PM, Andrew Clegg <
> >> andrew.clegg+mahout@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> If I create two schemas like this:
> >>>
> >>> Schema s1 = Utils.getSchemaFromString("b: {t: (f0: [])}");
> >>> Schema s2 = SchemaUtil.newBagSchema(new Byte[]{DataType.MAP});
> >>>
> >>> then compare them like this:
> >>>
> >>> boolean match = Schema.equals(expected, output, false, true);
> >>>
> >>> they don't match, even though the toString output of the two is
> identical:
> >>>
> >>> {b: {t: (f0: map[ ])}}
> >>>
> >>> This turns out to be because in s1, t.setTwoLevelAccessRequired is
> >>> true, but in s2, it's false.
> >>>
> >>> Which is correct? If I'm writing a UDF that returns a bag of maps,
> >>> what should I make the outputSchema method return?
> >>>
> >>> Thanks,
> >>>
> >>> Andrew.
> >>>
> >>> --
> >>>
> >>> http://tinyurl.com/andrew-clegg-linkedin |
> http://twitter.com/andrew_clegg
> >>>
> >>
> >
>
>
>
> --
>
> http://tinyurl.com/andrew-clegg-linkedin | http://twitter.com/andrew_clegg
>
Re: Two-level access in Pig 0.8.1
Posted by Andrew Clegg <an...@gmail.com>.
Thanks chaps. Just to clarify something though -- Daniel's answer
suggests twoLevelAccess is redundant but harmless in 0.9, but Dmitry's
advice suggests that it should actively be avoided. Unless I'm
misreading what one of you means?
On 31 October 2011 20:34, Dmitriy Ryaboy <dv...@gmail.com> wrote:
> When implementing Daniel's suggestion, future-proof yourself:
>
> https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/util/PigUtil.java
>
> (see the Pig9orNewer() method)
>
> D
>
> On Mon, Oct 31, 2011 at 12:59 PM, Daniel Dai <da...@hortonworks.com> wrote:
>> Always set twoLevelAccess to true in 0.8. From 0.9, don't worry about it
>> any more.
>>
>> Daniel
>>
>> On Mon, Oct 31, 2011 at 12:20 PM, Andrew Clegg <
>> andrew.clegg+mahout@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> If I create two schemas like this:
>>>
>>> Schema s1 = Utils.getSchemaFromString("b: {t: (f0: [])}");
>>> Schema s2 = SchemaUtil.newBagSchema(new Byte[]{DataType.MAP});
>>>
>>> then compare them like this:
>>>
>>> boolean match = Schema.equals(expected, output, false, true);
>>>
>>> they don't match, even though the toString output of the two is identical:
>>>
>>> {b: {t: (f0: map[ ])}}
>>>
>>> This turns out to be because in s1, t.setTwoLevelAccessRequired is
>>> true, but in s2, it's false.
>>>
>>> Which is correct? If I'm writing a UDF that returns a bag of maps,
>>> what should I make the outputSchema method return?
>>>
>>> Thanks,
>>>
>>> Andrew.
>>>
>>> --
>>>
>>> http://tinyurl.com/andrew-clegg-linkedin | http://twitter.com/andrew_clegg
>>>
>>
>
--
http://tinyurl.com/andrew-clegg-linkedin | http://twitter.com/andrew_clegg
Re: Two-level access in Pig 0.8.1
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
When implementing Daniel's suggestion, future-proof yourself:
https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/util/PigUtil.java
(see the Pig9orNewer() method)
D
On Mon, Oct 31, 2011 at 12:59 PM, Daniel Dai <da...@hortonworks.com> wrote:
> Always set twoLevelAccess to true in 0.8. From 0.9, don't worry about it
> any more.
>
> Daniel
>
> On Mon, Oct 31, 2011 at 12:20 PM, Andrew Clegg <
> andrew.clegg+mahout@gmail.com> wrote:
>
>> Hi,
>>
>> If I create two schemas like this:
>>
>> Schema s1 = Utils.getSchemaFromString("b: {t: (f0: [])}");
>> Schema s2 = SchemaUtil.newBagSchema(new Byte[]{DataType.MAP});
>>
>> then compare them like this:
>>
>> boolean match = Schema.equals(expected, output, false, true);
>>
>> they don't match, even though the toString output of the two is identical:
>>
>> {b: {t: (f0: map[ ])}}
>>
>> This turns out to be because in s1, t.setTwoLevelAccessRequired is
>> true, but in s2, it's false.
>>
>> Which is correct? If I'm writing a UDF that returns a bag of maps,
>> what should I make the outputSchema method return?
>>
>> Thanks,
>>
>> Andrew.
>>
>> --
>>
>> http://tinyurl.com/andrew-clegg-linkedin | http://twitter.com/andrew_clegg
>>
>
Re: Two-level access in Pig 0.8.1
Posted by Daniel Dai <da...@hortonworks.com>.
Always set twoLevelAccess to true in 0.8. From 0.9, don't worry about it
any more.
Daniel
On Mon, Oct 31, 2011 at 12:20 PM, Andrew Clegg <
andrew.clegg+mahout@gmail.com> wrote:
> Hi,
>
> If I create two schemas like this:
>
> Schema s1 = Utils.getSchemaFromString("b: {t: (f0: [])}");
> Schema s2 = SchemaUtil.newBagSchema(new Byte[]{DataType.MAP});
>
> then compare them like this:
>
> boolean match = Schema.equals(expected, output, false, true);
>
> they don't match, even though the toString output of the two is identical:
>
> {b: {t: (f0: map[ ])}}
>
> This turns out to be because in s1, t.setTwoLevelAccessRequired is
> true, but in s2, it's false.
>
> Which is correct? If I'm writing a UDF that returns a bag of maps,
> what should I make the outputSchema method return?
>
> Thanks,
>
> Andrew.
>
> --
>
> http://tinyurl.com/andrew-clegg-linkedin | http://twitter.com/andrew_clegg
>