You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Jeremy Hanna <je...@gmail.com> on 2011/03/24 19:11:49 UTC
LoadCaster, LoadStoreCaster usage and encoded output
I see that there are a few LoadCaster implementations in pig 0.8. There's the Utf8StorageConverter, the HBaseBinaryConverter, and a couple of others.
The HBaseStorage class uses the Utf8StorageConverter by default but can be configured to use the HBaseBinaryConverter. Also it's just used as a LoadCaster and I don't see where it uses a StoreCaster at all - like the LoadFunc interface has a getLoadCaster method to override, but I can't find anything that has a getStoreCaster or getLoadStoreCaster method to override.
Anyway, so I'm using the Cassandra loadfunc and getting LongType data returned with some special characters and I thought it might be because I'm not using a LoadCaster to convert to Pig types. So I tried both the UtfStorageConverter as well as created my own CassandraBinaryConverter (implementing LoadStoreCaster) to convert from Cassandra types to and from Pig basic types. Neither work though and I'm still getting the special character stuff when I dump to the console.
Any ideas on why LongTypes would be returning something like this: � as a value in a tuple? It's showing up just as a normal Long value on the cassandra cli. Oh, and I'm loading it with:
rows = load 'cassandra://MyKeyspace/MyColumnFamily' using CassandraStorage() as (key, columns: bag{T: tuple(name, value)});
A = limit rows 10;
dump A;
The value is the thing that is coming out seemingly encoded.
Thanks,
Jeremy
Re: LoadCaster, LoadStoreCaster usage and encoded output
Posted by jacob <ja...@gmail.com>.
We're still using a fork unfortunately. Jeremy is referencing the on in
trunk as far as I know though. Here we're waiting for when we switch
from our weird version of hbase (0.89somethingsomething) to 0.90 to make
the switch.
--jacob
On Thu, 2011-03-24 at 15:10 -0700, Dmitriy Ryaboy wrote:
> That's a good point about HBaseStorage not using the caster. I don't use it
> in prod so forgot to put it in.
> Jacob, are you guys using a fork or are you back on the official loader
> version?
>
> On Thu, Mar 24, 2011 at 12:03 PM, Jeremy Hanna
> <je...@gmail.com>wrote:
>
> > Hmmm, that never calls the bytesToLong method even with that specified in
> > the schema. I wonder if it's that when using a Cassandra validator on a
> > column, Cassandra tries its best to make the best guess about the value's
> > type which may not be compatible with the pig basic types (in this case
> > Cassandra's LongType). So it's never getting to the bytesToX methods of the
> > load caster. Talking with Brandon Williams a little, I may need to take it
> > to a higher level and do appropriate casting.
> >
> > On Mar 24, 2011, at 1:42 PM, jacob wrote:
> >
> > > Hmm. I bet I know what the issue is. It's not fun though. I'm thinking
> > > that loadcaster probably isn't even called unless you explicitly name
> > > the types at in the schema declaration.
> > >
> > > Try loading with:
> > >
> > > rows = load 'cassandra://MyKeyspace/MyColumnFamily' using
> > > CassandraStorage() as (key:chararray, columns: bag{T:
> > > tuple(name:chararray, value:long)});
> > >
> > > To see if it correctly treats the longtype values. This isn't good
> > > though since obviously not all of the values are longs. However, if it
> > > does work for the longtypes we know we're on the right track.
> > >
> > > We may have to go in and explicitly check the types of each column and
> > > cast manually.
> > >
> > > --jacob
> > >
> > > On Thu, 2011-03-24 at 13:11 -0500, Jeremy Hanna wrote:
> > >> I see that there are a few LoadCaster implementations in pig 0.8.
> > There's the Utf8StorageConverter, the HBaseBinaryConverter, and a couple of
> > others.
> > >>
> > >> The HBaseStorage class uses the Utf8StorageConverter by default but can
> > be configured to use the HBaseBinaryConverter. Also it's just used as a
> > LoadCaster and I don't see where it uses a StoreCaster at all - like the
> > LoadFunc interface has a getLoadCaster method to override, but I can't find
> > anything that has a getStoreCaster or getLoadStoreCaster method to override.
> > >>
> > >> Anyway, so I'm using the Cassandra loadfunc and getting LongType data
> > returned with some special characters and I thought it might be because I'm
> > not using a LoadCaster to convert to Pig types. So I tried both the
> > UtfStorageConverter as well as created my own CassandraBinaryConverter
> > (implementing LoadStoreCaster) to convert from Cassandra types to and from
> > Pig basic types. Neither work though and I'm still getting the special
> > character stuff when I dump to the console.
> > >>
> > >> Any ideas on why LongTypes would be returning something like this: � as
> > a value in a tuple? It's showing up just as a normal Long value on the
> > cassandra cli. Oh, and I'm loading it with:
> > >
> > >> rows = load 'cassandra://MyKeyspace/MyColumnFamily' using
> > CassandraStorage() as (key, columns: bag{T: tuple(name, value)});
> > >> A = limit rows 10;
> > >> dump A;
> > >>
> > >> The value is the thing that is coming out seemingly encoded.
> > >>
> > >> Thanks,
> > >>
> > >> Jeremy
> > >
> > >
> >
> >
Re: LoadCaster, LoadStoreCaster usage and encoded output
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
That's a good point about HBaseStorage not using the caster. I don't use it
in prod so forgot to put it in.
Jacob, are you guys using a fork or are you back on the official loader
version?
On Thu, Mar 24, 2011 at 12:03 PM, Jeremy Hanna
<je...@gmail.com>wrote:
> Hmmm, that never calls the bytesToLong method even with that specified in
> the schema. I wonder if it's that when using a Cassandra validator on a
> column, Cassandra tries its best to make the best guess about the value's
> type which may not be compatible with the pig basic types (in this case
> Cassandra's LongType). So it's never getting to the bytesToX methods of the
> load caster. Talking with Brandon Williams a little, I may need to take it
> to a higher level and do appropriate casting.
>
> On Mar 24, 2011, at 1:42 PM, jacob wrote:
>
> > Hmm. I bet I know what the issue is. It's not fun though. I'm thinking
> > that loadcaster probably isn't even called unless you explicitly name
> > the types at in the schema declaration.
> >
> > Try loading with:
> >
> > rows = load 'cassandra://MyKeyspace/MyColumnFamily' using
> > CassandraStorage() as (key:chararray, columns: bag{T:
> > tuple(name:chararray, value:long)});
> >
> > To see if it correctly treats the longtype values. This isn't good
> > though since obviously not all of the values are longs. However, if it
> > does work for the longtypes we know we're on the right track.
> >
> > We may have to go in and explicitly check the types of each column and
> > cast manually.
> >
> > --jacob
> >
> > On Thu, 2011-03-24 at 13:11 -0500, Jeremy Hanna wrote:
> >> I see that there are a few LoadCaster implementations in pig 0.8.
> There's the Utf8StorageConverter, the HBaseBinaryConverter, and a couple of
> others.
> >>
> >> The HBaseStorage class uses the Utf8StorageConverter by default but can
> be configured to use the HBaseBinaryConverter. Also it's just used as a
> LoadCaster and I don't see where it uses a StoreCaster at all - like the
> LoadFunc interface has a getLoadCaster method to override, but I can't find
> anything that has a getStoreCaster or getLoadStoreCaster method to override.
> >>
> >> Anyway, so I'm using the Cassandra loadfunc and getting LongType data
> returned with some special characters and I thought it might be because I'm
> not using a LoadCaster to convert to Pig types. So I tried both the
> UtfStorageConverter as well as created my own CassandraBinaryConverter
> (implementing LoadStoreCaster) to convert from Cassandra types to and from
> Pig basic types. Neither work though and I'm still getting the special
> character stuff when I dump to the console.
> >>
> >> Any ideas on why LongTypes would be returning something like this: � as
> a value in a tuple? It's showing up just as a normal Long value on the
> cassandra cli. Oh, and I'm loading it with:
> >
> >> rows = load 'cassandra://MyKeyspace/MyColumnFamily' using
> CassandraStorage() as (key, columns: bag{T: tuple(name, value)});
> >> A = limit rows 10;
> >> dump A;
> >>
> >> The value is the thing that is coming out seemingly encoded.
> >>
> >> Thanks,
> >>
> >> Jeremy
> >
> >
>
>
Re: LoadCaster, LoadStoreCaster usage and encoded output
Posted by Jeremy Hanna <je...@gmail.com>.
Hmmm, that never calls the bytesToLong method even with that specified in the schema. I wonder if it's that when using a Cassandra validator on a column, Cassandra tries its best to make the best guess about the value's type which may not be compatible with the pig basic types (in this case Cassandra's LongType). So it's never getting to the bytesToX methods of the load caster. Talking with Brandon Williams a little, I may need to take it to a higher level and do appropriate casting.
On Mar 24, 2011, at 1:42 PM, jacob wrote:
> Hmm. I bet I know what the issue is. It's not fun though. I'm thinking
> that loadcaster probably isn't even called unless you explicitly name
> the types at in the schema declaration.
>
> Try loading with:
>
> rows = load 'cassandra://MyKeyspace/MyColumnFamily' using
> CassandraStorage() as (key:chararray, columns: bag{T:
> tuple(name:chararray, value:long)});
>
> To see if it correctly treats the longtype values. This isn't good
> though since obviously not all of the values are longs. However, if it
> does work for the longtypes we know we're on the right track.
>
> We may have to go in and explicitly check the types of each column and
> cast manually.
>
> --jacob
>
> On Thu, 2011-03-24 at 13:11 -0500, Jeremy Hanna wrote:
>> I see that there are a few LoadCaster implementations in pig 0.8. There's the Utf8StorageConverter, the HBaseBinaryConverter, and a couple of others.
>>
>> The HBaseStorage class uses the Utf8StorageConverter by default but can be configured to use the HBaseBinaryConverter. Also it's just used as a LoadCaster and I don't see where it uses a StoreCaster at all - like the LoadFunc interface has a getLoadCaster method to override, but I can't find anything that has a getStoreCaster or getLoadStoreCaster method to override.
>>
>> Anyway, so I'm using the Cassandra loadfunc and getting LongType data returned with some special characters and I thought it might be because I'm not using a LoadCaster to convert to Pig types. So I tried both the UtfStorageConverter as well as created my own CassandraBinaryConverter (implementing LoadStoreCaster) to convert from Cassandra types to and from Pig basic types. Neither work though and I'm still getting the special character stuff when I dump to the console.
>>
>> Any ideas on why LongTypes would be returning something like this: � as a value in a tuple? It's showing up just as a normal Long value on the cassandra cli. Oh, and I'm loading it with:
>
>> rows = load 'cassandra://MyKeyspace/MyColumnFamily' using CassandraStorage() as (key, columns: bag{T: tuple(name, value)});
>> A = limit rows 10;
>> dump A;
>>
>> The value is the thing that is coming out seemingly encoded.
>>
>> Thanks,
>>
>> Jeremy
>
>
Re: LoadCaster, LoadStoreCaster usage and encoded output
Posted by jacob <ja...@gmail.com>.
Hmm. I bet I know what the issue is. It's not fun though. I'm thinking
that loadcaster probably isn't even called unless you explicitly name
the types at in the schema declaration.
Try loading with:
rows = load 'cassandra://MyKeyspace/MyColumnFamily' using
CassandraStorage() as (key:chararray, columns: bag{T:
tuple(name:chararray, value:long)});
To see if it correctly treats the longtype values. This isn't good
though since obviously not all of the values are longs. However, if it
does work for the longtypes we know we're on the right track.
We may have to go in and explicitly check the types of each column and
cast manually.
--jacob
On Thu, 2011-03-24 at 13:11 -0500, Jeremy Hanna wrote:
> I see that there are a few LoadCaster implementations in pig 0.8. There's the Utf8StorageConverter, the HBaseBinaryConverter, and a couple of others.
>
> The HBaseStorage class uses the Utf8StorageConverter by default but can be configured to use the HBaseBinaryConverter. Also it's just used as a LoadCaster and I don't see where it uses a StoreCaster at all - like the LoadFunc interface has a getLoadCaster method to override, but I can't find anything that has a getStoreCaster or getLoadStoreCaster method to override.
>
> Anyway, so I'm using the Cassandra loadfunc and getting LongType data returned with some special characters and I thought it might be because I'm not using a LoadCaster to convert to Pig types. So I tried both the UtfStorageConverter as well as created my own CassandraBinaryConverter (implementing LoadStoreCaster) to convert from Cassandra types to and from Pig basic types. Neither work though and I'm still getting the special character stuff when I dump to the console.
>
> Any ideas on why LongTypes would be returning something like this: � as a value in a tuple? It's showing up just as a normal Long value on the cassandra cli. Oh, and I'm loading it with:
> rows = load 'cassandra://MyKeyspace/MyColumnFamily' using CassandraStorage() as (key, columns: bag{T: tuple(name, value)});
> A = limit rows 10;
> dump A;
>
> The value is the thing that is coming out seemingly encoded.
>
> Thanks,
>
> Jeremy