You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Mck <mc...@apache.org> on 2011/05/19 09:30:11 UTC
[hadoop] Counters in ColumnFamilyOutputFormat?
I'd like to investigate using Counters in hadoop using
ColumnFamilyOutputFormat.
But i see that this class uses outdated ..hadoop.arvo classes
Does it make sense to use counters for hadoop output?
If i try rewriting ColumnFamilyOutputFormat and friends should it be to
the normal ..avro classes, or to something else?
~mck
Re: [hadoop] Counters in ColumnFamilyOutputFormat?
Posted by Jeremy Hanna <je...@gmail.com>.
FWIW, as I mentioned in the 1497 comments, the patch makes it abstract so that you can have any rpc/marshalling format you want with a simple extension point. So if we want to move to something besides avro, or even like I mentioned do something with Dumbo for streaming, it's easy to extend.
On May 19, 2011, at 9:23 AM, Jonathan Ellis wrote:
> Avro is there because (1) a long time ago (it now seems) we thought we
> were going to move the main RPC layer to Avro and (2) it simplifies
> using Avro for Streaming, which also seems misguided now
> (https://issues.apache.org/jira/browse/CASSANDRA-1497).
>
> Using "native" Thrift mutations makes the most sense to me now, which
> would keep it similar in structure but avoid the avroToThrift copy.
>
> On Thu, May 19, 2011 at 2:30 AM, Mck <mc...@apache.org> wrote:
>> I'd like to investigate using Counters in hadoop using
>> ColumnFamilyOutputFormat.
>>
>> But i see that this class uses outdated ..hadoop.arvo classes
>>
>> Does it make sense to use counters for hadoop output?
>>
>> If i try rewriting ColumnFamilyOutputFormat and friends should it be to
>> the normal ..avro classes, or to something else?
>>
>> ~mck
>>
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
Re: [hadoop] Counters in ColumnFamilyOutputFormat?
Posted by Mick Semb Wever <mc...@apache.org>.
On Thu, 2011-05-19 at 09:23 -0500, Jonathan Ellis wrote:
>
> Using "native" Thrift mutations makes the most sense to me now, which
> would keep it similar in structure but avoid the avroToThrift copy.
https://issues.apache.org/jira/browse/CASSANDRA-2667
--
"Everything you can imagine is real." Pablo Picasso
| http://semb.wever.org | http://sesat.no
| http://tech.finn.no | Java XSS Filter
Re: [hadoop] Counters in ColumnFamilyOutputFormat?
Posted by Jonathan Ellis <jb...@gmail.com>.
Avro is there because (1) a long time ago (it now seems) we thought we
were going to move the main RPC layer to Avro and (2) it simplifies
using Avro for Streaming, which also seems misguided now
(https://issues.apache.org/jira/browse/CASSANDRA-1497).
Using "native" Thrift mutations makes the most sense to me now, which
would keep it similar in structure but avoid the avroToThrift copy.
On Thu, May 19, 2011 at 2:30 AM, Mck <mc...@apache.org> wrote:
> I'd like to investigate using Counters in hadoop using
> ColumnFamilyOutputFormat.
>
> But i see that this class uses outdated ..hadoop.arvo classes
>
> Does it make sense to use counters for hadoop output?
>
> If i try rewriting ColumnFamilyOutputFormat and friends should it be to
> the normal ..avro classes, or to something else?
>
> ~mck
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com