You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Trung Tran <tr...@gmail.com> on 2011/01/18 09:40:10 UTC

Cassandra/Hadoop only write few columns

Hi,

I'm working on ColumnFamilyOutputFormat and for some reasons my reduce
class does not write all columns to cassandra. I tried to modify
mapreduce.output.columnfamilyoutputformat.batch.threshold with some
different values (1, 8, .. etc) but no thing changes.

What i'm having in my reduce class is :

ArrayList<Mutation> a = new ArrayList<Mutation>();

a.add(getMutation(colNam1, val1));
a.add(getMutation(colNam2, val2));
a.add(getMutation(colNam2, val2)); ...etc

context.write(key,a);

Only 2 columns are written in to cassandra, and no error log is found
on both hadoop and cassandra log. Any help is appreciated.

Thanks,
Trung.

Re: Cassandra/Hadoop only write few columns

Posted by Trung Tran <tr...@gmail.com>.
It was a typo in my example code in this email. I logged the list to
make sure that everything was correct before trigger the write.

Will try to enable debug on both cassandra and hadoop next.

Thanks,
Trung.

On Tue, Jan 18, 2011 at 1:21 AM, Aaron Morton <aa...@thelastpickle.com> wrote:
> May just be your example code, but you are repeating colName2 . Can you log the mutation list before you write it and confirm you have unique column names?
>
> Can you turn up the logging to DEBUG for the hadoop job and the Cassandra cluster to see what's happening?
>
> Aaron
>
> On 18/01/2011, at 9:40 PM, Trung Tran <tr...@gmail.com> wrote:
>
>> Hi,
>>
>> I'm working on ColumnFamilyOutputFormat and for some reasons my reduce
>> class does not write all columns to cassandra. I tried to modify
>> mapreduce.output.columnfamilyoutputformat.batch.threshold with some
>> different values (1, 8, .. etc) but no thing changes.
>>
>> What i'm having in my reduce class is :
>>
>> ArrayList<Mutation> a = new ArrayList<Mutation>();
>>
>> a.add(getMutation(colNam1, val1));
>> a.add(getMutation(colNam2, val2));
>> a.add(getMutation(colNam2, val2)); ...etc
>>
>> context.write(key,a);
>>
>> Only 2 columns are written in to cassandra, and no error log is found
>> on both hadoop and cassandra log. Any help is appreciated.
>>
>> Thanks,
>> Trung.
>

Re: Cassandra/Hadoop only write few columns

Posted by Aaron Morton <aa...@thelastpickle.com>.
May just be your example code, but you are repeating colName2 . Can you log the mutation list before you write it and confirm you have unique column names?

Can you turn up the logging to DEBUG for the hadoop job and the Cassandra cluster to see what's happening?

Aaron

On 18/01/2011, at 9:40 PM, Trung Tran <tr...@gmail.com> wrote:

> Hi,
> 
> I'm working on ColumnFamilyOutputFormat and for some reasons my reduce
> class does not write all columns to cassandra. I tried to modify
> mapreduce.output.columnfamilyoutputformat.batch.threshold with some
> different values (1, 8, .. etc) but no thing changes.
> 
> What i'm having in my reduce class is :
> 
> ArrayList<Mutation> a = new ArrayList<Mutation>();
> 
> a.add(getMutation(colNam1, val1));
> a.add(getMutation(colNam2, val2));
> a.add(getMutation(colNam2, val2)); ...etc
> 
> context.write(key,a);
> 
> Only 2 columns are written in to cassandra, and no error log is found
> on both hadoop and cassandra log. Any help is appreciated.
> 
> Thanks,
> Trung.