You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2014/03/11 18:51:45 UTC
[jira] [Commented] (CASSANDRA-6793) NPE in Hadoop Word count
example
[ https://issues.apache.org/jira/browse/CASSANDRA-6793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13930645#comment-13930645 ]
Jonathan Ellis commented on CASSANDRA-6793:
-------------------------------------------
I confess that I'm mystified by the schema introduced in CASSANDRA-4421:
{noformat}
/**
* This counts the occurrences of words in ColumnFamily
* cql3_worldcount ( user_id text,
* category_id text,
* sub_category_id text,
* title text,
* body text,
* PRIMARY KEY (user_id, category_id, sub_category_id))
*
* For each word, we output the total number of occurrences across all body texts.
*
* When outputting to Cassandra, we write the word counts to column family
* output_words ( row_id1 text,
* row_id2 text,
* word text,
* count_num text,
* PRIMARY KEY ((row_id1, row_id2), word))
* as a {word, count} to columns: word, count_num with a row key of "word sum"
*/
/**
* This counts the occurrences of words in ColumnFamily
* cql3_worldcount ( user_id text,
* category_id text,
* sub_category_id text,
* title text,
* body text,
* PRIMARY KEY (user_id, category_id, sub_category_id))
*
* For each word, we output the total number of occurrences across all body texts.
*
* When outputting to Cassandra, we write the word counts to column family
* output_words ( row_id1 text,
* row_id2 text,
* word text,
* count_num text,
* PRIMARY KEY ((row_id1, row_id2), word))
* as a {word, count} to columns: word, count_num with a row key of "word sum"
*/
{noformat}
Both the input and output tables look far more complex than necessary.
My preferred solution would be to just strip the output down to {(word text primary key, count int)}, and make a similar simplification for the input.
Can you shed any light [~alexliu68]?
> NPE in Hadoop Word count example
> --------------------------------
>
> Key: CASSANDRA-6793
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6793
> Project: Cassandra
> Issue Type: Bug
> Components: Examples
> Reporter: Chander S Pechetty
> Assignee: Chander S Pechetty
> Priority: Minor
> Labels: hadoop
> Attachments: trunk-6793.txt
>
>
> The partition keys requested in WordCount.java do not match the primary key set up in the table output_words. It looks this patch was not merged properly from [CASSANDRA-5622|https://issues.apache.org/jira/browse/CASSANDRA-5622].The attached patch addresses the NPE and uses the correct keys defined in #5622.
> I am assuming there is no need to fix the actual NPE like throwing an InvalidRequestException back to user to fix the partition keys, as it would be trivial to get the same from the TableMetadata using the driver API.
> java.lang.NullPointerException
> at org.apache.cassandra.dht.Murmur3Partitioner.getToken(Murmur3Partitioner.java:92)
> at org.apache.cassandra.dht.Murmur3Partitioner.getToken(Murmur3Partitioner.java:40)
> at org.apache.cassandra.client.RingCache.getRange(RingCache.java:117)
> at org.apache.cassandra.hadoop.cql3.CqlRecordWriter.write(CqlRecordWriter.java:163)
> at org.apache.cassandra.hadoop.cql3.CqlRecordWriter.write(CqlRecordWriter.java:63)
> at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:587)
> at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> at WordCount$ReducerToCassandra.reduce(Unknown Source)
> at WordCount$ReducerToCassandra.reduce(Unknown Source)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:260)
--
This message was sent by Atlassian JIRA
(v6.2#6252)