You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Aaron Morton (JIRA)" <ji...@apache.org> on 2011/08/01 05:49:09 UTC
[jira] [Created] (CASSANDRA-2981) Provide Hadoop read access to
Counter Columns.
Provide Hadoop read access to Counter Columns.
----------------------------------------------
Key: CASSANDRA-2981
URL: https://issues.apache.org/jira/browse/CASSANDRA-2981
Project: Cassandra
Issue Type: Improvement
Components: Hadoop
Affects Versions: 0.8.2
Reporter: Aaron Morton
Assignee: Aaron Morton
Priority: Minor
o.a.c.Hadoop.ColumnFamilyRecordReader does not test for counter columns, which are different objects in the ColumnOrSuperColumn struct. Currently it raises an error as it thinks it's a super column
{code:java}
2011-07-26 17:23:34,376 ERROR CliDriver (SessionState.java:printError(343)) - Failed with exception java.io.IOException:java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:341)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:133)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1114)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:456)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.NullPointerException
at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.unthriftifySuper(ColumnFamilyRecordReader.java:303)
at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.unthriftify(ColumnFamilyRecordReader.java:297)
at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:288)
at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:177)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(ColumnFamilyRecordReader.java:136)
at org.apache.hadoop.hive.cassandra.input.HiveCassandraStandardColumnInputFormat$2.next(HiveCassandraStandardColumnInputFormat.java:153)
at org.apache.hadoop.hive.cassandra.input.HiveCassandraStandardColumnInputFormat$2.next(HiveCassandraStandardColumnInputFormat.java:111)
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:326)
... 10 more
{code}
My plan is to return o.a.c.db.CounterColumn objects just like the o.a.c.db.Column and SuperColumn that are returned.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2981) Provide Hadoop read access to
Counter Columns.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13073573#comment-13073573 ]
Hudson commented on CASSANDRA-2981:
-----------------------------------
Integrated in Cassandra-0.8 #248 (See [https://builds.apache.org/job/Cassandra-0.8/248/])
add counter support to Hadoop InputFormat
patch by Aaron Morton; reviewed by jbellis for CASSANDRA-2981
jbellis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1152782
Files :
* /cassandra/branches/cassandra-0.8/CHANGES.txt
* /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/hadoop/ColumnFamilyRecordReader.java
* /cassandra/branches/cassandra-0.8/examples/hadoop_word_count/bin/word_count_counters
* /cassandra/branches/cassandra-0.8/examples/hadoop_word_count/src/WordCountCounters.java
* /cassandra/branches/cassandra-0.8/examples/hadoop_word_count/README.txt
* /cassandra/branches/cassandra-0.8/examples/hadoop_word_count/src/WordCountSetup.java
> Provide Hadoop read access to Counter Columns.
> ----------------------------------------------
>
> Key: CASSANDRA-2981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2981
> Project: Cassandra
> Issue Type: Improvement
> Components: Hadoop
> Affects Versions: 0.8.2
> Reporter: Aaron Morton
> Assignee: Aaron Morton
> Priority: Minor
> Fix For: 0.8.3
>
> Attachments: 0001-2981-hadoop-counters-input.patch
>
>
> o.a.c.Hadoop.ColumnFamilyRecordReader does not test for counter columns, which are different objects in the ColumnOrSuperColumn struct. Currently it raises an error as it thinks it's a super column
> {code:java}
> 2011-07-26 17:23:34,376 ERROR CliDriver (SessionState.java:printError(343)) - Failed with exception java.io.IOException:java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:341)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:133)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1114)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:456)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.NullPointerException
> at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.unthriftifySuper(ColumnFamilyRecordReader.java:303)
> at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.unthriftify(ColumnFamilyRecordReader.java:297)
> at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:288)
> at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:177)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> at org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(ColumnFamilyRecordReader.java:136)
> at org.apache.hadoop.hive.cassandra.input.HiveCassandraStandardColumnInputFormat$2.next(HiveCassandraStandardColumnInputFormat.java:153)
> at org.apache.hadoop.hive.cassandra.input.HiveCassandraStandardColumnInputFormat$2.next(HiveCassandraStandardColumnInputFormat.java:111)
> at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:326)
> ... 10 more
> {code}
> My plan is to return o.a.c.db.CounterColumn objects just like the o.a.c.db.Column and SuperColumn that are returned.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2981) Provide Hadoop read access to
Counter Columns.
Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Morton updated CASSANDRA-2981:
------------------------------------
Attachment: 0001-2981-hadoop-counters-input.patch
0001-2981-hadoop-counters-input.patch modifies the CFRR to turn CounterColumns returned through the thrift API into o.a.c.db.Column instances.
Could not use the CounterColumn as the CounterContext needs to read the node ID, and this requires the StorageService to be running and access to cassandra.yaml.
It's not great, but the full CounterColumn should not be needed as Hadoop is read only access. Let me know it's too hacky.
Also added another test to the hadoop_word_count example that sums the counter columns in a row.
> Provide Hadoop read access to Counter Columns.
> ----------------------------------------------
>
> Key: CASSANDRA-2981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2981
> Project: Cassandra
> Issue Type: Improvement
> Components: Hadoop
> Affects Versions: 0.8.2
> Reporter: Aaron Morton
> Assignee: Aaron Morton
> Priority: Minor
> Attachments: 0001-2981-hadoop-counters-input.patch
>
>
> o.a.c.Hadoop.ColumnFamilyRecordReader does not test for counter columns, which are different objects in the ColumnOrSuperColumn struct. Currently it raises an error as it thinks it's a super column
> {code:java}
> 2011-07-26 17:23:34,376 ERROR CliDriver (SessionState.java:printError(343)) - Failed with exception java.io.IOException:java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:341)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:133)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1114)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:456)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.NullPointerException
> at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.unthriftifySuper(ColumnFamilyRecordReader.java:303)
> at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.unthriftify(ColumnFamilyRecordReader.java:297)
> at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:288)
> at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:177)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> at org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(ColumnFamilyRecordReader.java:136)
> at org.apache.hadoop.hive.cassandra.input.HiveCassandraStandardColumnInputFormat$2.next(HiveCassandraStandardColumnInputFormat.java:153)
> at org.apache.hadoop.hive.cassandra.input.HiveCassandraStandardColumnInputFormat$2.next(HiveCassandraStandardColumnInputFormat.java:111)
> at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:326)
> ... 10 more
> {code}
> My plan is to return o.a.c.db.CounterColumn objects just like the o.a.c.db.Column and SuperColumn that are returned.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira