You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Adrian Mocanu <am...@verticalscope.com> on 2014/02/13 19:23:49 UTC
how to inc counter column w Calliope
Hi
I use Calliope (http://tuplejump.github.io/calliope/show-me-the-code.html) to update some counters into Cassandra.
If instead of counter column I use a text column the updates work, but with the counter type Calliope always fails to prepare cql query.
A: Counter of Text type:
val cas = CasBuilder.cql3.withColumnFamily("vsapi", "counters")
.saveWithQuery("UPDATE vsapi.counters set counter = ? ")
This works for text mode. No surprise here.
B: Counter of Counter type:
val cas = CasBuilder.cql3.withColumnFamily("vsapi", "counters")
.saveWithQuery("UPDATE vsapi.counters set counter = counter + ? ")
This gives some byte related error. Here's the stack trace... some of it:
14/02/13 13:13:51 ERROR executor.Executor: Exception in task ID 5
java.io.IOException: InvalidRequestException(why:Expected 8 or 0 byte long (4))
at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:245)
Caused by: InvalidRequestException(why:Expected 8 or 0 byte long (4))
at org.apache.cassandra.thrift.Cassandra$execute_prepared_cql3_query_result$execute_prepared_cql3_query_resultStandardScheme.read(Cassandra.java:53653)
at org.apache.cassandra.thrift.Cassandra$execute_prepared_cql3_query_result$execute_prepared_cql3_query_resultStandardScheme.read(Cassandra.java:53630)
at org.apache.cassandra.thrift.Cassandra$execute_prepared_cql3_query_result.read(Cassandra.java:53545)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at org.apache.cassandra.thrift.Cassandra$Client.recv_execute_prepared_cql3_query(Cassandra.java:1820)
at org.apache.cassandra.thrift.Cassandra$Client.execute_prepared_cql3_query(Cassandra.java:1805)
at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:229)
14/02/13 13:13:51 WARN scheduler.TaskSetManager: Lost TID 5 (task 7.0:0)
14/02/13 13:13:51 WARN scheduler.TaskSetManager: Loss was due to java.io.IOException
java.io.IOException: InvalidRequestException(why:Expected 8 or 0 byte long (4))
at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:245)
14/02/13 13:13:51 ERROR scheduler.TaskSetManager: Task 7.0:0 failed 1 times; aborting job
14/02/13 13:13:51 INFO scheduler.TaskSchedulerImpl: Remove TaskSet 7.0 from pool
14/02/13 13:13:52 INFO scheduler.DAGScheduler: Failed to run saveAsNewAPIHadoopFile at CassandraRDDFunctions.scala:185
Exception in thread "main" org.apache.spark.SparkException: Job aborted: Task 7.0:0 failed 1 times (most recent failure: Exception failure: java.io.IOException: InvalidRequestException(why:Expected 8 or 0 byte long (4)))
If I change the query back to one I use for counter columns of text type I get
InvalidRequestException(why:Cannot set the value of counter column counter (counters can only be incremented/decremented, not set))
And this is expected because I should not be allowed to set a counter, only to increment it.
Extra Notes:
These are the marshallers:
implicit def keyMarshaller(x: (String, String, String)): CQLRowKeyMap = Map("unique_name" -> x._2, "tuple_minute_timebin -> x._3)
implicit def rowMarshaller(x: (String, String, String)): CQLRowValues = List(x._1) <-----these are the values the counter is to be incremented by.
val cas = CasBuilder.cql3.withColumnFamily("vsapi", "counters")
.saveWithQuery("UPDATE vsapi.counters set ?WHAT TO PUT HERE? ")
dataMappedToColumnOrder.cql3SaveToCassandra(cas)
Any ideas how to increment counters with Calliope?
Thanks
-A
RE: how to inc counter column w Calliope
Posted by Adrian Mocanu <am...@verticalscope.com>.
That worked! Thanks a lot!
PS - Got your other message too
From: Rohit Rai [mailto:rohit@tuplejump.com]
Sent: February-14-14 12:32 AM
To: user
Subject: Re: how to inc counter column w Calliope
val cas = CasBuilder.cql3.withColumnFamily("vsapi", "counters")
.saveWithQuery("UPDATE vsapi.counters set counter = counter + ? ")
is the correct thing to use...
The problem here is in the marshaller for the rowValues...
implicit def rowMarshaller(x: (String, String, String)): CQLRowValues = List(x._1)
A counter value is a long and can only be incremented with a Long. so the marshaller will be -
implicit def rowMarshaller(x: (String, String, String)): CQLRowValues = List(x._1.toLong)
i.e. converting string to long... and that should solve the issues...
Let us know if you still face trouble.
Will add an example of this to test cases and documentation... Hold me to that :)
Cheers,
Rohit
Founder & CEO, Tuplejump, Inc.
____________________________
www.tuplejump.com<http://www.tuplejump.com>
The Data Engineering Platform
On Thu, Feb 13, 2014 at 11:53 PM, Adrian Mocanu <am...@verticalscope.com>> wrote:
Hi
I use Calliope (http://tuplejump.github.io/calliope/show-me-the-code.html) to update some counters into Cassandra.
If instead of counter column I use a text column the updates work, but with the counter type Calliope always fails to prepare cql query.
A: Counter of Text type:
val cas = CasBuilder.cql3.withColumnFamily("vsapi", "counters")
.saveWithQuery("UPDATE vsapi.counters set counter = ? ")
This works for text mode. No surprise here.
B: Counter of Counter type:
val cas = CasBuilder.cql3.withColumnFamily("vsapi", "counters")
.saveWithQuery("UPDATE vsapi.counters set counter = counter + ? ")
This gives some byte related error. Here's the stack trace... some of it:
14/02/13 13:13:51 ERROR executor.Executor: Exception in task ID 5
java.io.IOException: InvalidRequestException(why:Expected 8 or 0 byte long (4))
at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:245)
Caused by: InvalidRequestException(why:Expected 8 or 0 byte long (4))
at org.apache.cassandra.thrift.Cassandra$execute_prepared_cql3_query_result$execute_prepared_cql3_query_resultStandardScheme.read(Cassandra.java:53653)
at org.apache.cassandra.thrift.Cassandra$execute_prepared_cql3_query_result$execute_prepared_cql3_query_resultStandardScheme.read(Cassandra.java:53630)
at org.apache.cassandra.thrift.Cassandra$execute_prepared_cql3_query_result.read(Cassandra.java:53545)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at org.apache.cassandra.thrift.Cassandra$Client.recv_execute_prepared_cql3_query(Cassandra.java:1820)
at org.apache.cassandra.thrift.Cassandra$Client.execute_prepared_cql3_query(Cassandra.java:1805)
at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:229)
14/02/13 13:13:51 WARN scheduler.TaskSetManager: Lost TID 5 (task 7.0:0)
14/02/13 13:13:51 WARN scheduler.TaskSetManager: Loss was due to java.io.IOException
java.io.IOException: InvalidRequestException(why:Expected 8 or 0 byte long (4))
at org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:245)
14/02/13 13:13:51 ERROR scheduler.TaskSetManager: Task 7.0:0 failed 1 times; aborting job
14/02/13 13:13:51 INFO scheduler.TaskSchedulerImpl: Remove TaskSet 7.0 from pool
14/02/13 13:13:52 INFO scheduler.DAGScheduler: Failed to run saveAsNewAPIHadoopFile at CassandraRDDFunctions.scala:185
Exception in thread "main" org.apache.spark.SparkException: Job aborted: Task 7.0:0 failed 1 times (most recent failure: Exception failure: java.io.IOException: InvalidRequestException(why:Expected 8 or 0 byte long (4)))
If I change the query back to one I use for counter columns of text type I get
InvalidRequestException(why:Cannot set the value of counter column counter (counters can only be incremented/decremented, not set))
And this is expected because I should not be allowed to set a counter, only to increment it.
Extra Notes:
These are the marshallers:
implicit def keyMarshaller(x: (String, String, String)): CQLRowKeyMap = Map("unique_name" -> x._2, "tuple_minute_timebin -> x._3)
implicit def rowMarshaller(x: (String, String, String)): CQLRowValues = List(x._1) <-----these are the values the counter is to be incremented by.
val cas = CasBuilder.cql3.withColumnFamily("vsapi", "counters")
.saveWithQuery("UPDATE vsapi.counters set ?WHAT TO PUT HERE? ")
dataMappedToColumnOrder.cql3SaveToCassandra(cas)
Any ideas how to increment counters with Calliope?
Thanks
-A
Re: how to inc counter column w Calliope
Posted by Rohit Rai <ro...@tuplejump.com>.
val cas = CasBuilder.cql3.withColumnFamily("vsapi", "counters")
.saveWithQuery("UPDATE vsapi.counters set *counter =
counter + ?* ")
is the correct thing to use...
The problem here is in the marshaller for the rowValues...
implicit def rowMarshaller(x: (String, String, String)): CQLRowValues =
List(x._1*)*
A counter value is a long and can only be incremented with a Long. so the
marshaller will be -
implicit def rowMarshaller(x: (String, String, String)): CQLRowValues =
List(x._1*.toLong**)*
i.e. converting string to long... and that should solve the issues...
Let us know if you still face trouble.
Will add an example of this to test cases and documentation... Hold me to
that :)
Cheers,
Rohit
*Founder & CEO, **Tuplejump, Inc.*
____________________________
www.tuplejump.com
*The Data Engineering Platform*
On Thu, Feb 13, 2014 at 11:53 PM, Adrian Mocanu
<am...@verticalscope.com>wrote:
> Hi
>
> I use Calliope (http://tuplejump.github.io/calliope/show-me-the-code.html)
> to update some counters into Cassandra.
>
> If instead of counter column I use a text column the updates work, but
> with the counter type Calliope always fails to prepare cql query.
>
>
>
> *A: Counter of Text type:*
>
> val cas = CasBuilder.cql3.withColumnFamily("vsapi", "counters")
>
> .saveWithQuery("UPDATE vsapi.counters set counter =
> ? ")
>
> This works for text mode. No surprise here.
>
>
>
> *B: Counter of Counter type:*
>
> val cas = CasBuilder.cql3.withColumnFamily("vsapi", "counters")
>
> .saveWithQuery("UPDATE vsapi.counters set *counter
> = counter + ?* ")
>
>
>
> This gives some byte related error. Here's the stack trace... some of it:
>
>
>
> 14/02/13 13:13:51 ERROR executor.Executor: Exception in task ID 5
>
> java.io.IOException: InvalidRequestException(why:Expected 8 or 0 byte long
> (4))
>
> at
> org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:245)
>
> Caused by: InvalidRequestException(why:Expected 8 or 0 byte long (4))
>
> at
> org.apache.cassandra.thrift.Cassandra$execute_prepared_cql3_query_result$execute_prepared_cql3_query_resultStandardScheme.read(Cassandra.java:53653)
>
> at
> org.apache.cassandra.thrift.Cassandra$execute_prepared_cql3_query_result$execute_prepared_cql3_query_resultStandardScheme.read(Cassandra.java:53630)
>
> at
> org.apache.cassandra.thrift.Cassandra$execute_prepared_cql3_query_result.read(Cassandra.java:53545)
>
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
>
> at
> org.apache.cassandra.thrift.Cassandra$Client.recv_execute_prepared_cql3_query(Cassandra.java:1820)
>
> at
> org.apache.cassandra.thrift.Cassandra$Client.execute_prepared_cql3_query(Cassandra.java:1805)
>
> at
> org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:229)
>
> 14/02/13 13:13:51 WARN scheduler.TaskSetManager: Lost TID 5 (task 7.0:0)
>
> 14/02/13 13:13:51 WARN scheduler.TaskSetManager: Loss was due to
> java.io.IOException
>
> java.io.IOException: InvalidRequestException(why:Expected 8 or 0 byte long
> (4))
>
> at
> org.apache.cassandra.hadoop.cql3.CqlRecordWriter$RangeClient.run(CqlRecordWriter.java:245)
>
> 14/02/13 13:13:51 ERROR scheduler.TaskSetManager: Task 7.0:0 failed 1
> times; aborting job
>
> 14/02/13 13:13:51 INFO scheduler.TaskSchedulerImpl: Remove TaskSet 7.0
> from pool
>
> 14/02/13 13:13:52 INFO scheduler.DAGScheduler: Failed to run
> saveAsNewAPIHadoopFile at CassandraRDDFunctions.scala:185
>
> Exception in thread "main" org.apache.spark.SparkException: Job aborted:
> Task 7.0:0 failed 1 times (most recent failure: Exception failure:
> java.io.IOException: InvalidRequestException(why:Expected 8 or 0 byte long
> (4)))
>
>
>
>
>
> If I change the query back to one I use for counter columns of text type I
> get
>
> *InvalidRequestException(why:Cannot set the value of counter column
> counter (counters can only be incremented/decremented, not set))*
>
> And this is expected because I should not be allowed to set a counter,
> only to increment it.
>
>
>
>
>
> Extra Notes:
>
> *These are the marshallers*:
>
> implicit def keyMarshaller(x: (String, String, String)): CQLRowKeyMap =
> Map("unique_name" -> x._2, "tuple_minute_timebin -> x._3)
>
> implicit def rowMarshaller(x: (String, String, String)): CQLRowValues
> = List(x._1*) <-----these are the values the counter is to be
> incremented by.*
>
>
>
> val cas = CasBuilder.cql3.withColumnFamily("vsapi", "counters")
>
> .saveWithQuery("UPDATE vsapi.counters set ?*WHAT TO
> PUT HERE?* ")
>
> dataMappedToColumnOrder.cql3SaveToCassandra(cas)
>
>
>
>
>
> Any ideas how to increment counters with Calliope?
>
>
>
> Thanks
>
> -A
>
>
>