You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Ryan King <ry...@twitter.com> on 2011/06/14 22:57:55 UTC
dapper-style tracing in cassandra
I'll open a ticket on this soon, but I'd like to start a discussion first.
We're working on a distributed tracing system, whose design is
somewhat inspired by the Google Dapper paper [1]. We have instrumented
a bunch of our internal services through our custom networking stack
[2].
In a nutshell, the way it works is that each request is given a trace
id which gets passed through to each service involved in servicing
that request. Each hop in that tree is given a span id. Each node logs
its data to a local agent (we use scribe for this). An aggregator can
pull the pieces back together so you can do analysis.
I'd like to add the ability to plug tracers into cassandra. Like with
many things in cassandra, I think like many parts of Cassandra we
should make this an extensible point with a good default
implementation in place.
Here's what I propose:
1. Update the thrift server to allow clients to pass in tracing
details. I'll have docs soon on how we're doing this internally.
2. Add the necessary metadata to each message passed between cassandra
nodes. This should be easy to Message.java and thread through to the
places we need it.
3. Implement a universally useful version of this– one that's not
dependent on our system since it may not ever get open-sourced.
Perhaps writing to local files?
Thoughts? Opinions?
-ryan
1. http://research.google.com/pubs/pub36356.html
2. https://github.com/twitter/finagle/tree/master/finagle-b3
Re: JdbcDriverTest
Posted by Jonathan Ellis <jb...@gmail.com>.
FWIW I verified that the driver tests work today, with the patch from
http://issues.apache.org/jira/browse/CASSANDRA-2859 (will be committed
soon).
On Wed, Jun 15, 2011 at 7:15 AM, Vivek Mishra <vi...@yahoo.com> wrote:
> looking into logs leads me to this:
>
> java.lang.NoSuchMethodError: org.apache.commons.lang.StringUtils.join(Ljava/util/Collection;Ljava/lang/String;)Ljava/lang/String;
> at org.apache.cassandra.db.RowMutation.toString(RowMutation.java:263)
>
> I can see that version required and mentioned in build.xml is 2.4 of commons-lang. But same API is present with apache-rat version 0.6(included under maven-dependency)
>
> --- On Wed, 6/15/11, Vivek Mishra <vi...@yahoo.com> wrote:
>
> From: Vivek Mishra <vi...@yahoo.com>
> Subject: Re: JdbcDriverTest
> To: dev@cassandra.apache.org
> Date: Wednesday, June 15, 2011, 5:10 PM
>
> I can see that cassandra.yaml is not loading any default keyspace or column family now.
> Wondering if JdbcDriverTest should be modified to load required keyspace and Columnfamilies( e.g. JdbcInteger, Standard1) explicitly?
>
>
> --- On Wed, 6/15/11, Jonathan Ellis <jb...@gmail.com> wrote:
>
> From: Jonathan Ellis <jb...@gmail.com>
> Subject: Re: JdbcDriverTest
> To: dev@cassandra.apache.org
> Date: Wednesday, June 15, 2011, 5:05 PM
>
> Internal error means, the error is in the server logs.
> On Jun 15, 2011 5:33 AM, "Vivek Mishra" <vi...@yahoo.com> wrote:
>> Hi,
>> I am trying to run JdbcDriverTest. But somehow it is failing for
>>
>> java.sql.SQLException: Internal error processing execute_cql_query
>> at
> org.apache.cassandra.cql.jdbc.CassandraStatement.execute(CassandraStatement.java:176)
>>
>> This is happening on startServer(). Looking into Cassandra.yaml, i can see
> that definitions for JdbcInteger, Standard1 Col families are missing.
>>
>> Any idea about this?
>> I took latest 0.8.0 release.
>>
>> Vivek
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: JdbcDriverTest
Posted by Vivek Mishra <vi...@yahoo.com>.
looking into logs leads me to this:
java.lang.NoSuchMethodError: org.apache.commons.lang.StringUtils.join(Ljava/util/Collection;Ljava/lang/String;)Ljava/lang/String;
at org.apache.cassandra.db.RowMutation.toString(RowMutation.java:263)
I can see that version required and mentioned in build.xml is 2.4 of commons-lang. But same API is present with apache-rat version 0.6(included under maven-dependency)
--- On Wed, 6/15/11, Vivek Mishra <vi...@yahoo.com> wrote:
From: Vivek Mishra <vi...@yahoo.com>
Subject: Re: JdbcDriverTest
To: dev@cassandra.apache.org
Date: Wednesday, June 15, 2011, 5:10 PM
I can see that cassandra.yaml is not loading any default keyspace or column family now.
Wondering if JdbcDriverTest should be modified to load required keyspace and Columnfamilies( e.g. JdbcInteger, Standard1) explicitly?
--- On Wed, 6/15/11, Jonathan Ellis <jb...@gmail.com> wrote:
From: Jonathan Ellis <jb...@gmail.com>
Subject: Re: JdbcDriverTest
To: dev@cassandra.apache.org
Date: Wednesday, June 15, 2011, 5:05 PM
Internal error means, the error is in the server logs.
On Jun 15, 2011 5:33 AM, "Vivek Mishra" <vi...@yahoo.com> wrote:
> Hi,
> I am trying to run JdbcDriverTest. But somehow it is failing for
>
> java.sql.SQLException: Internal error processing execute_cql_query
> at
org.apache.cassandra.cql.jdbc.CassandraStatement.execute(CassandraStatement.java:176)
>
> This is happening on startServer(). Looking into Cassandra.yaml, i can see
that definitions for JdbcInteger, Standard1 Col families are missing.
>
> Any idea about this?
> I took latest 0.8.0 release.
>
> Vivek
Re: JdbcDriverTest
Posted by Vivek Mishra <vi...@yahoo.com>.
I can see that cassandra.yaml is not loading any default keyspace or column family now.
Wondering if JdbcDriverTest should be modified to load required keyspace and Columnfamilies( e.g. JdbcInteger, Standard1) explicitly?
--- On Wed, 6/15/11, Jonathan Ellis <jb...@gmail.com> wrote:
From: Jonathan Ellis <jb...@gmail.com>
Subject: Re: JdbcDriverTest
To: dev@cassandra.apache.org
Date: Wednesday, June 15, 2011, 5:05 PM
Internal error means, the error is in the server logs.
On Jun 15, 2011 5:33 AM, "Vivek Mishra" <vi...@yahoo.com> wrote:
> Hi,
> I am trying to run JdbcDriverTest. But somehow it is failing for
>
> java.sql.SQLException: Internal error processing execute_cql_query
> at
org.apache.cassandra.cql.jdbc.CassandraStatement.execute(CassandraStatement.java:176)
>
> This is happening on startServer(). Looking into Cassandra.yaml, i can see
that definitions for JdbcInteger, Standard1 Col families are missing.
>
> Any idea about this?
> I took latest 0.8.0 release.
>
> Vivek
Re: JdbcDriverTest
Posted by Jonathan Ellis <jb...@gmail.com>.
Internal error means, the error is in the server logs.
On Jun 15, 2011 5:33 AM, "Vivek Mishra" <vi...@yahoo.com> wrote:
> Hi,
> I am trying to run JdbcDriverTest. But somehow it is failing for
>
> java.sql.SQLException: Internal error processing execute_cql_query
> at
org.apache.cassandra.cql.jdbc.CassandraStatement.execute(CassandraStatement.java:176)
>
> This is happening on startServer(). Looking into Cassandra.yaml, i can see
that definitions for JdbcInteger, Standard1 Col families are missing.
>
> Any idea about this?
> I took latest 0.8.0 release.
>
> Vivek
JdbcDriverTest
Posted by Vivek Mishra <vi...@yahoo.com>.
Hi,
I am trying to run JdbcDriverTest. But somehow it is failing for
java.sql.SQLException: Internal error processing execute_cql_query
at org.apache.cassandra.cql.jdbc.CassandraStatement.execute(CassandraStatement.java:176)
This is happening on startServer(). Looking into Cassandra.yaml, i can see that definitions for JdbcInteger, Standard1 Col families are missing.
Any idea about this?
I took latest 0.8.0 release.
Vivek
Re: dapper-style tracing in cassandra
Posted by Ryan King <ry...@twitter.com>.
On Tue, Jun 14, 2011 at 2:02 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> Sounds a lot like
> https://issues.apache.org/jira/browse/CASSANDRA-1123. The main change
> you'd want is to allow passing an external trace ID.
Yeah, that patch seems like a good start. In addition to passing an
external trace id we need a way to plug in our our implementation of
what to do with the data (we want to publish thrift structs through
scribe).
-ryan
Re: dapper-style tracing in cassandra
Posted by Jonathan Ellis <jb...@gmail.com>.
Sounds a lot like
https://issues.apache.org/jira/browse/CASSANDRA-1123. The main change
you'd want is to allow passing an external trace ID.
On Tue, Jun 14, 2011 at 3:57 PM, Ryan King <ry...@twitter.com> wrote:
> I'll open a ticket on this soon, but I'd like to start a discussion first.
>
> We're working on a distributed tracing system, whose design is
> somewhat inspired by the Google Dapper paper [1]. We have instrumented
> a bunch of our internal services through our custom networking stack
> [2].
>
> In a nutshell, the way it works is that each request is given a trace
> id which gets passed through to each service involved in servicing
> that request. Each hop in that tree is given a span id. Each node logs
> its data to a local agent (we use scribe for this). An aggregator can
> pull the pieces back together so you can do analysis.
>
> I'd like to add the ability to plug tracers into cassandra. Like with
> many things in cassandra, I think like many parts of Cassandra we
> should make this an extensible point with a good default
> implementation in place.
>
> Here's what I propose:
>
> 1. Update the thrift server to allow clients to pass in tracing
> details. I'll have docs soon on how we're doing this internally.
> 2. Add the necessary metadata to each message passed between cassandra
> nodes. This should be easy to Message.java and thread through to the
> places we need it.
> 3. Implement a universally useful version of this– one that's not
> dependent on our system since it may not ever get open-sourced.
> Perhaps writing to local files?
>
> Thoughts? Opinions?
>
> -ryan
>
> 1. http://research.google.com/pubs/pub36356.html
> 2. https://github.com/twitter/finagle/tree/master/finagle-b3
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com