[jira] [Commented] (DRILL-92) Cassandra storage engine

["Robert Stupp (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-92?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14335238#comment-14335238 ] 

Robert Stupp commented on DRILL-92:
-----------------------------------

I went over your patch just to see how the actual C* integration has been implemented. Tbh - I don’t know how Drill works - but I know how C* and the Java Driver work.

Please let me explain some things in advance. A Cassandra cluster exists of many nodes. Some of them might be down without affecting the integrity of the whole cluster. Due to the fact that some hosts might be down, the Datastax Java Driver allows to specify *multiple* initial contact points - and multiple initial contact points (maybe 3 per data center) should be passed to {{Cluster.Builder}}. All connections to a C* cluster are managed by the {{Cluster}} instance - not directly by {{Session}}. That means: to effectively close connections to a cluster, you have to close the {{Cluster}} instance. Further, the {{Cluster}} instance learns about all other nodes in the C* cluster - i.e. it will know all nodes in the cluster, which token ranges they server, and it will perform a best-effort-approach to route direct DML statements (SELECT/INSERT/UPDATE/DELETE) to the nodes that hold replicas of them. A usual application does not care about where data actually lives - it’s handled by the Java Driver for you.

The lines in {{CassandraGroupScan}} calling {{com.datastax.driver.core.Metadata#getReplicas}} are wrong. Which nodes are replicas for a keyspace are defined by the replication strategy and the per-keyspace configuration. The method you’re calling determines the hosts for a specific _partition key_ - but you’re passing in the class name of partitioner. Those are completely different things.
Although not completely wrong, I’d encourage you not to assume which nodes hold the tokens you intend to request (in {{CassandraUtil}}). There are several other things that influence where data ”lives” - e.g. datacenter and rack awareness.

In {{CassandraSchemaFactory}} is a keyspace cache and a table cache. That’s completely superfluous since the Java Driver already holds that information in the {{Cluster}} instance and it gets automagically updated when the cluster topology and/or the schema changes. That kind of metadata is essential for the Java Driver to work and always present.

I’d recommend to start with a different approach and consider the current patch as a _proof-of-concept_ (you may of course take over working code):
# Learn a bit more about C* and the Java Driver architecture ;)
# Forget about accessing the ”nearest” node in an initial attempt - you can add that later anyway. BTW that does only make sense, if you have Drill slaves (don’t know if such exist) running on each C* node.
# Start with a simple cluster to work against. Take a look at _ccm_ - it’s a neat tool that spawns a C* using multiple nodes on your local machine: https://github.com/pcmanus/ccm/.
# If you have a basic implementation running you may improve it by adding datacenter-awareness to your client (it’s basically just a simple configuration using {{Cluster.Builder}}, authentication against the C* cluster and some other fine tuning

Feel free to ask questions on C* user mailing list user@cassandra.apache.org or on freenode IRC #cassandra. There are many people happy to answer individual questions. Just ask - don’t ask to ask :)


> Cassandra storage engine
> ------------------------
>
>                 Key: DRILL-92
>                 URL: https://issues.apache.org/jira/browse/DRILL-92
>             Project: Apache Drill
>          Issue Type: New Feature
>            Reporter: Steven Phillips
>            Assignee: Yash Sharma
>             Fix For: Future
>
>         Attachments: DRILL-92.patch, DRILL-CASSANDRA.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)