You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gora.apache.org by madhawa-gunasekara <gi...@git.apache.org> on 2017/06/16 18:03:32 UTC
[GitHub] gora pull request #110: Initial commit gora cassandra cql
GitHub user madhawa-gunasekara opened a pull request:
https://github.com/apache/gora/pull/110
Initial commit gora cassandra cql
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/madhawa-gunasekara/gora ISSUE-497
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/gora/pull/110.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #110
----
commit cd3522aa47c542594dc9cc4d91b023a65db71ae9
Author: madhawa-gunasekara <ma...@wso2.com>
Date: 2017-06-12T19:33:21Z
Initial commit gora cassandra cql with Connection parameters
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora issue #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on the issue:
https://github.com/apache/gora/pull/110
Hi @madhawa-gunasekara what is the status of this PR now? Thank you
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r129359927
--- Diff: gora-cassandra-cql/src/examples/avro/cassandraRecord.json ---
@@ -0,0 +1,106 @@
+{
+ "type": "record",
+ "name": "CassandraRecord",
+ "default": null,
+ "namespace": "org.apache.gora.cassandra.example.generated.avroSerialization",
--- End diff --
Please do the same with this schema.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora issue #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by madhawa-gunasekara <gi...@git.apache.org>.
Github user madhawa-gunasekara commented on the issue:
https://github.com/apache/gora/pull/110
Hi @lewismc It's complete.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r132826245
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/serializers/CassandraSerializer.java ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.serializers;
+
+import com.datastax.driver.core.KeyspaceMetadata;
+import com.datastax.driver.core.ResultSet;
+import com.datastax.driver.core.TableMetadata;
+import org.apache.avro.Schema;
+import org.apache.gora.cassandra.bean.Field;
+import org.apache.gora.cassandra.store.CassandraClient;
+import org.apache.gora.cassandra.store.CassandraMapping;
+import org.apache.gora.cassandra.store.CassandraStore;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.store.DataStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+
+/**
+ * This is the abstract Cassandra Serializer class.
+ */
+public abstract class CassandraSerializer<K, T extends Persistent> {
+ private static final Logger LOG = LoggerFactory.getLogger(CassandraStore.class);
+ protected Class<K> keyClass;
+
+ protected Class<T> persistentClass;
+ protected CassandraMapping mapping;
+ CassandraClient client;
--- End diff --
Please use suitable access modifier here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora issue #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by renato2099 <gi...@git.apache.org>.
Github user renato2099 commented on the issue:
https://github.com/apache/gora/pull/110
Hi @madhawa-gunasekara,
I think this is looking pretty good! Quick question, I saw that you have created a CassandraNativePersistent which implements Persistent, so for the Avro serialization will use the same right? So How is this going to play with the Gora Compiler? Will we have to modify that as well? Couldn't we just use the Persistent class? :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r132717674
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/serializers/CassandraSerializer.java ---
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.serializers;
+
+import com.datastax.driver.core.KeyspaceMetadata;
+import com.datastax.driver.core.ResultSet;
+import com.datastax.driver.core.TableMetadata;
+import org.apache.avro.Schema;
+import org.apache.gora.cassandra.bean.Field;
+import org.apache.gora.cassandra.store.CassandraClient;
+import org.apache.gora.cassandra.store.CassandraMapping;
+import org.apache.gora.cassandra.store.CassandraStore;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.store.DataStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+
+/**
+ * This is the abstract Cassandra Serializer class.
+ */
+public abstract class CassandraSerializer<K, T extends Persistent> {
+ private static final Logger LOG = LoggerFactory.getLogger(CassandraStore.class);
+ protected Class<K> keyClass;
+
+ protected Class<T> persistentClass;
+
+ private Map<String, String> userDefineTypeMaps;
+
+ protected CassandraMapping mapping;
+ private Schema persistentSchema;
+ CassandraClient client;
+
+ CassandraSerializer(CassandraClient cc, Class<K> keyClass, Class<T> persistantClass, CassandraMapping mapping, Schema schema) {
+ this.keyClass = keyClass;
+ this.persistentClass = persistantClass;
+ this.client = cc;
+ this.mapping = mapping;
+ persistentSchema = schema;
+ try {
+ analyzePersistent();
+ } catch (Exception e) {
+ throw new RuntimeException("Error occurred while analyzing the persistent class, :" + e.getMessage());
+ }
+ }
+
+ /**
+ * This method returns the Cassandra Serializer according the Cassandra serializer property.
+ *
+ * @param cc Cassandra Client
+ * @param type Serialization type
+ * @param dataStore Cassandra DataStore
+ * @param mapping Cassandra Mapping
+ * @param <K> key class
+ * @param <T> persistent class
+ * @return Serializer
+ */
+ public static <K, T extends Persistent> CassandraSerializer getSerializer(CassandraClient cc, String type, final DataStore<K, T> dataStore, CassandraMapping mapping, Schema schema) {
+ CassandraStore.SerializerType serType = type.isEmpty() ? CassandraStore.SerializerType.NATIVE : CassandraStore.SerializerType.valueOf(type.toUpperCase(Locale.ENGLISH));
+ CassandraSerializer serializer;
+ switch (serType) {
+ case AVRO:
+ serializer = new AvroSerializer(cc, dataStore, mapping, schema);
+ break;
+ case NATIVE:
+ default:
+ serializer = new NativeSerializer(cc, dataStore.getKeyClass(), dataStore.getPersistentClass(), mapping, schema);
+ }
+ return serializer;
+ }
+
+ private void analyzePersistent() throws Exception {
+ userDefineTypeMaps = new HashMap<>();
+ for (Field field : mapping.getFieldList()) {
+ String fieldType = field.getType();
+ if (fieldType.contains("frozen")) {
+ String udtType = fieldType.substring(fieldType.indexOf("<") + 1, fieldType.indexOf(">"));
+ if (this instanceof AvroSerializer) {
+ if (PersistentBase.class.isAssignableFrom(persistentClass)) {
+ Schema fieldSchema = persistentSchema.getField(field.getFieldName()).schema();
+ if (fieldSchema.getType().equals(Schema.Type.UNION)) {
+ for (Schema currentSchema : fieldSchema.getTypes()) {
+ if (currentSchema.getType().equals(Schema.Type.RECORD)) {
+ fieldSchema = currentSchema;
+ break;
+ }
+ }
+ }
+ String createQuery = CassandraQueryFactory.getCreateUDTTypeForAvro(mapping, udtType, fieldSchema);
+ userDefineTypeMaps.put(udtType, createQuery);
+ } else {
+ throw new RuntimeException("Unsupported Class for User Define Types, Please use PersistentBase class. field : " + udtType);
+ }
+ } else {
+ String createQuery = CassandraQueryFactory.getCreateUDTTypeForNative(mapping, persistentClass, udtType, field.getFieldName());
+ userDefineTypeMaps.put(udtType, createQuery);
+ }
+ }
+ }
+
+ }
+
--- End diff --
Better if this unnecessary new lines are removed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r130024629
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/bean/ClusterKeyField.java ---
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.bean;
+
+/**
+ * This class represents Cassandra Clustering Key.
+ */
+public class ClusterKeyField {
+
+ public String getColumnName() {
+ return columnName;
+ }
+
+ public void setColumnName(String columnName) {
+ this.columnName = columnName;
+ }
+
+ private String columnName;
+
+ public ClusterKeyField() {
+
+ }
+
+ public enum Order {
--- End diff --
Please document these enum fields - DESC, ASC,
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r132715495
--- Diff: gora-cassandra-cql/pom.xml ---
@@ -0,0 +1,238 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora</artifactId>
+ <version>0.8-SNAPSHOT</version>
+ <relativePath>../</relativePath>
+ </parent>
+ <artifactId>gora-cassandra-cql</artifactId>
+ <packaging>bundle</packaging>
+
+ <name>Apache Gora :: Cassandra - CQL</name>
+ <url>http://gora.apache.org</url>
+ <description>The Apache Gora open source framework provides an in-memory data model and
+ persistence for big data. Gora supports persisting to column stores, key value stores,
+ document stores and RDBMSs, and analyzing the data with extensive Apache Hadoop MapReduce
+ support.
+ </description>
+ <inceptionYear>2010</inceptionYear>
+ <organization>
+ <name>The Apache Software Foundation</name>
+ <url>http://www.apache.org/</url>
+ </organization>
+ <issueManagement>
+ <system>JIRA</system>
+ <url>https://issues.apache.org/jira/browse/GORA</url>
+ </issueManagement>
+ <ciManagement>
+ <system>Jenkins</system>
+ <url>https://builds.apache.org/job/Gora-trunk/</url>
+ </ciManagement>
+
+ <properties>
+ <guava.version>18.0</guava.version>
+ <osgi.import>*</osgi.import>
+ <osgi.export>org.apache.gora.cassandra*;version="${project.version}";-noimport:=true</osgi.export>
+ </properties>
+
+ <build>
+ <directory>target</directory>
+ <outputDirectory>target/classes</outputDirectory>
+ <finalName>${project.artifactId}-${project.version}</finalName>
+ <testOutputDirectory>target/test-classes</testOutputDirectory>
+ <testSourceDirectory>src/test/java</testSourceDirectory>
+ <sourceDirectory>src/main/java</sourceDirectory>
+ <testResources>
+ <testResource>
+ <directory>${project.basedir}/src/test/conf</directory>
+ <includes>
+ <include>**/*</include>
+ </includes>
+ <!--targetPath>${project.basedir}/target/classes/</targetPath-->
+ </testResource>
+ </testResources>
+ <plugins>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>build-helper-maven-plugin</artifactId>
+ <version>${build-helper-maven-plugin.version}</version>
+ <executions>
+ <execution>
+ <phase>generate-sources</phase>
+ <goals>
+ <goal>add-source</goal>
+ </goals>
+ <configuration>
+ <sources>
+ <source>src/examples/java</source>
+ </sources>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-plugin</artifactId>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.maven.surefire</groupId>
+ <artifactId>surefire-junit47</artifactId>
+ <version>2.20</version>
+ </dependency>
+ </dependencies>
+ <configuration>
+ <forkMode>always</forkMode>
+ <argLine>-Xmx1024m -XX:MaxPermSize=512m</argLine>
+ </configuration>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-jar-plugin</artifactId>
+ <configuration>
+ <archive>
+ <manifest>
+ <mainClass>org.apache.gora.cassandra.compiler.GoraCassandraNativeCompiler</mainClass>
+ <packageName>org.apache.gora.cassandra.compiler</packageName>
+ </manifest>
+ </archive>
+ </configuration>
+ </plugin>
+ </plugins>
+ </build>
+
+ <dependencies>
+ <!-- Gora Internal Dependencies -->
+ <dependency>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora-core</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora-core</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+
+ <!-- Cassandra Driver Dependencies -->
+ <dependency>
+ <groupId>com.datastax.cassandra</groupId>
+ <artifactId>cassandra-driver-core</artifactId>
+ <version>${cassandra-driver.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>com.datastax.cassandra</groupId>
+ <artifactId>cassandra-driver-mapping</artifactId>
+ <version>${cassandra-driver.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>com.datastax.cassandra</groupId>
+ <artifactId>cassandra-driver-extras</artifactId>
+ <version>${cassandra-driver.version}</version>
+ </dependency>
+
+ <!-- Cassandra Dependencies -->
+ <dependency>
+ <groupId>org.apache.cassandra</groupId>
+ <artifactId>cassandra-all</artifactId>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.cassandra.deps</groupId>
+ <artifactId>avro</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-log4j12</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-handler</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>log4j-over-slf4j</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+<!-- <dependency>
--- End diff --
Shall we cleanup this commented dependency?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r130027514
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/GoraCassandraTestDriver.java ---
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * @author lewismc
+ *
+ */
+
+package org.apache.gora.cassandra;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.apache.cassandra.io.util.FileUtils;
+import org.apache.cassandra.service.CassandraDaemon;
+import org.apache.gora.GoraTestDriver;
+import org.apache.gora.cassandra.store.CassandraStore;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.store.DataStoreFactory;
+import org.apache.gora.util.GoraException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.util.Properties;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+
+// Logging imports
+
+/**
+ * Helper class for third party tests using gora-cassandra backend.
+ * @see GoraTestDriver for test specifics.
+ * This driver is the base for all test cases that require an embedded Cassandra
+ * server. In this case we draw on Hector's @see EmbeddedServerHelper.
+ * It starts (setUp) and stops (tearDown) embedded Cassandra server.
+ *
+ */
+
+public class GoraCassandraTestDriver extends GoraTestDriver {
+ private static Logger log = LoggerFactory.getLogger(GoraCassandraTestDriver.class);
+
+ private static String baseDirectory = "target/test";
+
+ private CassandraDaemon cassandraDaemon;
+
+ private Thread cassandraThread;
+
+ private Properties properties;
+
+ public void setParameters(Properties parameters) {
+ this.properties = parameters;
+ }
+
+ @Override
+ public <K, T extends Persistent> DataStore<K, T> createDataStore(Class<K> keyClass, Class<T> persistentClass) throws GoraException {
+ return DataStoreFactory.createDataStore(CassandraStore.class, keyClass, persistentClass , conf, properties, null);
+ }
+
+ /**
+ * @return temporary base directory of running cassandra instance
+ */
+ public String getBaseDirectory() {
+ return baseDirectory;
+ }
+
+ public GoraCassandraTestDriver() {
+ super(CassandraStore.class);
+ }
+
+ /**
+ * Starts embedded Cassandra server.
+ *
+ * @throws Exception
+ * if an error occurs
+ */
+ @Override
+ public void setUpClass(){
--- End diff --
Seems this is not formatted properly - setUpClass() {
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r124050129
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java ---
@@ -0,0 +1,788 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.store;
+
+import com.datastax.driver.core.Cluster;
+import com.datastax.driver.core.ConsistencyLevel;
+import com.datastax.driver.core.HostDistance;
+import com.datastax.driver.core.KeyspaceMetadata;
+import com.datastax.driver.core.PoolingOptions;
+import com.datastax.driver.core.ProtocolOptions;
+import com.datastax.driver.core.ProtocolVersion;
+import com.datastax.driver.core.QueryOptions;
+import com.datastax.driver.core.Session;
+import com.datastax.driver.core.SocketOptions;
+import com.datastax.driver.core.TableMetadata;
+import com.datastax.driver.core.TypeCodec;
+import com.datastax.driver.core.policies.ConstantReconnectionPolicy;
+import com.datastax.driver.core.policies.DCAwareRoundRobinPolicy;
+import com.datastax.driver.core.policies.DefaultRetryPolicy;
+import com.datastax.driver.core.policies.DowngradingConsistencyRetryPolicy;
+import com.datastax.driver.core.policies.ExponentialReconnectionPolicy;
+import com.datastax.driver.core.policies.FallthroughRetryPolicy;
+import com.datastax.driver.core.policies.LatencyAwarePolicy;
+import com.datastax.driver.core.policies.LoggingRetryPolicy;
+import com.datastax.driver.core.policies.RoundRobinPolicy;
+import com.datastax.driver.core.policies.TokenAwarePolicy;
+import com.datastax.driver.mapping.Mapper;
+import com.datastax.driver.mapping.MappingManager;
+import org.apache.gora.cassandra.bean.CassandraKey;
+import org.apache.gora.cassandra.bean.ClusterKeyField;
+import org.apache.gora.cassandra.bean.Field;
+import org.apache.gora.cassandra.bean.KeySpace;
+import org.apache.gora.cassandra.bean.PartitionKeyField;
+import org.apache.gora.persistency.BeanFactory;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.store.DataStoreFactory;
+import org.jdom.Attribute;
+import org.jdom.Document;
+import org.jdom.Element;
+import org.jdom.JDOMException;
+import org.jdom.input.SAXBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Locale;
+import java.util.Properties;
+
+/**
+ * Implementation of Cassandra Store.
+ *
+ * @param <K> key class
+ * @param <T> persistent class
+ */
+public class CassandraStore<K, T extends Persistent> implements DataStore<K, T> {
+
+ private static final String DEFAULT_MAPPING_FILE = "gora-cassandra-mapping.xml";
+
+ public static final Logger LOG = LoggerFactory.getLogger(CassandraStore.class);
+
+ private BeanFactory<K, T> beanFactory;
+
+ private Cluster cluster;
+
+ private Class keyClass;
+
+ private Class persistentClass;
+
+ private CassandraMapping mapping;
+
+ private boolean isUseNativeSerialization;
+
+ private Mapper<T> mapper;
+
+ private Session session;
+
+ public CassandraStore() {
+ super();
+ }
+
+ /**
+ * In initializing the cassandra datastore, read the mapping file, creates the basic connection to cassandra cluster,
+ * according to the gora properties
+ *
+ * @param keyClass key class
+ * @param persistentClass persistent class
+ * @param properties properties
+ */
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) {
+ LOG.debug("Initializing Cassandra store");
+ try {
+ this.keyClass = keyClass;
+ this.persistentClass = persistentClass;
+ String mappingFile = DataStoreFactory.getMappingFile(properties, this, DEFAULT_MAPPING_FILE);
+ List<String> codecs = readCustomCodec(properties);
+ mapping = readMapping(mappingFile);
+ isUseNativeSerialization = Boolean.parseBoolean(properties.getProperty(CassandraStoreParameters.USE_CASSANDRA_NATIVE_SERIALIZATION));
+ Cluster.Builder builder = Cluster.builder();
+ builder = populateSettings(builder, properties);
+ this.cluster = builder.build();
+ if (codecs != null) {
+ registerCustomCodecs(codecs);
+ }
+ this.session = this.cluster.connect();
+ if (isUseNativeSerialization) {
+ this.createSchema();
+ MappingManager mappingManager = new MappingManager(session);
+ mapper = mappingManager.mapper(persistentClass);
+ }
+
+ } catch (Exception e) {
+ LOG.error("Error while initializing Cassandra store: {}",
+ new Object[]{e.getMessage()});
+ throw new RuntimeException(e);
+ }
+ }
+
+ private void registerCustomCodecs(List<String> codecs) throws Exception {
+ for (String codec : codecs) {
+ this.cluster.getConfiguration().getCodecRegistry().register((TypeCodec<?>) Class.forName(codec).newInstance());
+ }
+ }
+
+ /**
+ * In this method we reads the mapping file and creates the Cassandra Mapping.
+ *
+ * @param filename mapping file name
+ * @return @{@link CassandraMapping}
+ * @throws IOException
+ */
+ private CassandraMapping readMapping(String filename) throws IOException {
+ CassandraMapping map = new CassandraMapping();
--- End diff --
It is lot more clean to move this code to separate mapping builder class.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r130021447
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/serializers/CassandraQueryFactory.java ---
@@ -0,0 +1,701 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.cassandra.serializers;
+
+import com.datastax.driver.core.querybuilder.Delete;
+import com.datastax.driver.core.querybuilder.QueryBuilder;
+import com.datastax.driver.core.querybuilder.Select;
+import com.datastax.driver.core.querybuilder.Update;
+import org.apache.gora.cassandra.bean.CassandraKey;
+import org.apache.gora.cassandra.bean.ClusterKeyField;
+import org.apache.gora.cassandra.bean.Field;
+import org.apache.gora.cassandra.bean.KeySpace;
+import org.apache.gora.cassandra.bean.PartitionKeyField;
+import org.apache.gora.cassandra.query.CassandraQuery;
+import org.apache.gora.cassandra.store.CassandraMapping;
+import org.apache.gora.query.Query;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+
+/**
+ * This class is used create Cassandra Queries.
+ */
+class CassandraQueryFactory {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CassandraQueryFactory.class);
+
+ /**
+ * This method returns the CQL query to create key space.
+ * refer : http://docs.datastax.com/en/cql/3.1/cql/cql_reference/create_keyspace_r.html
+ *
+ * @param mapping Cassandra Mapping {@link CassandraMapping}
+ * @return CQL Query
+ */
+ static String getCreateKeySpaceQuery(CassandraMapping mapping) {
+ KeySpace keySpace = mapping.getKeySpace();
+ StringBuilder stringBuffer = new StringBuilder();
+ stringBuffer.append("CREATE KEYSPACE IF NOT EXISTS ").append(keySpace.getName()).append(" WITH REPLICATION = { 'class' : ");
+ KeySpace.PlacementStrategy placementStrategy = keySpace.getPlacementStrategy();
+ stringBuffer.append("'").append(placementStrategy).append("'").append(", ").append("'");
+ switch (placementStrategy) {
+ case SimpleStrategy:
+ stringBuffer.append("replication_factor").append("'").append(" : ").append(keySpace.getReplicationFactor()).append(" }");
+ break;
+ case NetworkTopologyStrategy:
+ boolean isCommaNeeded = false;
+ for (Map.Entry<String, Integer> entry : keySpace.getDataCenters().entrySet()) {
+ if (isCommaNeeded) {
+ stringBuffer.append(", '");
+ }
+ stringBuffer.append(entry.getKey()).append("'").append(" : ").append(entry.getValue());
+ isCommaNeeded = true;
+ }
+ stringBuffer.append(" }");
+ break;
+ }
+
+ if (keySpace.isDurableWritesEnabled()) {
+ stringBuffer.append(" AND DURABLE_WRITES = ").append(keySpace.isDurableWritesEnabled());
+ }
+ return stringBuffer.toString();
+ }
+
+ /**
+ * This method returns the CQL query to table.
+ * refer : http://docs.datastax.com/en/cql/3.1/cql/cql_reference/create_table_r.html
+ * <p>
+ * Trick : To have a consistency of the order of the columns, first we append partition keys, second cluster keys and finally other columns.
+ * It's very much needed to follow the same order in other CRUD operations as well.
+ *
+ * @param mapping Cassandra mapping {@link CassandraMapping}
+ * @return CQL Query
+ */
+ static String getCreateTableQuery(CassandraMapping mapping) {
+ StringBuilder stringBuffer = new StringBuilder();
+ stringBuffer.append("CREATE TABLE IF NOT EXISTS ").append(mapping.getKeySpace().getName()).append(".").append(mapping.getCoreName()).append(" (");
+ boolean isCommaNeeded = false;
+ CassandraKey cassandraKey = mapping.getCassandraKey();
+ // appending Cassandra key columns into db schema
+ for (Field field : mapping.getFieldList()) {
+ if (isCommaNeeded) {
+ stringBuffer.append(", ");
+ }
+ stringBuffer.append(field.getColumnName()).append(" ").append(field.getType());
+ boolean isStaticColumn = Boolean.parseBoolean(field.getProperty("static"));
+ boolean isPrimaryKey = Boolean.parseBoolean(field.getProperty("primarykey"));
+ if (isStaticColumn) {
+ stringBuffer.append(" STATIC");
+ }
+ if (isPrimaryKey) {
+ stringBuffer.append(" PRIMARY KEY ");
+ }
+ isCommaNeeded = true;
+ }
+
+ if (cassandraKey != null) {
+ List<PartitionKeyField> pkey = cassandraKey.getPartitionKeyFields();
+ if (pkey != null) {
+ stringBuffer.append(", PRIMARY KEY (");
+ boolean isCommaNeededToApply = false;
+ for (PartitionKeyField keyField : pkey) {
+ if (isCommaNeededToApply) {
+ stringBuffer.append(",");
+ }
+ if (keyField.isComposite()) {
+ stringBuffer.append("(");
+ boolean isCommaNeededHere = false;
+ for (Field field : keyField.getFields()) {
+ if (isCommaNeededHere) {
+ stringBuffer.append(", ");
+ }
+ stringBuffer.append(field.getColumnName());
+ isCommaNeededHere = true;
+ }
+ stringBuffer.append(")");
+ } else {
+ stringBuffer.append(keyField.getColumnName());
+ }
+ isCommaNeededToApply = true;
+ }
+ stringBuffer.append(")");
+ }
+ } else {
+ if (!stringBuffer.toString().toLowerCase(Locale.ENGLISH).contains("primary key")) {
+ Field field = mapping.getDefaultCassandraKey();
+ stringBuffer.append(", ").append(field.getFieldName()).append(" ").append(field.getType()).append(" PRIMARY KEY ");
+ }
+ }
+
+ stringBuffer.append(")");
+ boolean isWithNeeded = true;
+ if (Boolean.parseBoolean(mapping.getProperty("compactStorage"))) {
+ stringBuffer.append(" WITH COMPACT STORAGE ");
+ isWithNeeded = false;
+ }
+
+ String id = mapping.getProperty("id");
+ if (id != null) {
+ if (isWithNeeded) {
+ stringBuffer.append(" WITH ");
+ } else {
+ stringBuffer.append(" AND ");
+ }
+ stringBuffer.append("ID = '").append(id).append("'");
+ isWithNeeded = false;
+ }
+ if (cassandraKey != null) {
+ List<ClusterKeyField> clusterKeyFields = cassandraKey.getClusterKeyFields();
+ if (clusterKeyFields != null) {
+ if (isWithNeeded) {
+ stringBuffer.append(" WITH ");
+ } else {
+ stringBuffer.append(" AND ");
+ }
+ stringBuffer.append(" CLUSTERING ORDER BY (");
+ boolean isCommaNeededToApply = false;
+ for (ClusterKeyField keyField : clusterKeyFields) {
+ if (isCommaNeededToApply) {
+ stringBuffer.append(", ");
+ }
+ stringBuffer.append(keyField.getColumnName()).append(" ");
+ if (keyField.getOrder() != null) {
+ stringBuffer.append(keyField.getOrder());
+ }
+ isCommaNeededToApply = true;
+ }
+ stringBuffer.append(")");
+ }
+ }
+ return stringBuffer.toString();
+ }
+
+ /**
+ * This method returns the CQL query to drop table.
+ * refer : http://docs.datastax.com/en/cql/3.1/cql/cql_reference/drop_table_r.html
+ *
+ * @param mapping Cassandra Mapping {@link CassandraMapping}
+ * @return CQL query
+ */
+ static String getDropTableQuery(CassandraMapping mapping) {
+ return "DROP TABLE IF EXISTS " + mapping.getKeySpace().getName() + "." + mapping.getCoreName();
+ }
+
+ /**
+ * This method returns the CQL query to drop key space.
+ * refer : http://docs.datastax.com/en/cql/3.1/cql/cql_reference/drop_keyspace_r.html
+ *
+ * @param mapping Cassandra Mapping {@link CassandraMapping}
+ * @return CQL query
+ */
+ static String getDropKeySpaceQuery(CassandraMapping mapping) {
+ return "DROP KEYSPACE IF EXISTS " + mapping.getKeySpace().getName();
+ }
+
+ /**
+ * This method returns the CQL query to truncate (removes all the data) in the table.
+ * refer : http://docs.datastax.com/en/cql/3.1/cql/cql_reference/truncate_r.html
+ *
+ * @param mapping Cassandra Mapping {@link CassandraMapping}
+ * @return CQL query
+ */
+ static String getTruncateTableQuery(CassandraMapping mapping) {
+ return "TRUNCATE TABLE " + mapping.getKeySpace().getName() + "." + mapping.getCoreName();
+ }
+
+ /**
+ * This method return the CQL query to insert data in to the table.
+ * refer : http://docs.datastax.com/en/cql/3.1/cql/cql_reference/insert_r.html
+ *
+ * @param mapping Cassandra Mapping {@link CassandraMapping}
+ * @param fields available fields
+ * @return CQL Query
+ */
+ static String getInsertDataQuery(CassandraMapping mapping, List<String> fields) {
+ String[] columnNames = getColumnNames(mapping, fields);
+ String[] objects = new String[fields.size()];
+ Arrays.fill(objects, "?");
+ return QueryBuilder.insertInto(mapping.getKeySpace().getName(), mapping.getCoreName()).values(columnNames, objects).getQueryString();
+ }
+
+ /**
+ * This method return the CQL query to delete a persistent in the table.
+ * refer : http://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlDelete.html
+ *
+ * @param mapping Cassandra Mapping {@link CassandraMapping}
+ * @param fields filed list to be deleted
+ * @return CQL Query
+ */
+ static String getDeleteDataQuery(CassandraMapping mapping, List<String> fields) {
+ String[] columnNames = getColumnNames(mapping, fields);
+ String[] objects = new String[fields.size()];
+ Arrays.fill(objects, "?");
+ Delete delete = QueryBuilder.delete().from(mapping.getKeySpace().getName(), mapping.getCoreName());
+ Delete.Where query = null;
+ boolean isWhereNeeded = true;
+ for (String columnName : columnNames) {
+ if (isWhereNeeded) {
+ query = delete.where(QueryBuilder.eq(columnName, "?"));
+ isWhereNeeded = false;
+ } else {
+ query = query.and(QueryBuilder.eq(columnName, "?"));
+ }
+ }
+ return query.getQueryString();
+ }
+
+ /**
+ * This method returns the CQL Select query to retrieve data from the table.
+ * refer: http://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlSelect.html
+ *
+ * @param mapping Cassandra Mapping {@link CassandraMapping}
+ * @param keyFields key fields
+ * @return CQL Query
+ */
+ static String getSelectObjectQuery(CassandraMapping mapping, List<String> keyFields) {
+ Select select = QueryBuilder.select().from(mapping.getKeySpace().getName(), mapping.getCoreName());
+ if (Boolean.parseBoolean(mapping.getProperty("allowFiltering"))) {
+ select.allowFiltering();
+ }
+ String[] columnNames = getColumnNames(mapping, keyFields);
+ Select.Where query = null;
+ boolean isWhereNeeded = true;
+ for (String columnName : columnNames) {
+ if (isWhereNeeded) {
+ query = select.where(QueryBuilder.eq(columnName, "?"));
+ isWhereNeeded = false;
+ } else {
+ query = query.and(QueryBuilder.eq(columnName, "?"));
+ }
+ }
+ return query.getQueryString();
+ }
+
+ /**
+ * This method returns CQL Select query to retrieve data from the table with given fields.
+ * This method is used for Avro Serialization
+ * refer: http://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlSelect.html
+ *
+ * @param mapping Cassandra Mapping {@link CassandraMapping}
+ * @param fields Given fields to retrieve
+ * @param keyFields key fields
+ * @return CQL Query
+ */
+ static String getSelectObjectWithFieldsQuery(CassandraMapping mapping, String[] fields, List<String> keyFields) {
+ Select select = QueryBuilder.select(getColumnNames(mapping, Arrays.asList(fields))).from(mapping.getKeySpace().getName(), mapping.getCoreName());
+ if (Boolean.parseBoolean(mapping.getProperty("allowFiltering"))) {
+ select.allowFiltering();
+ }
+ String[] columnNames = getColumnNames(mapping, keyFields);
+ Select.Where query = null;
+ boolean isWhereNeeded = true;
+ for (String columnName : columnNames) {
+ if (isWhereNeeded) {
+ query = select.where(QueryBuilder.eq(columnName, "?"));
+ isWhereNeeded = false;
+ } else {
+ query = query.and(QueryBuilder.eq(columnName, "?"));
+ }
+ }
+ return query.getQueryString();
+ }
+
+ /**
+ * This method returns CQL Select query to retrieve data from the table with given fields.
+ * This method is used for Native Serialization
+ * refer: http://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlSelect.html
+ *
+ * @param mapping Cassandra Mapping {@link CassandraMapping}
+ * @param fields Given fields to retrieve
+ * @return CQL Query
+ */
+ static String getSelectObjectWithFieldsQuery(CassandraMapping mapping, String[] fields) {
+ String cqlQuery = null;
+ String[] columnNames = getColumnNames(mapping, Arrays.asList(fields));
+ Select select = QueryBuilder.select(columnNames).from(mapping.getKeySpace().getName(), mapping.getCoreName());
+ if (Boolean.parseBoolean(mapping.getProperty("allowFiltering"))) {
+ select.allowFiltering();
+ }
+ CassandraKey cKey = mapping.getCassandraKey();
+ if (cKey != null) {
+ Select.Where query = null;
+ boolean isWhereNeeded = true;
+ for (Field field : cKey.getFieldList()) {
+ if (isWhereNeeded) {
+ query = select.where(QueryBuilder.eq(field.getColumnName(), "?"));
+ isWhereNeeded = false;
+ } else {
+ query = query.and(QueryBuilder.eq(field.getColumnName(), "?"));
+ }
+ }
+ cqlQuery = query != null ? query.getQueryString() : null;
+ } else {
+ for (Field field : mapping.getFieldList()) {
+ boolean isPrimaryKey = Boolean.parseBoolean(field.getProperty("primarykey"));
+ if (isPrimaryKey) {
+ cqlQuery = select.where(QueryBuilder.eq(field.getColumnName(), "?")).getQueryString();
+ break;
+ }
+ }
+ }
+ return cqlQuery;
+ }
+
+
+ /**
+ * This method returns CQL Query for execute method. This CQL contains a Select Query to retrieve data from the table
+ *
+ * @param mapping Cassandra Mapping {@link CassandraMapping}
+ * @param cassandraQuery Query {@link CassandraQuery}
+ * @param objects object list
+ * @return CQL Query
+ */
+ static String getExecuteQuery(CassandraMapping mapping, Query cassandraQuery, List<Object> objects) {
+ String[] fields = cassandraQuery.getFields();
+ fields = fields != null ? fields : mapping.getFieldNames();
+ Object startKey = cassandraQuery.getStartKey();
+ Object endKey = cassandraQuery.getEndKey();
+ Object key = cassandraQuery.getKey();
+ String primaryKey = null;
+ long limit = cassandraQuery.getLimit();
+ Select select = QueryBuilder.select(getColumnNames(mapping, Arrays.asList(fields))).from(mapping.getKeySpace().getName(), mapping.getCoreName());
+ if (limit > 0) {
+ select = select.limit((int) limit);
+ }
+ if (Boolean.parseBoolean(mapping.getProperty("allowFiltering"))) {
+ select.allowFiltering();
+ }
+ Select.Where query = null;
+ boolean isWhereNeeded = true;
+ if (key != null) {
+ if (mapping.getCassandraKey() != null) {
+ ArrayList<String> cassandraKeys = new ArrayList<>();
--- End diff --
For three cases key != null, startKey != null, endKey != null, I see very similar code is duplicated across 3 cases. Please see whether we can generalize and utility method that can be reused all three cases. Similar duplicated logic can bee seen in methods - getExecuteQuery, getDeleteByQuery, getUpdateByQuery.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by madhawa-gunasekara <gi...@git.apache.org>.
Github user madhawa-gunasekara commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r123820206
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/store/TestCassandraStoreWithNativeSerialization.java ---
@@ -26,15 +26,20 @@ public static void setUpClass() throws Exception {
setProperties();
testDriver.setParameters(parameter);
testDriver.setUpClass();
+ // This time is added since embedded cassandra service needs some time to start up.
--- End diff --
I will try to fix this :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by madhawa-gunasekara <gi...@git.apache.org>.
Github user madhawa-gunasekara commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r134843337
--- Diff: gora-cassandra-cql/src/examples/java/org/apache/gora/cassandra/example/generated/nativeSerialization/ComplexTypes.java ---
@@ -0,0 +1,102 @@
+package org.apache.gora.cassandra.example.generated.nativeSerialization;
--- End diff --
done :+1:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r132823832
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/compiler/GoraCassandraNativeCompiler.java ---
@@ -0,0 +1,319 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.compiler;
+
+import org.apache.commons.io.FilenameUtils;
+import org.apache.gora.cassandra.bean.Field;
+import org.apache.gora.cassandra.store.CassandraMapping;
+import org.apache.gora.cassandra.store.CassandraMappingBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.Writer;
+import java.nio.charset.Charset;
+import java.util.List;
+import java.util.Locale;
+
+public class GoraCassandraNativeCompiler {
--- End diff --
Please add java doc to the class.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r130024911
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/bean/KeySpace.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.bean;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * This class represents the Cassandra Keyspace.
+ */
+public class KeySpace {
+
+ public enum PlacementStrategy {
--- End diff --
Please document these enum fields - SimpleStrategy, NetworkTopologyStrategy
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by madhawa-gunasekara <gi...@git.apache.org>.
Github user madhawa-gunasekara commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r123885507
--- Diff: gora-cassandra-cql/pom.xml ---
@@ -0,0 +1,230 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora</artifactId>
+ <version>0.8-SNAPSHOT</version>
+ <relativePath>../</relativePath>
+ </parent>
+ <artifactId>gora-cassandra-cql</artifactId>
+ <packaging>bundle</packaging>
+
+ <name>Apache Gora :: Cassandra - CQL</name>
+ <url>http://gora.apache.org</url>
+ <description>The Apache Gora open source framework provides an in-memory data model and
+ persistence for big data. Gora supports persisting to column stores, key value stores,
+ document stores and RDBMSs, and analyzing the data with extensive Apache Hadoop MapReduce
+ support.</description>
+ <inceptionYear>2010</inceptionYear>
+ <organization>
+ <name>The Apache Software Foundation</name>
+ <url>http://www.apache.org/</url>
+ </organization>
+ <issueManagement>
+ <system>JIRA</system>
+ <url>https://issues.apache.org/jira/browse/GORA</url>
+ </issueManagement>
+ <ciManagement>
+ <system>Jenkins</system>
+ <url>https://builds.apache.org/job/Gora-trunk/</url>
+ </ciManagement>
+
+ <properties>
+ <osgi.import>*</osgi.import>
+ <osgi.export>org.apache.gora.cassandra*;version="${project.version}";-noimport:=true</osgi.export>
+ </properties>
+
+ <build>
+ <directory>target</directory>
+ <outputDirectory>target/classes</outputDirectory>
+ <finalName>${project.artifactId}-${project.version}</finalName>
+ <testOutputDirectory>target/test-classes</testOutputDirectory>
+ <testSourceDirectory>src/test/java</testSourceDirectory>
+ <sourceDirectory>src/main/java</sourceDirectory>
+ <testResources>
+ <testResource>
+ <directory>${project.basedir}/src/test/conf</directory>
+ <includes>
+ <include>**/*</include>
+ </includes>
+ <!--targetPath>${project.basedir}/target/classes/</targetPath-->
+ </testResource>
+ </testResources>
+ <plugins>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>build-helper-maven-plugin</artifactId>
+ <version>${build-helper-maven-plugin.version}</version>
+ <executions>
+ <execution>
+ <phase>generate-sources</phase>
+ <goals>
+ <goal>add-source</goal>
+ </goals>
+ <configuration>
+ <sources>
+ <source>src/examples/java</source>
+ </sources>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+
+ <dependencies>
+
+ <!-- OSX JDK 7 -->
+ <!-- should be removed once fixed in Cassandra -->
+ <dependency>
+ <groupId>org.xerial.snappy</groupId>
+ <artifactId>snappy-java</artifactId>
+ <version>1.0.5-M3</version>
+ <scope>test</scope>
+ </dependency>
+
+ <!-- Gora Internal Dependencies -->
+ <dependency>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora-core</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora-core</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+
+ <!-- Cassandra Driver Dependencies -->
+ <dependency>
+ <groupId>com.datastax.cassandra</groupId>
+ <artifactId>cassandra-driver-core</artifactId>
+ <version>${cassandra-driver.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <!-- Cassandra Dependencies -->
+ <dependency>
+ <groupId>org.apache.cassandra</groupId>
+ <artifactId>cassandra-all</artifactId>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.cassandra.deps</groupId>
+ <artifactId>avro</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-log4j12</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>log4j-over-slf4j</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <!-- dependency>
+ <groupId>org.apache.cassandra</groupId>
+ <artifactId>cassandra-thrift</artifactId>
+ </dependency-->
+
+ <!--<dependency>-->
+ <!--<groupId>org.hectorclient</groupId>-->
+ <!--<artifactId>hector-core</artifactId>-->
+ <!--<exclusions>-->
+ <!--<exclusion>-->
+ <!--<groupId>org.apache.cassandra</groupId>-->
+ <!--<artifactId>cassandra-all</artifactId>-->
+ <!--</exclusion>-->
+ <!--</exclusions>-->
+ <!--</dependency>-->
+
+ <!-- Misc Dependencies -->
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.jdom</groupId>
+ <artifactId>jdom</artifactId>
+ </dependency>
+
--- End diff --
Removed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by madhawa-gunasekara <gi...@git.apache.org>.
Github user madhawa-gunasekara commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r134844397
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/serializers/AvroCassandraUtils.java ---
@@ -0,0 +1,274 @@
+package org.apache.gora.cassandra.serializers;
--- End diff --
done :+1:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora issue #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by madhawa-gunasekara <gi...@git.apache.org>.
Github user madhawa-gunasekara commented on the issue:
https://github.com/apache/gora/pull/110
Hi @renato2099
Thanks for the comments. I'm using Persistent Class for the avro serialization :) because there are some limitations in the Cassandra native serializations, like compound keys, composite keys don't support in native serialization [https://datastax-oss.atlassian.net/browse/JAVA-904](1) therefore, I'm going support that using avro serialization by doing the mapping manually. I'm not using Cassandra Native Persistent for avro, It's only for native serialization. I followed your architecture :)
I will be able to complete some methods before this weekend.
Thanks,
Madhawa
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r130064480
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/store/TestCassandraStoreWithNativeSerialization.java ---
@@ -0,0 +1,305 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.store;
+
+import org.apache.gora.cassandra.GoraCassandraTestDriver;
+import org.apache.gora.cassandra.example.generated.nativeSerialization.ComplexTypes;
+import org.apache.gora.cassandra.example.generated.nativeSerialization.User;
+import org.apache.gora.cassandra.query.CassandraQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.util.GoraException;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+
+/**
+ * This class tests Cassandra Store functionality with Cassandra Native Serialization.
+ */
+public class TestCassandraStoreWithNativeSerialization {
--- End diff --
It seems current branch does not include any tests for AVRO serialization. Please work on covering Gora datastore base test case for AVRO serialization.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r130027402
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/GoraCassandraTestDriver.java ---
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * @author lewismc
+ *
+ */
+
+package org.apache.gora.cassandra;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.apache.cassandra.io.util.FileUtils;
+import org.apache.cassandra.service.CassandraDaemon;
+import org.apache.gora.GoraTestDriver;
+import org.apache.gora.cassandra.store.CassandraStore;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.store.DataStoreFactory;
+import org.apache.gora.util.GoraException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.util.Properties;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+
+// Logging imports
+
+/**
+ * Helper class for third party tests using gora-cassandra backend.
+ * @see GoraTestDriver for test specifics.
+ * This driver is the base for all test cases that require an embedded Cassandra
+ * server. In this case we draw on Hector's @see EmbeddedServerHelper.
+ * It starts (setUp) and stops (tearDown) embedded Cassandra server.
+ *
+ */
+
+public class GoraCassandraTestDriver extends GoraTestDriver {
+ private static Logger log = LoggerFactory.getLogger(GoraCassandraTestDriver.class);
+
+ private static String baseDirectory = "target/test";
+
+ private CassandraDaemon cassandraDaemon;
+
+ private Thread cassandraThread;
+
+ private Properties properties;
+
+ public void setParameters(Properties parameters) {
+ this.properties = parameters;
+ }
+
+ @Override
+ public <K, T extends Persistent> DataStore<K, T> createDataStore(Class<K> keyClass, Class<T> persistentClass) throws GoraException {
+ return DataStoreFactory.createDataStore(CassandraStore.class, keyClass, persistentClass , conf, properties, null);
+ }
+
+ /**
+ * @return temporary base directory of running cassandra instance
+ */
+ public String getBaseDirectory() {
+ return baseDirectory;
+ }
+
+ public GoraCassandraTestDriver() {
+ super(CassandraStore.class);
+ }
+
+ /**
+ * Starts embedded Cassandra server.
+ *
+ * @throws Exception
+ * if an error occurs
+ */
+ @Override
+ public void setUpClass(){
+ log.info("Starting embedded Cassandra Server...");
+ try {
+ cleanupDirectoriesFailover();
+ FileUtils.createDirectory(baseDirectory);
+ System.setProperty("log4j.configuration", "log4j-server.properties");
+ System.setProperty("cassandra.config", "cassandra.yaml");
+
+ cassandraDaemon = new CassandraDaemon();
+ cassandraDaemon.completeSetup();
+ cassandraDaemon.applyConfig();
+ cassandraDaemon.init(null);
+ cassandraThread = new Thread(new Runnable() {
+
+ public void run() {
+ try {
+ cassandraDaemon.start();
+ } catch (Exception e) {
+ log.error("Embedded casandra server run failed!", e);
+ }
+ }
+ });
+
+ cassandraThread.setDaemon(true);
+ cassandraThread.start();
+ } catch (Exception e) {
+ log.error("Embedded casandra server start failed!", e);
+
+ // cleanup
+ tearDownClass();
+ }
+ }
+
+
+ /**
+ * Stops embedded Cassandra server.
+ *
+ * @throws Exception
+ * if an error occurs
+ */
+ @Override
+ public void tearDownClass(){
--- End diff --
Seems this is not formatted properly - tearDownClass() {
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: Initial commit gora cassandra cql
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r122807843
--- Diff: gora-cassandra-cql/pom.xml ---
@@ -0,0 +1,230 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <!--
--- End diff --
Please remove the double license.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r132824702
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/GoraCassandraTestDriver.java ---
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * @author lewismc
+ */
+
+package org.apache.gora.cassandra;
+
+import org.apache.cassandra.io.util.FileUtils;
+import org.apache.cassandra.service.CassandraDaemon;
+import org.apache.gora.GoraTestDriver;
+import org.apache.gora.cassandra.store.CassandraStore;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.store.DataStoreFactory;
+import org.apache.gora.util.GoraException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.util.Properties;
+
+// Logging imports
+
+/**
+ * Helper class for third party tests using gora-cassandra backend.
+ *
+ * @see GoraTestDriver for test specifics.
+ * This driver is the base for all test cases that require an embedded Cassandra
+ * server. In this case we draw on Hector's @see EmbeddedServerHelper.
+ * It starts (setUp) and stops (tearDown) embedded Cassandra server.
+ */
+
--- End diff --
Better if we can remove this unnecessary new lines.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r124048279
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/test/nativeSerialization/User.java ---
@@ -0,0 +1,66 @@
+package org.apache.gora.cassandra.test.nativeSerialization;
--- End diff --
Please add License header here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by madhawa-gunasekara <gi...@git.apache.org>.
Github user madhawa-gunasekara commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r123885498
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/store/CassandraQueryFactory.java ---
@@ -0,0 +1,4 @@
+package org.apache.gora.cassandra.store;
--- End diff --
License header added.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r130017932
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/serializers/AvroCassandraUtils.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.serializers;
+
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificData;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.cassandra.bean.CassandraKey;
+import org.apache.gora.cassandra.bean.Field;
+import org.apache.gora.cassandra.store.CassandraMapping;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.DirtyListWrapper;
+import org.apache.gora.persistency.impl.DirtyMapWrapper;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * This class is Utils class for Avro serialization.
+ */
+class AvroCassandraUtils {
+
+ /**
+ * Default schema index with value "0" used when AVRO Union data types are stored.
+ */
+ private static final int DEFAULT_UNION_SCHEMA = 0;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AvroCassandraUtils.class);
+
+ static void processKeys(CassandraMapping cassandraMapping, Object key, List<String> keys, List<Object> values) {
+ CassandraKey cassandraKey = cassandraMapping.getCassandraKey();
+ if (cassandraMapping.isPartitionKeyDefined()) {
+ if (cassandraKey != null) {
+ if (key instanceof PersistentBase) {
+ PersistentBase keyBase = (PersistentBase) key;
+ for (Schema.Field field : keyBase.getSchema().getFields()) {
+ if (cassandraMapping.getFieldFromFieldName(field.name()) != null) {
+ keys.add(field.name());
+ Object value = keyBase.get(field.pos());
+ value = getFieldValueFromAvroBean(field.schema(), field.schema().getType(), value);
+ values.add(value);
+ } else {
+ LOG.debug("Ignoring field {}, Since field couldn't find in the {} mapping", new Object[]{field.name(), cassandraMapping.getPersistentClass()});
+ }
+ }
+ } else {
+ LOG.error("Key bean isn't extended by {} .", new Object[]{cassandraMapping.getKeyClass(), PersistentBase.class});
+ }
+ } else {
+ for (Field field : cassandraMapping.getInlinedDefinedPartitionKeys()) {
+ keys.add(field.getFieldName());
+ values.add(key);
+ }
+ }
+ } else {
+ keys.add(cassandraMapping.getDefaultCassandraKey().getFieldName());
+ values.add(key.toString());
+ }
+ }
+
+ /**
+ * For every field within an object, we pass in a field schema, Type and value.
+ * This enables us to process fields (based on their characteristics)
+ * preparing them for persistence.
+ *
+ * @param fieldSchema the associated field schema
+ * @param type the field type
+ * @param fieldValue the field value.
+ * @return field value
+ */
+ static Object getFieldValueFromAvroBean(Schema fieldSchema, Schema.Type type, Object fieldValue) {
+ switch (type) {
+ case RECORD:
+ PersistentBase persistent = (PersistentBase) fieldValue;
+ PersistentBase newRecord = (PersistentBase) SpecificData.get().newRecord(persistent, persistent.getSchema());
+ for (Schema.Field member : fieldSchema.getFields()) {
+ if (member.pos() == 0 || !persistent.isDirty()) {
+ continue;
+ }
+ Schema memberSchema = member.schema();
+ Schema.Type memberType = memberSchema.getType();
+ Object memberValue = persistent.get(member.pos());
+ newRecord.put(member.pos(), getFieldValueFromAvroBean(memberSchema, memberType, memberValue));
+ }
+ fieldValue = newRecord;
+ break;
+ case MAP:
+ Schema valueSchema = fieldSchema.getValueType();
+ Schema.Type valuetype = valueSchema.getType();
+ HashMap<String, Object> map = new HashMap<>();
+ for (Map.Entry<CharSequence, ?> e : ((Map<CharSequence, ?>) fieldValue).entrySet()) {
+ String mapKey = e.getKey().toString();
+ Object mapValue = e.getValue();
+ mapValue = getFieldValueFromAvroBean(valueSchema, valuetype, mapValue);
+ map.put(mapKey, mapValue);
+ }
+ fieldValue = map;
+ break;
+ case ARRAY:
+ valueSchema = fieldSchema.getElementType();
+ valuetype = valueSchema.getType();
+ ArrayList<Object> list = new ArrayList<>();
+ for (Object item : (Collection<?>) fieldValue) {
+ Object value = getFieldValueFromAvroBean(valueSchema, valuetype, item);
+ list.add(value);
+ }
+ fieldValue = list;
+ break;
+ case UNION:
+ // storing the union selected schema, the actual value will
+ // be stored as soon as we get break out.
+ if (fieldValue != null) {
+ int schemaPos = getUnionSchema(fieldValue, fieldSchema);
+ Schema unionSchema = fieldSchema.getTypes().get(schemaPos);
+ Schema.Type unionType = unionSchema.getType();
+ fieldValue = getFieldValueFromAvroBean(unionSchema, unionType, fieldValue);
+ }
+ break;
+ case STRING:
+ fieldValue = fieldValue.toString();
+ break;
+ default:
+ break;
+ }
+ return fieldValue;
+ }
+
+ /**
+ * Given an object and the object schema this function obtains,
+ * from within the UNION schema, the position of the type used.
+ * If no data type can be inferred then we return a default value
+ * of position 0.
+ *
+ * @param pValue Object
+ * @param pUnionSchema avro Schema
+ * @return the unionSchemaPosition.
+ */
+ private static int getUnionSchema(Object pValue, Schema pUnionSchema) {
+ int unionSchemaPos = 0;
+// String valueType = pValue.getClass().getSimpleName();
+ for (Schema currentSchema : pUnionSchema.getTypes()) {
+ Schema.Type schemaType = currentSchema.getType();
+ if (pValue instanceof CharSequence && schemaType.equals(Schema.Type.STRING))
+ return unionSchemaPos;
+ else if (pValue instanceof ByteBuffer && schemaType.equals(Schema.Type.BYTES))
+ return unionSchemaPos;
+ else if (pValue instanceof Integer && schemaType.equals(Schema.Type.INT))
+ return unionSchemaPos;
+ else if (pValue instanceof Long && schemaType.equals(Schema.Type.LONG))
+ return unionSchemaPos;
+ else if (pValue instanceof Double && schemaType.equals(Schema.Type.DOUBLE))
+ return unionSchemaPos;
+ else if (pValue instanceof Float && schemaType.equals(Schema.Type.FLOAT))
+ return unionSchemaPos;
+ else if (pValue instanceof Boolean && schemaType.equals(Schema.Type.BOOLEAN))
+ return unionSchemaPos;
+ else if (pValue instanceof Map && schemaType.equals(Schema.Type.MAP))
+ return unionSchemaPos;
+ else if (pValue instanceof List && schemaType.equals(Schema.Type.ARRAY))
+ return unionSchemaPos;
+ else if (pValue instanceof Persistent && schemaType.equals(Schema.Type.RECORD))
+ return unionSchemaPos;
+ unionSchemaPos++;
+ }
+ // if we weren't able to determine which data type it is, then we return the default
+ return DEFAULT_UNION_SCHEMA;
+ }
+
+ static String encodeFieldKey(final String key) {
--- End diff --
This method doesn't have any usages.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r130028052
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/GoraCassandraTestDriver.java ---
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * @author lewismc
+ *
+ */
+
+package org.apache.gora.cassandra;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.apache.cassandra.io.util.FileUtils;
+import org.apache.cassandra.service.CassandraDaemon;
+import org.apache.gora.GoraTestDriver;
+import org.apache.gora.cassandra.store.CassandraStore;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.store.DataStoreFactory;
+import org.apache.gora.util.GoraException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.util.Properties;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+
+// Logging imports
+
+/**
+ * Helper class for third party tests using gora-cassandra backend.
+ * @see GoraTestDriver for test specifics.
+ * This driver is the base for all test cases that require an embedded Cassandra
+ * server. In this case we draw on Hector's @see EmbeddedServerHelper.
+ * It starts (setUp) and stops (tearDown) embedded Cassandra server.
+ *
+ */
+
+public class GoraCassandraTestDriver extends GoraTestDriver {
+ private static Logger log = LoggerFactory.getLogger(GoraCassandraTestDriver.class);
+
+ private static String baseDirectory = "target/test";
+
+ private CassandraDaemon cassandraDaemon;
+
+ private Thread cassandraThread;
+
+ private Properties properties;
+
+ public void setParameters(Properties parameters) {
+ this.properties = parameters;
+ }
+
+ @Override
+ public <K, T extends Persistent> DataStore<K, T> createDataStore(Class<K> keyClass, Class<T> persistentClass) throws GoraException {
+ return DataStoreFactory.createDataStore(CassandraStore.class, keyClass, persistentClass , conf, properties, null);
+ }
+
+ /**
+ * @return temporary base directory of running cassandra instance
+ */
+ public String getBaseDirectory() {
+ return baseDirectory;
+ }
+
+ public GoraCassandraTestDriver() {
+ super(CassandraStore.class);
+ }
+
+ /**
+ * Starts embedded Cassandra server.
+ *
+ * @throws Exception
+ * if an error occurs
+ */
+ @Override
+ public void setUpClass(){
+ log.info("Starting embedded Cassandra Server...");
+ try {
+ cleanupDirectoriesFailover();
+ FileUtils.createDirectory(baseDirectory);
+ System.setProperty("log4j.configuration", "log4j-server.properties");
+ System.setProperty("cassandra.config", "cassandra.yaml");
+
+ cassandraDaemon = new CassandraDaemon();
+ cassandraDaemon.completeSetup();
+ cassandraDaemon.applyConfig();
+ cassandraDaemon.init(null);
+ cassandraThread = new Thread(new Runnable() {
+
+ public void run() {
+ try {
+ cassandraDaemon.start();
+ } catch (Exception e) {
+ log.error("Embedded casandra server run failed!", e);
+ }
+ }
+ });
+
+ cassandraThread.setDaemon(true);
+ cassandraThread.start();
+ } catch (Exception e) {
+ log.error("Embedded casandra server start failed!", e);
+
+ // cleanup
+ tearDownClass();
+ }
+ }
+
+
+ /**
+ * Stops embedded Cassandra server.
+ *
+ * @throws Exception
+ * if an error occurs
+ */
+ @Override
+ public void tearDownClass(){
+ log.info("Shutting down Embedded Cassandra server...");
+ if (cassandraThread != null) {
+ cassandraDaemon.stop();
+ cassandraDaemon.destroy();
+// cassandraThread.interrupt();
+// cassandraThread = null;
+ }
+/* Thread cleanupThread = new Thread(() -> {
+ try {
+ Thread.sleep(5000);
+ cleanupDirectoriesFailover();
+ } catch (Exception e) {
+ log.error("Embedded casandra server run failed!", e);
+ }
+ });
+ cleanupThread.setDaemon(true);
+ cleanupThread.start();*/
+
+ }
+
+ /**
+ * Cleans up cassandra's temporary base directory.
+ *
+ * In case o failure waits for 250 msecs and then tries it again, 3 times totally.
+ */
+ public static void cleanupDirectoriesFailover() {
+ int tries = 3;
+ while (tries-- > 0) {
+ try {
+ cleanupDirectories();
+ break;
+ } catch (Exception e) {
+ // ignore exception
+ try {
+ Thread.sleep(2500);
--- End diff --
Can you please explain why this Thread.sleep() is used?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r129362202
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/store/CassandraMappingBuilder.java ---
@@ -135,13 +137,16 @@ public CassandraMapping readMapping(String filename) throws IOException {
}
break;
}
- map.setKeySpace(keyspace);
+ cassandraMapping.setKeySpace(keyspace);
break;
}
}
}
+ else {
+ throw new RuntimeException("KeySpace couldn't be able to found in the cassandra mapping. Please configure the cassandra mapping correctly.");
--- End diff --
Please change the following statement ```KeySpace couldn't be able to found in the cassandra mapping. Please configure the cassandra mapping correctly.``` to ```Couldn't find KeySpace in the Cassandra mapping. Please configure the cassandra mapping correctly.```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: Initial commit gora cassandra cql
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r122811683
--- Diff: gora-cassandra-cql/src/test/conf/gora.properties ---
@@ -0,0 +1,30 @@
+#
--- End diff --
Please make sure that all configuration options are documented ans included here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by madhawa-gunasekara <gi...@git.apache.org>.
Github user madhawa-gunasekara commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r134844996
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/store/TestCassandraStoreWithNativeSerialization.java ---
@@ -0,0 +1,305 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.store;
+
+import org.apache.gora.cassandra.GoraCassandraTestDriver;
+import org.apache.gora.cassandra.example.generated.nativeSerialization.ComplexTypes;
+import org.apache.gora.cassandra.example.generated.nativeSerialization.User;
+import org.apache.gora.cassandra.query.CassandraQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.util.GoraException;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+
+/**
+ * This class tests Cassandra Store functionality with Cassandra Native Serialization.
+ */
+public class TestCassandraStoreWithNativeSerialization {
+ private static GoraCassandraTestDriver testDriver = new GoraCassandraTestDriver();
+ private static DataStore<UUID, User> userDataStore;
+ private static Properties parameter;
+
+ @BeforeClass
+ public static void setUpClass() throws Exception {
+ setProperties();
+ testDriver.setParameters(parameter);
+ testDriver.setUpClass();
+ userDataStore = testDriver.createDataStore(UUID.class, User.class);
+ }
+
+ private static void setProperties() {
+ parameter = new Properties();
+ parameter.setProperty(CassandraStoreParameters.CASSANDRA_SERVERS, "localhost");
+ parameter.setProperty(CassandraStoreParameters.PORT, "9042");
+ parameter.setProperty(CassandraStoreParameters.CASSANDRA_SERIALIZATION_TYPE, "native");
+ parameter.setProperty(CassandraStoreParameters.PROTOCOL_VERSION, "3");
+ parameter.setProperty(CassandraStoreParameters.CLUSTER_NAME, "Test Cluster");
+ parameter.setProperty("gora.cassandrastore.mapping.file", "nativeSerialization/gora-cassandra-mapping.xml");
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ testDriver.tearDown();
+ }
+
+ @AfterClass
+ public static void tearDownClass() throws Exception {
+ testDriver.tearDownClass();
+ }
+
+ /**
+ * In this test case, put and get behavior of the data store are testing.
+ */
+ @Test
+ public void testSimplePutAndGet() {
+ UUID id = UUID.randomUUID();
+ User user1 = new User(id, "madhawa", Date.from(Instant.now()));
+ // storing data;
+ userDataStore.put(id, user1);
+ // get data;
+ User olduser = userDataStore.get(id);
+ Assert.assertEquals(olduser.getName(), user1.getName());
+ Assert.assertEquals(olduser.getDateOfBirth(), user1.getDateOfBirth());
+ }
+
+ /**
+ * In this test case, put and delete behavior of the data store are testing.
+ */
+ @Test
+ public void testSimplePutDeleteAndGet() {
+ UUID id = UUID.randomUUID();
+ User user1 = new User(id, "kasun", Date.from(Instant.now()));
+ // storing data;
+ userDataStore.put(id, user1);
+ // get data;
+ User olduser = userDataStore.get(user1.getUserId());
+ Assert.assertEquals(olduser.getName(), user1.getName());
+ Assert.assertEquals(olduser.getDateOfBirth(), user1.getDateOfBirth());
+ // delete data;
+ userDataStore.delete(user1.getUserId());
+ // get data
+ User deletedUser = userDataStore.get(id);
+ Assert.assertNull(deletedUser);
+ }
+
+ /**
+ * In this test case, schema exists method behavior of the data store is testing.
+ */
+ @Test()
+ public void testSchemaExists() {
+ userDataStore.deleteSchema();
+ Assert.assertFalse(userDataStore.schemaExists());
+ userDataStore.createSchema();
+ Assert.assertTrue(userDataStore.schemaExists());
+ }
+
+ /**
+ * In this test case, schema exists method behavior of the data store is testing.
+ */
+ @Test
+ public void testTruncateSchema() {
+ if (!userDataStore.schemaExists()) {
+ userDataStore.createSchema();
+ }
+ UUID id = UUID.randomUUID();
+ User user1 = new User(id, "Madhawa Kasun", Date.from(Instant.now()));
+ userDataStore.put(id, user1);
+ User olduser = userDataStore.get(id);
+ Assert.assertEquals(olduser.getName(), user1.getName());
+ Assert.assertEquals(olduser.getDateOfBirth(), user1.getDateOfBirth());
+ userDataStore.truncateSchema();
+ olduser = userDataStore.get(id);
+ Assert.assertNull(olduser);
+ }
+
+ /**
+ * In this test case, get with fields method behavior of the data store is testing.
+ */
+ @Test
+ public void testGetWithFields() {
+ UUID id = UUID.randomUUID();
+ User user1 = new User(id, "Madhawa Kasun Gunasekara", Date.from(Instant.now()));
+ userDataStore.put(id, user1);
+ // get data;
+ User olduser = userDataStore.get(user1.getUserId());
+ Assert.assertEquals(olduser.getName(), user1.getName());
+ Assert.assertEquals(olduser.getDateOfBirth(), user1.getDateOfBirth());
+ User olduserWithFields = userDataStore.get(id, new String[]{"name"});
+ Assert.assertNull(olduserWithFields.getDateOfBirth());
+ }
+
+ /**
+ * In this test case, get with fields method behavior of the data store is testing.
+ */
+ @Test
+ public void testExecute() throws Exception {
+ userDataStore.truncateSchema();
+ Map<UUID, User> users = new HashMap<>();
+ UUID id1 = UUID.randomUUID();
+ User user1 = new User(id1, "user1", Date.from(Instant.now()));
+ users.put(id1, user1);
+ userDataStore.put(id1, user1);
+ UUID id2 = UUID.randomUUID();
+ User user2 = new User(id2, "user2", Date.from(Instant.now()));
+ users.put(id2, user2);
+ userDataStore.put(id2, user2);
+ UUID id3 = UUID.randomUUID();
+ User user3 = new User(id3, "user3", Date.from(Instant.now()));
+ users.put(id3, user3);
+ userDataStore.put(id3, user3);
+ Query<UUID, User> query1 = userDataStore.newQuery();
+ Result<UUID, User> result1 = userDataStore.execute(query1);
+ int i = 0;
+ Assert.assertEquals(result1.getProgress(),0.0,0.0);
+ while (result1.next()) {
+ // check objects values
+ Assert.assertEquals(result1.get().getName(), users.get(result1.getKey()).getName());
+ Assert.assertEquals(result1.get().getDateOfBirth(), users.get(result1.getKey()).getDateOfBirth());
+ Assert.assertEquals(result1.get().getUserId(), users.get(result1.getKey()).getUserId());
+ i++;
+ }
+ Assert.assertEquals(result1.getProgress(),1.0,0.0);
+ Assert.assertEquals(3, i);
+
+ // Check limit query
+ Query<UUID, User> query2 = userDataStore.newQuery();
+ query2.setLimit(2);
+ Result<UUID, User> result2 = userDataStore.execute(query2);
+ i = 0;
+ while (result2.next()) {
+ Assert.assertEquals(result2.get().getName(), users.get(result2.getKey()).getName());
+ Assert.assertEquals(result2.get().getDateOfBirth(), users.get(result2.getKey()).getDateOfBirth());
+ Assert.assertEquals(result2.get().getUserId(), users.get(result2.getKey()).getUserId());
+ i++;
+ }
+ Assert.assertEquals(2, i);
+
+ // check key element
+ Query<UUID, User> query3 = userDataStore.newQuery();
+ query3.setKey(id1);
+
+ Result<UUID, User> result3 = userDataStore.execute(query3);
+ i = 0;
+ while (result3.next()) {
+ Assert.assertEquals(result3.get().getName(), users.get(result3.getKey()).getName());
+ Assert.assertEquals(result3.get().getDateOfBirth(), users.get(result3.getKey()).getDateOfBirth());
+ Assert.assertEquals(result3.get().getUserId(), users.get(result3.getKey()).getUserId());
+ i++;
+ }
+ Assert.assertEquals(1, i);
+ }
+
+ /**
+ * In this test case, delete by query method behavior of the data store is testing.
+ */
+ @Test
+ public void testDeleteByQuery() throws Exception {
+ userDataStore.truncateSchema();
+ UUID id1 = UUID.randomUUID();
+ User user1 = new User(id1, "user1", Date.from(Instant.now()));
+ userDataStore.put(id1, user1);
+ UUID id2 = UUID.randomUUID();
+ User user2 = new User(id2, "user2", Date.from(Instant.now()));
+ userDataStore.put(id2, user2);
+ Query<UUID, User> query1 = userDataStore.newQuery();
+ query1.setKey(id1);
+ userDataStore.deleteByQuery(query1);
+ User user = userDataStore.get(id1);
+ Assert.assertNull(user);
+
+ //test deleteByFields
+ Query<UUID, User> query2 = userDataStore.newQuery();
+ query2.setKey(id2);
+ query2.setFields("name");
+ userDataStore.deleteByQuery(query2);
+ User partialDeletedUser = userDataStore.get(id2);
+ Assert.assertNull(partialDeletedUser.getName());
+ Assert.assertEquals(partialDeletedUser.getDateOfBirth(),user2.getDateOfBirth());
+ }
+
+ /**
+ * In this test case, update by quert method behavior of the data store is testing.
+ */
+ @Test
+ public void testUpdateByQuery() {
+ userDataStore.truncateSchema();
+ UUID id1 = UUID.randomUUID();
+ User user1 = new User(id1, "user1", Date.from(Instant.now()));
+ userDataStore.put(id1, user1);
+ UUID id2 = UUID.randomUUID();
+ User user2 = new User(id2, "user2", Date.from(Instant.now()));
+ userDataStore.put(id2, user2);
+ Query<UUID, User> query1 = userDataStore.newQuery();
+ if(query1 instanceof CassandraQuery) {
+ ((CassandraQuery) query1).addUpdateField("name", "madhawa");
+ }
+ query1.setKey(id1);
+ if(userDataStore instanceof CassandraStore) {
+ ((CassandraStore) userDataStore).updateByQuery(query1);
+ }
+ User user = userDataStore.get(id1);
+ Assert.assertEquals(user.getName(),"madhawa");
+ }
+
+ @Test
+ public void testComplexTypes() throws GoraException {
+ DataStore<String, ComplexTypes> documentDataStore = testDriver.createDataStore(String.class, ComplexTypes.class);
--- End diff --
added
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: Initial commit gora cassandra cql
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r122811392
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/store/Constants.java ---
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.store;
+
+public class Constants {
+
+ /* string (multiple values with comma separated) */
--- End diff --
Can you make sure all of the fields are documented correctly other than just the type.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r124047919
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/store/TestCassandraStoreWithNativeSerialization.java ---
@@ -0,0 +1,117 @@
+package org.apache.gora.cassandra.store;
--- End diff --
License header please?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r124045668
--- Diff: gora-cassandra-cql/src/examples/java/.gitignore ---
@@ -0,0 +1,15 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
--- End diff --
Is this file added intentionally?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r123814041
--- Diff: gora-cassandra-cql/pom.xml ---
@@ -144,12 +144,28 @@
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
<exclusion>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-handler</artifactId>
+ </exclusion>
+ <exclusion>
<groupId>org.slf4j</groupId>
<artifactId>log4j-over-slf4j</artifactId>
</exclusion>
</exclusions>
</dependency>
+<!-- <dependency>
--- End diff --
Please remove commented out code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by madhawa-gunasekara <gi...@git.apache.org>.
Github user madhawa-gunasekara commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r134844565
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/store/CassandraClient.java ---
@@ -0,0 +1,501 @@
+package org.apache.gora.cassandra.store;
--- End diff --
done :+1:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r129359618
--- Diff: gora-cassandra-cql/src/examples/avro/cassandraKey.json ---
@@ -0,0 +1,21 @@
+{
+ "type": "record",
+ "name": "CassandraKey",
+ "default": null,
+ "namespace": "org.apache.gora.cassandra.example.generated.avroSerialization",
--- End diff --
Please change ```avroSerialization``` to ```AvroSerialization```, please also rename ```cassandraKey.json``` to ```CassandraKey.json```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r132825580
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java ---
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.store;
+
+import org.apache.avro.Schema;
--- End diff --
I think by design, this CassandraStore class should be free from AVRO related imports since you only implement DataStore<K, T> interface.
AVRO related imports should only be resided in AvroSerializer class.
Native datastax related imports should only be resided in NativeSerializer class.
That way CassandraStore is decoupled from AVRO related dependencies.
I went through your code this AVRO import has only been used in one location in persistentSchema class level variable. I think you can easily get rid of this bit of code change, may be you change CassandraSerializer.getSerializer() method by passing BeanFactory<K, T> beanFactory directly rather assigning to class level variable.
I think one this type of design would be start point, one day Gora should be refactored to support pluggable serializers not just AVRO. WDYT @lewismc
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r132824045
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/store/TestCassandraStore.java ---
@@ -0,0 +1,204 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ * <p>
+ * Testing class for all standard gora-cassandra functionality.
+ * We extend DataStoreTestBase enabling us to run the entire base test
+ * suite for Gora.
+ * <p>
+ * Testing class for all standard gora-cassandra functionality.
+ * We extend DataStoreTestBase enabling us to run the entire base test
+ * suite for Gora.
+ * <p>
+ * Testing class for all standard gora-cassandra functionality.
+ * We extend DataStoreTestBase enabling us to run the entire base test
+ * suite for Gora.
+ * <p>
+ * Testing class for all standard gora-cassandra functionality.
+ * We extend DataStoreTestBase enabling us to run the entire base test
+ * suite for Gora.
+ * <p>
+ * Testing class for all standard gora-cassandra functionality.
+ * We extend DataStoreTestBase enabling us to run the entire base test
+ * suite for Gora.
+ * <p>
+ * Testing class for all standard gora-cassandra functionality.
+ * We extend DataStoreTestBase enabling us to run the entire base test
+ * suite for Gora.
+ * <p>
+ * Testing class for all standard gora-cassandra functionality.
+ * We extend DataStoreTestBase enabling us to run the entire base test
+ * suite for Gora.
+ */
+
+/**
+ * Testing class for all standard gora-cassandra functionality.
+ * We extend DataStoreTestBase enabling us to run the entire base test
+ * suite for Gora.
+ */
--- End diff --
May be we should add this as class level java doc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by madhawa-gunasekara <gi...@git.apache.org>.
Github user madhawa-gunasekara commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r123820061
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/GoraCassandraTestDriver.java ---
@@ -148,7 +156,7 @@ public void cleanupDirectoriesFailover() {
} catch (Exception e) {
// ignore exception
try {
- Thread.sleep(250);
+ Thread.sleep(2500);
--- End diff --
Yes this was for debugging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r130016857
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/store/CassandraMappingBuilder.java ---
@@ -0,0 +1,240 @@
+package org.apache.gora.cassandra.store;
--- End diff --
Please add License header here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r124048160
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/test/nativeSerialization/DateAsStringCodec.java ---
@@ -0,0 +1,33 @@
+package org.apache.gora.cassandra.test.nativeSerialization;
--- End diff --
Please add License header here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r123556881
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/store/CassandraQueryFactory.java ---
@@ -0,0 +1,4 @@
+package org.apache.gora.cassandra.store;
--- End diff --
Add license header
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r130017958
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/serializers/AvroCassandraUtils.java ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.serializers;
+
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificData;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.cassandra.bean.CassandraKey;
+import org.apache.gora.cassandra.bean.Field;
+import org.apache.gora.cassandra.store.CassandraMapping;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.DirtyListWrapper;
+import org.apache.gora.persistency.impl.DirtyMapWrapper;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * This class is Utils class for Avro serialization.
+ */
+class AvroCassandraUtils {
+
+ /**
+ * Default schema index with value "0" used when AVRO Union data types are stored.
+ */
+ private static final int DEFAULT_UNION_SCHEMA = 0;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AvroCassandraUtils.class);
+
+ static void processKeys(CassandraMapping cassandraMapping, Object key, List<String> keys, List<Object> values) {
+ CassandraKey cassandraKey = cassandraMapping.getCassandraKey();
+ if (cassandraMapping.isPartitionKeyDefined()) {
+ if (cassandraKey != null) {
+ if (key instanceof PersistentBase) {
+ PersistentBase keyBase = (PersistentBase) key;
+ for (Schema.Field field : keyBase.getSchema().getFields()) {
+ if (cassandraMapping.getFieldFromFieldName(field.name()) != null) {
+ keys.add(field.name());
+ Object value = keyBase.get(field.pos());
+ value = getFieldValueFromAvroBean(field.schema(), field.schema().getType(), value);
+ values.add(value);
+ } else {
+ LOG.debug("Ignoring field {}, Since field couldn't find in the {} mapping", new Object[]{field.name(), cassandraMapping.getPersistentClass()});
+ }
+ }
+ } else {
+ LOG.error("Key bean isn't extended by {} .", new Object[]{cassandraMapping.getKeyClass(), PersistentBase.class});
+ }
+ } else {
+ for (Field field : cassandraMapping.getInlinedDefinedPartitionKeys()) {
+ keys.add(field.getFieldName());
+ values.add(key);
+ }
+ }
+ } else {
+ keys.add(cassandraMapping.getDefaultCassandraKey().getFieldName());
+ values.add(key.toString());
+ }
+ }
+
+ /**
+ * For every field within an object, we pass in a field schema, Type and value.
+ * This enables us to process fields (based on their characteristics)
+ * preparing them for persistence.
+ *
+ * @param fieldSchema the associated field schema
+ * @param type the field type
+ * @param fieldValue the field value.
+ * @return field value
+ */
+ static Object getFieldValueFromAvroBean(Schema fieldSchema, Schema.Type type, Object fieldValue) {
+ switch (type) {
+ case RECORD:
+ PersistentBase persistent = (PersistentBase) fieldValue;
+ PersistentBase newRecord = (PersistentBase) SpecificData.get().newRecord(persistent, persistent.getSchema());
+ for (Schema.Field member : fieldSchema.getFields()) {
+ if (member.pos() == 0 || !persistent.isDirty()) {
+ continue;
+ }
+ Schema memberSchema = member.schema();
+ Schema.Type memberType = memberSchema.getType();
+ Object memberValue = persistent.get(member.pos());
+ newRecord.put(member.pos(), getFieldValueFromAvroBean(memberSchema, memberType, memberValue));
+ }
+ fieldValue = newRecord;
+ break;
+ case MAP:
+ Schema valueSchema = fieldSchema.getValueType();
+ Schema.Type valuetype = valueSchema.getType();
+ HashMap<String, Object> map = new HashMap<>();
+ for (Map.Entry<CharSequence, ?> e : ((Map<CharSequence, ?>) fieldValue).entrySet()) {
+ String mapKey = e.getKey().toString();
+ Object mapValue = e.getValue();
+ mapValue = getFieldValueFromAvroBean(valueSchema, valuetype, mapValue);
+ map.put(mapKey, mapValue);
+ }
+ fieldValue = map;
+ break;
+ case ARRAY:
+ valueSchema = fieldSchema.getElementType();
+ valuetype = valueSchema.getType();
+ ArrayList<Object> list = new ArrayList<>();
+ for (Object item : (Collection<?>) fieldValue) {
+ Object value = getFieldValueFromAvroBean(valueSchema, valuetype, item);
+ list.add(value);
+ }
+ fieldValue = list;
+ break;
+ case UNION:
+ // storing the union selected schema, the actual value will
+ // be stored as soon as we get break out.
+ if (fieldValue != null) {
+ int schemaPos = getUnionSchema(fieldValue, fieldSchema);
+ Schema unionSchema = fieldSchema.getTypes().get(schemaPos);
+ Schema.Type unionType = unionSchema.getType();
+ fieldValue = getFieldValueFromAvroBean(unionSchema, unionType, fieldValue);
+ }
+ break;
+ case STRING:
+ fieldValue = fieldValue.toString();
+ break;
+ default:
+ break;
+ }
+ return fieldValue;
+ }
+
+ /**
+ * Given an object and the object schema this function obtains,
+ * from within the UNION schema, the position of the type used.
+ * If no data type can be inferred then we return a default value
+ * of position 0.
+ *
+ * @param pValue Object
+ * @param pUnionSchema avro Schema
+ * @return the unionSchemaPosition.
+ */
+ private static int getUnionSchema(Object pValue, Schema pUnionSchema) {
+ int unionSchemaPos = 0;
+// String valueType = pValue.getClass().getSimpleName();
+ for (Schema currentSchema : pUnionSchema.getTypes()) {
+ Schema.Type schemaType = currentSchema.getType();
+ if (pValue instanceof CharSequence && schemaType.equals(Schema.Type.STRING))
+ return unionSchemaPos;
+ else if (pValue instanceof ByteBuffer && schemaType.equals(Schema.Type.BYTES))
+ return unionSchemaPos;
+ else if (pValue instanceof Integer && schemaType.equals(Schema.Type.INT))
+ return unionSchemaPos;
+ else if (pValue instanceof Long && schemaType.equals(Schema.Type.LONG))
+ return unionSchemaPos;
+ else if (pValue instanceof Double && schemaType.equals(Schema.Type.DOUBLE))
+ return unionSchemaPos;
+ else if (pValue instanceof Float && schemaType.equals(Schema.Type.FLOAT))
+ return unionSchemaPos;
+ else if (pValue instanceof Boolean && schemaType.equals(Schema.Type.BOOLEAN))
+ return unionSchemaPos;
+ else if (pValue instanceof Map && schemaType.equals(Schema.Type.MAP))
+ return unionSchemaPos;
+ else if (pValue instanceof List && schemaType.equals(Schema.Type.ARRAY))
+ return unionSchemaPos;
+ else if (pValue instanceof Persistent && schemaType.equals(Schema.Type.RECORD))
+ return unionSchemaPos;
+ unionSchemaPos++;
+ }
+ // if we weren't able to determine which data type it is, then we return the default
+ return DEFAULT_UNION_SCHEMA;
+ }
+
+ static String encodeFieldKey(final String key) {
+ if (key == null) {
+ return null;
+ }
+ return key.replace(".", "\u00B7")
+ .replace(":", "\u00FF")
+ .replace(";", "\u00FE")
+ .replace(" ", "\u00FD")
+ .replace("%", "\u00FC")
+ .replace("=", "\u00FB");
+ }
+
+ static String decodeFieldKey(final String key) {
--- End diff --
This method doesn't have any usages.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r130023044
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/query/CassandraResultSet.java ---
@@ -0,0 +1,120 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.query;
+
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
--- End diff --
Please remove unused imports.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r130019986
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/serializers/CassandraQueryFactory.java ---
@@ -0,0 +1,701 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.cassandra.serializers;
+
+import com.datastax.driver.core.querybuilder.Delete;
+import com.datastax.driver.core.querybuilder.QueryBuilder;
--- End diff --
Can't we further minimize usages of string manipulations for queries using query builder for cql - com.datastax.driver.core.querybuilder.QueryBuilder ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora issue #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on the issue:
https://github.com/apache/gora/pull/110
@madhawa-gunasekara this looks very good :) heading in right direction. Please address my comments as go though implementing the rest of the datastore methods.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r123814653
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/store/TestCassandraStoreWithNativeSerialization.java ---
@@ -26,15 +26,20 @@ public static void setUpClass() throws Exception {
setProperties();
testDriver.setParameters(parameter);
testDriver.setUpClass();
+ // This time is added since embedded cassandra service needs some time to start up.
--- End diff --
ack... why so long I wonder? same as hot loading in Tomcat.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: Initial commit gora cassandra cql
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r122812132
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/store/TestCassandraStore.java ---
@@ -0,0 +1,92 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * Testing class for all standard gora-cassandra functionality.
+ * We extend DataStoreTestBase enabling us to run the entire base test
+ * suite for Gora.
+ */
+package org.apache.gora.cassandra.store;
+
+import java.io.IOException;
+
+import org.apache.gora.cassandra.GoraCassandraTestDriver;
+import org.apache.gora.examples.generated.Employee;
+import org.apache.gora.examples.generated.WebPage;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.store.DataStoreFactory;
+import org.apache.gora.store.DataStoreTestBase;
+import org.apache.hadoop.conf.Configuration;
+import org.junit.Before;
+import org.junit.Ignore;
+
+/**
+ * Test for CassandraStore.
+ */
+public class TestCassandraStore extends DataStoreTestBase{
+
+ private Configuration conf;
+
+ static {
+ setTestDriver(new GoraCassandraTestDriver());
+ }
+
+ @Before
+ public void setUp() throws Exception {
+ super.setUp();
+ }
+
+ public GoraCassandraTestDriver getTestDriver() {
+ return (GoraCassandraTestDriver) testDriver;
+ }
+
+
+ @Ignore("GORA-299 o.a.g.cassandra.CassandraStore#newQuery() should not use query.setFields(getFieldsToQuery(null));")
--- End diff --
Please remove all @ Ignore tags here and make sure that these tests run.. if they do not run then we need to implement the functionality.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r130023982
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/bean/CassandraKey.java ---
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.bean;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * This Class represents the Cassandra Key.
+ */
+public class CassandraKey{
--- End diff --
Seems this is not Formatted correctly. Should be - CassandraKey {
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: Initial commit gora cassandra cql
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r122809493
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java ---
@@ -0,0 +1,516 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.store;
+
+import com.datastax.driver.core.*;
+import com.datastax.driver.core.policies.*;
+import org.apache.gora.persistency.BeanFactory;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.hadoop.conf.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.List;
+import java.util.Properties;
+
+public class CassandraStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ /**
+ * Consistency property level for Cassandra column families
+ */
+ private static final String COL_FAM_CL = "cf.consistency.level";
+
+ /**
+ * Consistency property level for Cassandra read operations.
+ */
+ private static final String READ_OP_CL = "read.consistency.level";
+
+ /**
+ * Consistency property level for Cassandra write operations.
+ */
+ private static final String WRITE_OP_CL = "write.consistency.level";
+
+ public static final Logger LOG = LoggerFactory.getLogger(CassandraStore.class);
+
+ private Cluster cluster;
+
+ private Session session;
+
+ public CassandraStore() {
+ super();
+ }
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) {
+ LOG.debug("Initializing Cassandra store");
+ super.initialize(keyClass, persistentClass, properties);
+ try {
+ Cluster.Builder builder = Cluster.builder();
+ populateSettings(builder, properties);
+ this.cluster = builder.build();
+ this.session = this.cluster.connect();
+ } catch (Exception e) {
+ LOG.error("Error while initializing Cassandra store: {}",
+ new Object[]{e.getMessage()});
+ throw new RuntimeException(e);
+ }
+ }
+
+ private void populateSettings(Cluster.Builder builder, Properties properties) {
+ String serversParam = properties.getProperty(Constants.CASSANDRA_SERVERS);
+ String[] servers = serversParam.split(",");
+ for (String server : servers) {
+ builder = builder.addContactPoint(server);
+ }
+ String portProp = properties.getProperty(Constants.PORT);
+ if (portProp != null) {
+ builder = builder.withPort(Integer.parseInt(portProp));
+ }
+ String clusterNameProp = properties.getProperty(Constants.CLUSTER_NAME);
+ if (clusterNameProp != null) {
+ builder = builder.withClusterName(clusterNameProp);
+ }
+ String compressionProp = properties.getProperty(Constants.COMPRESSION);
+ if (compressionProp != null) {
+ builder = builder.withCompression(ProtocolOptions.Compression.valueOf(compressionProp));
+ }
+ builder = this.populateCredentials(properties, builder);
+ builder = this.populateLoadBalancingProp(properties, builder);
+ String enableJMXProp = properties.getProperty(Constants.ENABLE_JMX_REPORTING);
+ if (enableJMXProp != null) {
+ if (!Boolean.parseBoolean(enableJMXProp)) {
+ builder = builder.withoutJMXReporting();
+ }
+ }
+ String enableMetricsProp = properties.getProperty(Constants.ENABLE_METRICS);
+ if (enableMetricsProp != null) {
+ if (!Boolean.parseBoolean(enableMetricsProp)) {
+ builder = builder.withoutMetrics();
+ }
+ }
+ builder = this.populatePoolingSettings(properties, builder);
+ String versionProp = properties.getProperty(Constants.PROTOCOL_VERSION);
+ if (versionProp != null) {
+ builder = builder.withProtocolVersion(ProtocolVersion.fromInt(Integer.parseInt(versionProp)));
+ }
+ builder = this.populateQueryOptions(properties, builder);
+ builder = this.populateReconnectPolicy(properties, builder);
+ builder = this.populateRetrytPolicy(properties, builder);
+ builder = this.populateSocketOptions(properties, builder);
+ String enableSSLProp = properties.getProperty(Constants.ENABLE_SSL);
+ if (enableSSLProp != null) {
+ if (Boolean.parseBoolean(enableSSLProp)) {
+ builder = builder.withSSL();
+ }
+ }
+ }
+
+
+ private Cluster.Builder populateLoadBalancingProp(Properties properties, Cluster.Builder builder) {
+ String loadBalancingProp = properties.getProperty(Constants.LOAD_BALANCING_POLICY);
+ if (loadBalancingProp != null) {
+ switch (loadBalancingProp) {
+ case "LatencyAwareRoundRobinPolicy":
+ builder = builder.withLoadBalancingPolicy(LatencyAwarePolicy.builder(new RoundRobinPolicy()).build());
+ break;
+ case "RoundRobinPolicy":
+ builder = builder.withLoadBalancingPolicy(new RoundRobinPolicy());
+ break;
+ case "DCAwareRoundRobinPolicy": {
+ String dataCenter = properties.getProperty(Constants.DATA_CENTER);
+ boolean allowRemoteDCsForLocalConsistencyLevel = Boolean.parseBoolean(
+ properties.getProperty(Constants.ALLOW_REMOTE_DCS_FOR_LOCAL_CONSISTENCY_LEVEL));
+ if (dataCenter != null && !dataCenter.isEmpty()) {
+ if (allowRemoteDCsForLocalConsistencyLevel) {
+ builder = builder.withLoadBalancingPolicy(
+ DCAwareRoundRobinPolicy.builder().withLocalDc(dataCenter)
+ .allowRemoteDCsForLocalConsistencyLevel().build());
+ } else {
+ builder = builder.withLoadBalancingPolicy(
+ DCAwareRoundRobinPolicy.builder().withLocalDc(dataCenter).build());
+ }
+ } else {
+ if (allowRemoteDCsForLocalConsistencyLevel) {
+ builder = builder.withLoadBalancingPolicy(
+ (DCAwareRoundRobinPolicy.builder().allowRemoteDCsForLocalConsistencyLevel().build()));
+ } else {
+ builder = builder.withLoadBalancingPolicy((DCAwareRoundRobinPolicy.builder().build()));
+ }
+ }
+ break;
+ }
+ case "TokenAwareRoundRobinPolicy":
+ builder = builder.withLoadBalancingPolicy(new TokenAwarePolicy(new RoundRobinPolicy()));
+ break;
+ case "TokenAwareDCAwareRoundRobinPolicy": {
+ String dataCenter = properties.getProperty(Constants.DATA_CENTER);
+ boolean allowRemoteDCsForLocalConsistencyLevel = Boolean.parseBoolean(
+ properties.getProperty(Constants.ALLOW_REMOTE_DCS_FOR_LOCAL_CONSISTENCY_LEVEL));
+ if (dataCenter != null && !dataCenter.isEmpty()) {
+ if (allowRemoteDCsForLocalConsistencyLevel) {
+ builder = builder.withLoadBalancingPolicy(new TokenAwarePolicy(
+ DCAwareRoundRobinPolicy.builder().withLocalDc(dataCenter)
+ .allowRemoteDCsForLocalConsistencyLevel().build()));
+ } else {
+ builder = builder.withLoadBalancingPolicy(new TokenAwarePolicy(
+ DCAwareRoundRobinPolicy.builder().withLocalDc(dataCenter).build()));
+ }
+ } else {
+ if (allowRemoteDCsForLocalConsistencyLevel) {
+ builder = builder.withLoadBalancingPolicy(new TokenAwarePolicy(
+ DCAwareRoundRobinPolicy.builder().allowRemoteDCsForLocalConsistencyLevel().build()));
+ } else {
+ builder = builder.withLoadBalancingPolicy(
+ new TokenAwarePolicy(DCAwareRoundRobinPolicy.builder().build()));
+ }
+ }
+ break;
+ }
+ default:
+ LOG.error("Unsupported Cassandra load balancing " + "policy: " + loadBalancingProp);
+ break;
+ }
+ }
+ return builder;
+ }
+
+ private Cluster.Builder populateCredentials(Properties properties, Cluster.Builder builder) {
+ String usernameProp = properties.getProperty(Constants.USERNAME);
+ String passwordProp = properties.getProperty(Constants.PASSWORD);
+ if (usernameProp != null) {
+ builder = builder.withCredentials(usernameProp, passwordProp);
+ }
+ return builder;
+ }
+
+ private Cluster.Builder populatePoolingSettings(Properties properties, Cluster.Builder builder) {
+ String localCoreConnectionsPerHost = properties.getProperty(Constants.LOCAL_CORE_CONNECTIONS_PER_HOST);
+ String remoteCoreConnectionsPerHost = properties.getProperty(Constants.REMOTE_CORE_CONNECTIONS_PER_HOST);
+ String localMaxConnectionsPerHost = properties.getProperty(Constants.LOCAL_MAX_CONNECTIONS_PER_HOST);
+ String remoteMaxConnectionsPerHost = properties.getProperty(Constants.REMOTE_MAX_CONNECTIONS_PER_HOST);
+ String localNewConnectionThreshold = properties.getProperty(Constants.LOCAL_NEW_CONNECTION_THRESHOLD);
+ String remoteNewConnectionThreshold = properties.getProperty(Constants.REMOTE_NEW_CONNECTION_THRESHOLD);
+ String localMaxRequestsPerConnection = properties.getProperty(Constants.LOCAL_MAX_REQUESTS_PER_CONNECTION);
+ String remoteMaxRequestsPerConnection = properties.getProperty(Constants.REMOTE_MAX_REQUESTS_PER_CONNECTION);
+ PoolingOptions options = new PoolingOptions();
+ if (localCoreConnectionsPerHost != null) {
+ options.setCoreConnectionsPerHost(HostDistance.LOCAL, Integer.parseInt(localCoreConnectionsPerHost));
+ }
+ if (remoteCoreConnectionsPerHost != null) {
+ options.setCoreConnectionsPerHost(HostDistance.REMOTE, Integer.parseInt(remoteCoreConnectionsPerHost));
+ }
+ if (localMaxConnectionsPerHost != null) {
+ options.setMaxConnectionsPerHost(HostDistance.LOCAL, Integer.parseInt(localMaxConnectionsPerHost));
+ }
+ if (remoteMaxConnectionsPerHost != null) {
+ options.setMaxConnectionsPerHost(HostDistance.REMOTE, Integer.parseInt(remoteMaxConnectionsPerHost));
+ }
+ if (localNewConnectionThreshold != null) {
+ options.setNewConnectionThreshold(HostDistance.LOCAL, Integer.parseInt(localNewConnectionThreshold));
+ }
+ if (remoteNewConnectionThreshold != null) {
+ options.setNewConnectionThreshold(HostDistance.REMOTE, Integer.parseInt(remoteNewConnectionThreshold));
+ }
+ if (localMaxRequestsPerConnection != null) {
+ options.setMaxRequestsPerConnection(HostDistance.LOCAL, Integer.parseInt(localMaxRequestsPerConnection));
+ }
+ if (remoteMaxRequestsPerConnection != null) {
+ options.setMaxRequestsPerConnection(HostDistance.REMOTE, Integer.parseInt(remoteMaxRequestsPerConnection));
+ }
+ builder = builder.withPoolingOptions(options);
+ return builder;
+ }
+
+ private Cluster.Builder populateQueryOptions(Properties properties, Cluster.Builder builder) {
+ String consistencyLevelProp = properties.getProperty(Constants.CONSISTENCY_LEVEL);
+ String serialConsistencyLevelProp = properties.getProperty(Constants.SERIAL_CONSISTENCY_LEVEL);
+ String fetchSize = properties.getProperty(Constants.FETCH_SIZE);
+ QueryOptions options = new QueryOptions();
+ if (consistencyLevelProp != null) {
+ options.setConsistencyLevel(ConsistencyLevel.valueOf(consistencyLevelProp));
+ }
+ if (serialConsistencyLevelProp != null) {
+ options.setSerialConsistencyLevel(ConsistencyLevel.valueOf(serialConsistencyLevelProp));
+ }
+ if (fetchSize != null) {
+ options.setFetchSize(Integer.parseInt(fetchSize));
+ }
+ return builder.withQueryOptions(options);
+ }
+
+ private Cluster.Builder populateReconnectPolicy(Properties properties, Cluster.Builder builder) {
+ String reconnectionPolicy = properties.getProperty(Constants.RECONNECTION_POLICY);
+ if (reconnectionPolicy != null) {
+ switch (reconnectionPolicy) {
+ case "ConstantReconnectionPolicy": {
+ String constantReconnectionPolicyDelay = properties.getProperty(Constants.CONSTANT_RECONNECTION_POLICY_DELAY);
+
+ ConstantReconnectionPolicy policy = new ConstantReconnectionPolicy(Long.parseLong(constantReconnectionPolicyDelay));
+ builder = builder.withReconnectionPolicy(policy);
+ break;
+ }
+ case "ExponentialReconnectionPolicy": {
+ String exponentialReconnectionPolicyBaseDelay = properties.getProperty(Constants.EXPONENTIAL_RECONNECTION_POLICY_BASE_DELAY);
+ String exponentialReconnectionPolicyMaxDelay = properties.getProperty(Constants.EXPONENTIAL_RECONNECTION_POLICY_MAX_DELAY);
+
+ ExponentialReconnectionPolicy policy = new ExponentialReconnectionPolicy(Long.parseLong(exponentialReconnectionPolicyBaseDelay),
+ Long.parseLong(exponentialReconnectionPolicyMaxDelay));
+ builder = builder.withReconnectionPolicy(policy);
+ break;
+ }
+ default:
+ LOG.error("Unsupported reconnection policy " + reconnectionPolicy);
+ }
+ }
+ return builder;
+ }
+
+ private Cluster.Builder populateRetrytPolicy(Properties properties, Cluster.Builder builder) {
+ String retryPolicy = properties.getProperty(Constants.RETRY_POLICY);
+ if (retryPolicy != null) {
+ switch (retryPolicy) {
+ case "DefaultRetryPolicy":
+ builder = builder.withRetryPolicy(DefaultRetryPolicy.INSTANCE);
+ break;
+ case "DowngradingConsistencyRetryPolicy":
+ builder = builder.withRetryPolicy(DowngradingConsistencyRetryPolicy.INSTANCE);
+ break;
+ case "FallthroughRetryPolicy":
+ builder = builder.withRetryPolicy(FallthroughRetryPolicy.INSTANCE);
+ break;
+ case "LoggingDefaultRetryPolicy":
+ builder = builder.withRetryPolicy(new LoggingRetryPolicy(DefaultRetryPolicy.INSTANCE));
+ break;
+ case "LoggingDowngradingConsistencyRetryPolicy":
+ builder = builder.withRetryPolicy(new LoggingRetryPolicy(DowngradingConsistencyRetryPolicy.INSTANCE));
+ break;
+ case "LoggingFallthroughRetryPolicy":
+ builder = builder.withRetryPolicy(new LoggingRetryPolicy(FallthroughRetryPolicy.INSTANCE));
+ break;
+ default:
+ LOG.error("Unsupported retry policy " + retryPolicy);
+ break;
+ }
+ }
+ return builder;
+ }
+
+ private Cluster.Builder populateSocketOptions(Properties properties, Cluster.Builder builder) {
+ String connectionTimeoutMillisProp = properties.getProperty(Constants.CONNECTION_TIMEOUT_MILLIS);
+ String keepAliveProp = properties.getProperty(Constants.KEEP_ALIVE);
+ String readTimeoutMillisProp = properties.getProperty(Constants.READ_TIMEOUT_MILLIS);
+ String receiveBufferSizeProp = properties.getProperty(Constants.RECEIVER_BUFFER_SIZE);
+ String reuseAddress = properties.getProperty(Constants.REUSE_ADDRESS);
+ String sendBufferSize = properties.getProperty(Constants.SEND_BUFFER_SIZE);
+ String soLinger = properties.getProperty(Constants.SO_LINGER);
+ String tcpNoDelay = properties.getProperty(Constants.TCP_NODELAY);
+ SocketOptions options = new SocketOptions();
+ if (connectionTimeoutMillisProp != null) {
+ options.setConnectTimeoutMillis(Integer.parseInt(connectionTimeoutMillisProp));
+ }
+ if (keepAliveProp != null) {
+ options.setKeepAlive(Boolean.parseBoolean(keepAliveProp));
+ }
+ if (readTimeoutMillisProp != null) {
+ options.setReadTimeoutMillis(Integer.parseInt(readTimeoutMillisProp));
+ }
+ if (receiveBufferSizeProp != null) {
+ options.setReceiveBufferSize(Integer.parseInt(receiveBufferSizeProp));
+ }
+ if (reuseAddress != null) {
+ options.setReuseAddress(Boolean.parseBoolean(reuseAddress));
+ }
+ if (sendBufferSize != null) {
+ options.setSendBufferSize(Integer.parseInt(sendBufferSize));
+ }
+ if (soLinger != null) {
+ options.setSoLinger(Integer.parseInt(soLinger));
+ }
+ if (tcpNoDelay != null) {
+ options.setTcpNoDelay(Boolean.parseBoolean(tcpNoDelay));
+ }
+ return builder.withSocketOptions(options);
+ }
+
+ @Override
+ public void setPersistentClass(Class<T> persistentClass) {
--- End diff --
Please make sure that all public methods are documented.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora issue #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on the issue:
https://github.com/apache/gora/pull/110
@madhawa-gunasekara what is the status of this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r130071694
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/GoraCassandraTestDriver.java ---
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * @author lewismc
+ *
+ */
+
+package org.apache.gora.cassandra;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.apache.cassandra.io.util.FileUtils;
+import org.apache.cassandra.service.CassandraDaemon;
+import org.apache.gora.GoraTestDriver;
+import org.apache.gora.cassandra.store.CassandraStore;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.store.DataStoreFactory;
+import org.apache.gora.util.GoraException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.util.Properties;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+
+// Logging imports
+
+/**
+ * Helper class for third party tests using gora-cassandra backend.
+ * @see GoraTestDriver for test specifics.
+ * This driver is the base for all test cases that require an embedded Cassandra
+ * server. In this case we draw on Hector's @see EmbeddedServerHelper.
+ * It starts (setUp) and stops (tearDown) embedded Cassandra server.
+ *
+ */
+
+public class GoraCassandraTestDriver extends GoraTestDriver {
+ private static Logger log = LoggerFactory.getLogger(GoraCassandraTestDriver.class);
+
+ private static String baseDirectory = "target/test";
+
+ private CassandraDaemon cassandraDaemon;
+
+ private Thread cassandraThread;
+
+ private Properties properties;
+
+ public void setParameters(Properties parameters) {
+ this.properties = parameters;
+ }
+
+ @Override
+ public <K, T extends Persistent> DataStore<K, T> createDataStore(Class<K> keyClass, Class<T> persistentClass) throws GoraException {
+ return DataStoreFactory.createDataStore(CassandraStore.class, keyClass, persistentClass , conf, properties, null);
+ }
+
+ /**
+ * @return temporary base directory of running cassandra instance
+ */
+ public String getBaseDirectory() {
+ return baseDirectory;
+ }
+
+ public GoraCassandraTestDriver() {
+ super(CassandraStore.class);
+ }
+
+ /**
+ * Starts embedded Cassandra server.
+ *
+ * @throws Exception
+ * if an error occurs
+ */
+ @Override
+ public void setUpClass(){
+ log.info("Starting embedded Cassandra Server...");
+ try {
+ cleanupDirectoriesFailover();
+ FileUtils.createDirectory(baseDirectory);
+ System.setProperty("log4j.configuration", "log4j-server.properties");
+ System.setProperty("cassandra.config", "cassandra.yaml");
+
+ cassandraDaemon = new CassandraDaemon();
+ cassandraDaemon.completeSetup();
+ cassandraDaemon.applyConfig();
+ cassandraDaemon.init(null);
+ cassandraThread = new Thread(new Runnable() {
+
+ public void run() {
+ try {
+ cassandraDaemon.start();
+ } catch (Exception e) {
+ log.error("Embedded casandra server run failed!", e);
+ }
+ }
+ });
+
+ cassandraThread.setDaemon(true);
+ cassandraThread.start();
+ } catch (Exception e) {
+ log.error("Embedded casandra server start failed!", e);
+
+ // cleanup
+ tearDownClass();
+ }
+ }
+
+
+ /**
+ * Stops embedded Cassandra server.
+ *
+ * @throws Exception
+ * if an error occurs
+ */
+ @Override
+ public void tearDownClass(){
+ log.info("Shutting down Embedded Cassandra server...");
+ if (cassandraThread != null) {
+ cassandraDaemon.stop();
+ cassandraDaemon.destroy();
+// cassandraThread.interrupt();
+// cassandraThread = null;
+ }
+/* Thread cleanupThread = new Thread(() -> {
--- End diff --
If not needed, please remove this commented code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r132824362
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/store/TestCassandraStore.java ---
@@ -0,0 +1,204 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ * <p>
+ * Testing class for all standard gora-cassandra functionality.
--- End diff --
This License header is modified from standard Apache header. Please correct this as the rest of the headers.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r130028159
--- Diff: gora-cassandra-cql/src/test/conf/gora.properties ---
@@ -0,0 +1,30 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+gora.datastore.default=org.apache.gora.cassandra.CassandraStore
+gora.cassandrastore.cluster=Gora Cassandra Test Cluster
+gora.cassandrastore.host=localhost:9160
+# property is annotated in CassandraClient#checkKeyspace()
+# options are ANY, ONE, TWO, THREE, LOCAL_QUORUM, EACH_QUORUM, QUORUM and ALL.
+gora.cassandrastore.cf.consistency.level=ONE
+gora.cassandrastore.read.consistency.level=QUORUM
+gora.cassandrastore.write.consistency.level=ONE
+
+
--- End diff --
Please remove this unnecessary new lines.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r129360422
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/query/CassandraResultSet.java ---
@@ -79,4 +79,8 @@ public void addResultElement(K key, T token) {
this.persistentObject.add(token);
this.size++;
}
+
+ public void setLimit(long limit) {
--- End diff --
Can you please make sure to document all (or as many as possible) public method you implement. This is good practice.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by madhawa-gunasekara <gi...@git.apache.org>.
Github user madhawa-gunasekara commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r134845082
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/store/TestNativeSerializationWithUDT.java ---
@@ -0,0 +1,84 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.store;
+
+import org.apache.gora.cassandra.GoraCassandraTestDriver;
+import org.apache.gora.cassandra.example.generated.nativeSerialization.Customer;
+import org.apache.gora.cassandra.example.generated.nativeSerialization.Document;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.util.Properties;
+
+public class TestNativeSerializationWithUDT {
+
+ private static GoraCassandraTestDriver testDriver = new GoraCassandraTestDriver();
+ private static CassandraStore<String, Document> documentCassandraStore;
+ private static Properties parameter;
+
+ @BeforeClass
+ public static void setUpClass() throws Exception {
+ setProperties();
+ testDriver.setParameters(parameter);
+ testDriver.setUpClass();
+ documentCassandraStore = (CassandraStore<String, Document>) testDriver.createDataStore(String.class, Document.class);
+ }
+
+ private static void setProperties() {
+ parameter = new Properties();
+ parameter.setProperty(CassandraStoreParameters.CASSANDRA_SERVERS, "localhost");
+ parameter.setProperty(CassandraStoreParameters.PORT, "9042");
+ parameter.setProperty(CassandraStoreParameters.CASSANDRA_SERIALIZATION_TYPE, "native");
+ parameter.setProperty(CassandraStoreParameters.PROTOCOL_VERSION, "3");
+ parameter.setProperty(CassandraStoreParameters.CLUSTER_NAME, "Test Cluster");
+ parameter.setProperty("gora.cassandrastore.mapping.file", "nativeUDT/gora-cassandra-mapping.xml");
+ }
+
+ @AfterClass
+ public static void tearDownClass() throws Exception {
+ testDriver.tearDownClass();
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ testDriver.tearDown();
+ }
+
+ /**
+ * This is for testGetNested() with UDT dataType with native serialization.
+ */
+ @Test
+ public void testSimplePutAndGEt() {
--- End diff --
It's my pleasure
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora issue #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on the issue:
https://github.com/apache/gora/pull/110
Hi @madhawa-gunasekara you have not addressed all of my comments on the PR... please look over my comments, thank you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by renato2099 <gi...@git.apache.org>.
Github user renato2099 commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r130792458
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/serializers/CassandraQueryFactory.java ---
@@ -0,0 +1,701 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.cassandra.serializers;
+
+import com.datastax.driver.core.querybuilder.Delete;
+import com.datastax.driver.core.querybuilder.QueryBuilder;
+import com.datastax.driver.core.querybuilder.Select;
+import com.datastax.driver.core.querybuilder.Update;
+import org.apache.gora.cassandra.bean.CassandraKey;
+import org.apache.gora.cassandra.bean.ClusterKeyField;
+import org.apache.gora.cassandra.bean.Field;
+import org.apache.gora.cassandra.bean.KeySpace;
+import org.apache.gora.cassandra.bean.PartitionKeyField;
+import org.apache.gora.cassandra.query.CassandraQuery;
+import org.apache.gora.cassandra.store.CassandraMapping;
+import org.apache.gora.query.Query;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+
+/**
+ * This class is used create Cassandra Queries.
+ */
+class CassandraQueryFactory {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CassandraQueryFactory.class);
+
+ /**
+ * This method returns the CQL query to create key space.
+ * refer : http://docs.datastax.com/en/cql/3.1/cql/cql_reference/create_keyspace_r.html
+ *
+ * @param mapping Cassandra Mapping {@link CassandraMapping}
+ * @return CQL Query
+ */
+ static String getCreateKeySpaceQuery(CassandraMapping mapping) {
+ KeySpace keySpace = mapping.getKeySpace();
+ StringBuilder stringBuffer = new StringBuilder();
+ stringBuffer.append("CREATE KEYSPACE IF NOT EXISTS ").append(keySpace.getName()).append(" WITH REPLICATION = { 'class' : ");
+ KeySpace.PlacementStrategy placementStrategy = keySpace.getPlacementStrategy();
+ stringBuffer.append("'").append(placementStrategy).append("'").append(", ").append("'");
+ switch (placementStrategy) {
+ case SimpleStrategy:
+ stringBuffer.append("replication_factor").append("'").append(" : ").append(keySpace.getReplicationFactor()).append(" }");
+ break;
+ case NetworkTopologyStrategy:
+ boolean isCommaNeeded = false;
+ for (Map.Entry<String, Integer> entry : keySpace.getDataCenters().entrySet()) {
+ if (isCommaNeeded) {
+ stringBuffer.append(", '");
+ }
+ stringBuffer.append(entry.getKey()).append("'").append(" : ").append(entry.getValue());
+ isCommaNeeded = true;
+ }
+ stringBuffer.append(" }");
+ break;
+ }
+
+ if (keySpace.isDurableWritesEnabled()) {
+ stringBuffer.append(" AND DURABLE_WRITES = ").append(keySpace.isDurableWritesEnabled());
+ }
+ return stringBuffer.toString();
+ }
+
+ /**
+ * This method returns the CQL query to table.
+ * refer : http://docs.datastax.com/en/cql/3.1/cql/cql_reference/create_table_r.html
+ * <p>
+ * Trick : To have a consistency of the order of the columns, first we append partition keys, second cluster keys and finally other columns.
+ * It's very much needed to follow the same order in other CRUD operations as well.
+ *
+ * @param mapping Cassandra mapping {@link CassandraMapping}
+ * @return CQL Query
+ */
+ static String getCreateTableQuery(CassandraMapping mapping) {
+ StringBuilder stringBuffer = new StringBuilder();
+ stringBuffer.append("CREATE TABLE IF NOT EXISTS ").append(mapping.getKeySpace().getName()).append(".").append(mapping.getCoreName()).append(" (");
+ boolean isCommaNeeded = false;
+ CassandraKey cassandraKey = mapping.getCassandraKey();
+ // appending Cassandra key columns into db schema
+ for (Field field : mapping.getFieldList()) {
+ if (isCommaNeeded) {
+ stringBuffer.append(", ");
+ }
+ stringBuffer.append(field.getColumnName()).append(" ").append(field.getType());
+ boolean isStaticColumn = Boolean.parseBoolean(field.getProperty("static"));
+ boolean isPrimaryKey = Boolean.parseBoolean(field.getProperty("primarykey"));
+ if (isStaticColumn) {
+ stringBuffer.append(" STATIC");
+ }
+ if (isPrimaryKey) {
+ stringBuffer.append(" PRIMARY KEY ");
+ }
+ isCommaNeeded = true;
+ }
+
+ if (cassandraKey != null) {
+ List<PartitionKeyField> pkey = cassandraKey.getPartitionKeyFields();
+ if (pkey != null) {
+ stringBuffer.append(", PRIMARY KEY (");
+ boolean isCommaNeededToApply = false;
+ for (PartitionKeyField keyField : pkey) {
+ if (isCommaNeededToApply) {
+ stringBuffer.append(",");
+ }
+ if (keyField.isComposite()) {
+ stringBuffer.append("(");
+ boolean isCommaNeededHere = false;
+ for (Field field : keyField.getFields()) {
+ if (isCommaNeededHere) {
+ stringBuffer.append(", ");
+ }
+ stringBuffer.append(field.getColumnName());
+ isCommaNeededHere = true;
+ }
+ stringBuffer.append(")");
+ } else {
+ stringBuffer.append(keyField.getColumnName());
+ }
+ isCommaNeededToApply = true;
+ }
+ stringBuffer.append(")");
+ }
+ } else {
+ if (!stringBuffer.toString().toLowerCase(Locale.ENGLISH).contains("primary key")) {
+ Field field = mapping.getDefaultCassandraKey();
+ stringBuffer.append(", ").append(field.getFieldName()).append(" ").append(field.getType()).append(" PRIMARY KEY ");
+ }
+ }
+
+ stringBuffer.append(")");
+ boolean isWithNeeded = true;
+ if (Boolean.parseBoolean(mapping.getProperty("compactStorage"))) {
+ stringBuffer.append(" WITH COMPACT STORAGE ");
+ isWithNeeded = false;
+ }
+
+ String id = mapping.getProperty("id");
+ if (id != null) {
+ if (isWithNeeded) {
+ stringBuffer.append(" WITH ");
+ } else {
+ stringBuffer.append(" AND ");
+ }
+ stringBuffer.append("ID = '").append(id).append("'");
+ isWithNeeded = false;
+ }
+ if (cassandraKey != null) {
+ List<ClusterKeyField> clusterKeyFields = cassandraKey.getClusterKeyFields();
+ if (clusterKeyFields != null) {
+ if (isWithNeeded) {
+ stringBuffer.append(" WITH ");
+ } else {
+ stringBuffer.append(" AND ");
+ }
+ stringBuffer.append(" CLUSTERING ORDER BY (");
+ boolean isCommaNeededToApply = false;
+ for (ClusterKeyField keyField : clusterKeyFields) {
+ if (isCommaNeededToApply) {
+ stringBuffer.append(", ");
+ }
+ stringBuffer.append(keyField.getColumnName()).append(" ");
+ if (keyField.getOrder() != null) {
+ stringBuffer.append(keyField.getOrder());
+ }
+ isCommaNeededToApply = true;
+ }
+ stringBuffer.append(")");
+ }
+ }
+ return stringBuffer.toString();
+ }
+
+ /**
+ * This method returns the CQL query to drop table.
+ * refer : http://docs.datastax.com/en/cql/3.1/cql/cql_reference/drop_table_r.html
+ *
+ * @param mapping Cassandra Mapping {@link CassandraMapping}
+ * @return CQL query
+ */
+ static String getDropTableQuery(CassandraMapping mapping) {
+ return "DROP TABLE IF EXISTS " + mapping.getKeySpace().getName() + "." + mapping.getCoreName();
+ }
+
+ /**
+ * This method returns the CQL query to drop key space.
+ * refer : http://docs.datastax.com/en/cql/3.1/cql/cql_reference/drop_keyspace_r.html
+ *
+ * @param mapping Cassandra Mapping {@link CassandraMapping}
+ * @return CQL query
+ */
+ static String getDropKeySpaceQuery(CassandraMapping mapping) {
+ return "DROP KEYSPACE IF EXISTS " + mapping.getKeySpace().getName();
+ }
+
+ /**
+ * This method returns the CQL query to truncate (removes all the data) in the table.
+ * refer : http://docs.datastax.com/en/cql/3.1/cql/cql_reference/truncate_r.html
+ *
+ * @param mapping Cassandra Mapping {@link CassandraMapping}
+ * @return CQL query
+ */
+ static String getTruncateTableQuery(CassandraMapping mapping) {
+ return "TRUNCATE TABLE " + mapping.getKeySpace().getName() + "." + mapping.getCoreName();
+ }
+
+ /**
+ * This method return the CQL query to insert data in to the table.
+ * refer : http://docs.datastax.com/en/cql/3.1/cql/cql_reference/insert_r.html
+ *
+ * @param mapping Cassandra Mapping {@link CassandraMapping}
+ * @param fields available fields
+ * @return CQL Query
+ */
+ static String getInsertDataQuery(CassandraMapping mapping, List<String> fields) {
+ String[] columnNames = getColumnNames(mapping, fields);
+ String[] objects = new String[fields.size()];
+ Arrays.fill(objects, "?");
+ return QueryBuilder.insertInto(mapping.getKeySpace().getName(), mapping.getCoreName()).values(columnNames, objects).getQueryString();
+ }
+
+ /**
+ * This method return the CQL query to delete a persistent in the table.
+ * refer : http://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlDelete.html
+ *
+ * @param mapping Cassandra Mapping {@link CassandraMapping}
+ * @param fields filed list to be deleted
+ * @return CQL Query
+ */
+ static String getDeleteDataQuery(CassandraMapping mapping, List<String> fields) {
+ String[] columnNames = getColumnNames(mapping, fields);
+ String[] objects = new String[fields.size()];
+ Arrays.fill(objects, "?");
+ Delete delete = QueryBuilder.delete().from(mapping.getKeySpace().getName(), mapping.getCoreName());
+ Delete.Where query = null;
+ boolean isWhereNeeded = true;
+ for (String columnName : columnNames) {
+ if (isWhereNeeded) {
+ query = delete.where(QueryBuilder.eq(columnName, "?"));
+ isWhereNeeded = false;
+ } else {
+ query = query.and(QueryBuilder.eq(columnName, "?"));
+ }
+ }
+ return query.getQueryString();
+ }
+
+ /**
+ * This method returns the CQL Select query to retrieve data from the table.
+ * refer: http://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlSelect.html
+ *
+ * @param mapping Cassandra Mapping {@link CassandraMapping}
+ * @param keyFields key fields
+ * @return CQL Query
+ */
+ static String getSelectObjectQuery(CassandraMapping mapping, List<String> keyFields) {
+ Select select = QueryBuilder.select().from(mapping.getKeySpace().getName(), mapping.getCoreName());
+ if (Boolean.parseBoolean(mapping.getProperty("allowFiltering"))) {
+ select.allowFiltering();
+ }
+ String[] columnNames = getColumnNames(mapping, keyFields);
+ Select.Where query = null;
+ boolean isWhereNeeded = true;
+ for (String columnName : columnNames) {
+ if (isWhereNeeded) {
+ query = select.where(QueryBuilder.eq(columnName, "?"));
+ isWhereNeeded = false;
+ } else {
+ query = query.and(QueryBuilder.eq(columnName, "?"));
+ }
+ }
+ return query.getQueryString();
+ }
+
+ /**
+ * This method returns CQL Select query to retrieve data from the table with given fields.
+ * This method is used for Avro Serialization
+ * refer: http://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlSelect.html
+ *
+ * @param mapping Cassandra Mapping {@link CassandraMapping}
+ * @param fields Given fields to retrieve
+ * @param keyFields key fields
+ * @return CQL Query
+ */
+ static String getSelectObjectWithFieldsQuery(CassandraMapping mapping, String[] fields, List<String> keyFields) {
+ Select select = QueryBuilder.select(getColumnNames(mapping, Arrays.asList(fields))).from(mapping.getKeySpace().getName(), mapping.getCoreName());
+ if (Boolean.parseBoolean(mapping.getProperty("allowFiltering"))) {
+ select.allowFiltering();
+ }
+ String[] columnNames = getColumnNames(mapping, keyFields);
+ Select.Where query = null;
+ boolean isWhereNeeded = true;
+ for (String columnName : columnNames) {
+ if (isWhereNeeded) {
+ query = select.where(QueryBuilder.eq(columnName, "?"));
+ isWhereNeeded = false;
+ } else {
+ query = query.and(QueryBuilder.eq(columnName, "?"));
+ }
+ }
+ return query.getQueryString();
+ }
+
+ /**
+ * This method returns CQL Select query to retrieve data from the table with given fields.
+ * This method is used for Native Serialization
+ * refer: http://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlSelect.html
+ *
+ * @param mapping Cassandra Mapping {@link CassandraMapping}
+ * @param fields Given fields to retrieve
+ * @return CQL Query
+ */
+ static String getSelectObjectWithFieldsQuery(CassandraMapping mapping, String[] fields) {
+ String cqlQuery = null;
+ String[] columnNames = getColumnNames(mapping, Arrays.asList(fields));
+ Select select = QueryBuilder.select(columnNames).from(mapping.getKeySpace().getName(), mapping.getCoreName());
+ if (Boolean.parseBoolean(mapping.getProperty("allowFiltering"))) {
+ select.allowFiltering();
+ }
+ CassandraKey cKey = mapping.getCassandraKey();
+ if (cKey != null) {
+ Select.Where query = null;
+ boolean isWhereNeeded = true;
+ for (Field field : cKey.getFieldList()) {
+ if (isWhereNeeded) {
+ query = select.where(QueryBuilder.eq(field.getColumnName(), "?"));
+ isWhereNeeded = false;
+ } else {
+ query = query.and(QueryBuilder.eq(field.getColumnName(), "?"));
+ }
+ }
+ cqlQuery = query != null ? query.getQueryString() : null;
+ } else {
+ for (Field field : mapping.getFieldList()) {
+ boolean isPrimaryKey = Boolean.parseBoolean(field.getProperty("primarykey"));
+ if (isPrimaryKey) {
+ cqlQuery = select.where(QueryBuilder.eq(field.getColumnName(), "?")).getQueryString();
+ break;
+ }
+ }
+ }
+ return cqlQuery;
+ }
+
+
+ /**
+ * This method returns CQL Query for execute method. This CQL contains a Select Query to retrieve data from the table
+ *
+ * @param mapping Cassandra Mapping {@link CassandraMapping}
+ * @param cassandraQuery Query {@link CassandraQuery}
+ * @param objects object list
+ * @return CQL Query
+ */
+ static String getExecuteQuery(CassandraMapping mapping, Query cassandraQuery, List<Object> objects) {
+ String[] fields = cassandraQuery.getFields();
+ fields = fields != null ? fields : mapping.getFieldNames();
+ Object startKey = cassandraQuery.getStartKey();
+ Object endKey = cassandraQuery.getEndKey();
+ Object key = cassandraQuery.getKey();
+ String primaryKey = null;
+ long limit = cassandraQuery.getLimit();
+ Select select = QueryBuilder.select(getColumnNames(mapping, Arrays.asList(fields))).from(mapping.getKeySpace().getName(), mapping.getCoreName());
+ if (limit > 0) {
+ select = select.limit((int) limit);
+ }
+ if (Boolean.parseBoolean(mapping.getProperty("allowFiltering"))) {
+ select.allowFiltering();
+ }
+ Select.Where query = null;
+ boolean isWhereNeeded = true;
+ if (key != null) {
+ if (mapping.getCassandraKey() != null) {
+ ArrayList<String> cassandraKeys = new ArrayList<>();
--- End diff --
👍
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora issue #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on the issue:
https://github.com/apache/gora/pull/110
@madhawa-gunasekara great, I am testing it and will commit to master once all tests pass.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r130025998
--- Diff: gora-cassandra-cql/src/examples/java/org/apache/gora/cassandra/example/generated/nativeSerialization/ComplexTypes.java ---
@@ -0,0 +1,102 @@
+package org.apache.gora.cassandra.example.generated.nativeSerialization;
--- End diff --
Please add License header here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r132823884
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/store/CassandraClient.java ---
@@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.store;
+
+import com.datastax.driver.core.Cluster;
+import com.datastax.driver.core.ConsistencyLevel;
+import com.datastax.driver.core.DataType;
+import com.datastax.driver.core.HostDistance;
+import com.datastax.driver.core.PoolingOptions;
+import com.datastax.driver.core.ProtocolOptions;
+import com.datastax.driver.core.ProtocolVersion;
+import com.datastax.driver.core.QueryOptions;
+import com.datastax.driver.core.Session;
+import com.datastax.driver.core.SocketOptions;
+import com.datastax.driver.core.TypeCodec;
+import com.datastax.driver.core.policies.ConstantReconnectionPolicy;
+import com.datastax.driver.core.policies.DCAwareRoundRobinPolicy;
+import com.datastax.driver.core.policies.DefaultRetryPolicy;
+import com.datastax.driver.core.policies.DowngradingConsistencyRetryPolicy;
+import com.datastax.driver.core.policies.ExponentialReconnectionPolicy;
+import com.datastax.driver.core.policies.FallthroughRetryPolicy;
+import com.datastax.driver.core.policies.LatencyAwarePolicy;
+import com.datastax.driver.core.policies.LoggingRetryPolicy;
+import com.datastax.driver.core.policies.RoundRobinPolicy;
+import com.datastax.driver.core.policies.TokenAwarePolicy;
+import com.datastax.driver.extras.codecs.arrays.DoubleArrayCodec;
+import com.datastax.driver.extras.codecs.arrays.FloatArrayCodec;
+import com.datastax.driver.extras.codecs.arrays.IntArrayCodec;
+import com.datastax.driver.extras.codecs.arrays.LongArrayCodec;
+import com.datastax.driver.extras.codecs.arrays.ObjectArrayCodec;
+import com.datastax.driver.extras.codecs.date.SimpleDateCodec;
+import com.datastax.driver.extras.codecs.date.SimpleTimestampCodec;
+import com.datastax.driver.extras.codecs.jdk8.OptionalCodec;
+import org.apache.gora.cassandra.bean.Field;
+import org.jdom.Document;
+import org.jdom.Element;
+import org.jdom.JDOMException;
+import org.jdom.input.SAXBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Locale;
+import java.util.Properties;
+
+/**
+ * This class provides the Cassandra Client Connection.
+ * Initialize the Cassandra Connection according to the Properties.
+ */
+public class CassandraClient {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CassandraClient.class);
+
+
--- End diff --
Better if we can remove this unnecessary new lines.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: Initial commit gora cassandra cql
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r122808040
--- Diff: gora-cassandra-cql/pom.xml ---
@@ -0,0 +1,230 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora</artifactId>
+ <version>0.8-SNAPSHOT</version>
+ <relativePath>../</relativePath>
+ </parent>
+ <artifactId>gora-cassandra-cql</artifactId>
+ <packaging>bundle</packaging>
+
+ <name>Apache Gora :: Cassandra - CQL</name>
+ <url>http://gora.apache.org</url>
+ <description>The Apache Gora open source framework provides an in-memory data model and
+ persistence for big data. Gora supports persisting to column stores, key value stores,
+ document stores and RDBMSs, and analyzing the data with extensive Apache Hadoop MapReduce
+ support.</description>
+ <inceptionYear>2010</inceptionYear>
+ <organization>
+ <name>The Apache Software Foundation</name>
+ <url>http://www.apache.org/</url>
+ </organization>
+ <issueManagement>
+ <system>JIRA</system>
+ <url>https://issues.apache.org/jira/browse/GORA</url>
+ </issueManagement>
+ <ciManagement>
+ <system>Jenkins</system>
+ <url>https://builds.apache.org/job/Gora-trunk/</url>
+ </ciManagement>
+
+ <properties>
+ <osgi.import>*</osgi.import>
+ <osgi.export>org.apache.gora.cassandra*;version="${project.version}";-noimport:=true</osgi.export>
+ </properties>
+
+ <build>
+ <directory>target</directory>
+ <outputDirectory>target/classes</outputDirectory>
+ <finalName>${project.artifactId}-${project.version}</finalName>
+ <testOutputDirectory>target/test-classes</testOutputDirectory>
+ <testSourceDirectory>src/test/java</testSourceDirectory>
+ <sourceDirectory>src/main/java</sourceDirectory>
+ <testResources>
+ <testResource>
+ <directory>${project.basedir}/src/test/conf</directory>
+ <includes>
+ <include>**/*</include>
+ </includes>
+ <!--targetPath>${project.basedir}/target/classes/</targetPath-->
+ </testResource>
+ </testResources>
+ <plugins>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>build-helper-maven-plugin</artifactId>
+ <version>${build-helper-maven-plugin.version}</version>
+ <executions>
+ <execution>
+ <phase>generate-sources</phase>
+ <goals>
+ <goal>add-source</goal>
+ </goals>
+ <configuration>
+ <sources>
+ <source>src/examples/java</source>
+ </sources>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+
+ <dependencies>
+
+ <!-- OSX JDK 7 -->
+ <!-- should be removed once fixed in Cassandra -->
+ <dependency>
+ <groupId>org.xerial.snappy</groupId>
+ <artifactId>snappy-java</artifactId>
+ <version>1.0.5-M3</version>
+ <scope>test</scope>
+ </dependency>
+
+ <!-- Gora Internal Dependencies -->
+ <dependency>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora-core</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora-core</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+
+ <!-- Cassandra Driver Dependencies -->
+ <dependency>
+ <groupId>com.datastax.cassandra</groupId>
+ <artifactId>cassandra-driver-core</artifactId>
+ <version>${cassandra-driver.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <!-- Cassandra Dependencies -->
+ <dependency>
+ <groupId>org.apache.cassandra</groupId>
+ <artifactId>cassandra-all</artifactId>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.cassandra.deps</groupId>
+ <artifactId>avro</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-log4j12</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>log4j-over-slf4j</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <!-- dependency>
+ <groupId>org.apache.cassandra</groupId>
+ <artifactId>cassandra-thrift</artifactId>
+ </dependency-->
+
+ <!--<dependency>-->
+ <!--<groupId>org.hectorclient</groupId>-->
+ <!--<artifactId>hector-core</artifactId>-->
+ <!--<exclusions>-->
+ <!--<exclusion>-->
+ <!--<groupId>org.apache.cassandra</groupId>-->
+ <!--<artifactId>cassandra-all</artifactId>-->
+ <!--</exclusion>-->
+ <!--</exclusions>-->
+ <!--</dependency>-->
+
+ <!-- Misc Dependencies -->
+ <dependency>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.jdom</groupId>
+ <artifactId>jdom</artifactId>
+ </dependency>
+
--- End diff --
Remove extra lines
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r124049334
--- Diff: gora-cassandra-cql/pom.xml ---
@@ -0,0 +1,232 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora</artifactId>
+ <version>0.8-SNAPSHOT</version>
+ <relativePath>../</relativePath>
+ </parent>
+ <artifactId>gora-cassandra-cql</artifactId>
+ <packaging>bundle</packaging>
+
+ <name>Apache Gora :: Cassandra - CQL</name>
+ <url>http://gora.apache.org</url>
+ <description>The Apache Gora open source framework provides an in-memory data model and
+ persistence for big data. Gora supports persisting to column stores, key value stores,
+ document stores and RDBMSs, and analyzing the data with extensive Apache Hadoop MapReduce
+ support.
+ </description>
+ <inceptionYear>2010</inceptionYear>
+ <organization>
+ <name>The Apache Software Foundation</name>
+ <url>http://www.apache.org/</url>
+ </organization>
+ <issueManagement>
+ <system>JIRA</system>
+ <url>https://issues.apache.org/jira/browse/GORA</url>
+ </issueManagement>
+ <ciManagement>
+ <system>Jenkins</system>
+ <url>https://builds.apache.org/job/Gora-trunk/</url>
+ </ciManagement>
+
+ <properties>
+ <guava.version>18.0</guava.version>
+ <osgi.import>*</osgi.import>
+ <osgi.export>org.apache.gora.cassandra*;version="${project.version}";-noimport:=true</osgi.export>
+ </properties>
+
+ <build>
+ <directory>target</directory>
+ <outputDirectory>target/classes</outputDirectory>
+ <finalName>${project.artifactId}-${project.version}</finalName>
+ <testOutputDirectory>target/test-classes</testOutputDirectory>
+ <testSourceDirectory>src/test/java</testSourceDirectory>
+ <sourceDirectory>src/main/java</sourceDirectory>
+ <testResources>
+ <testResource>
+ <directory>${project.basedir}/src/test/conf</directory>
+ <includes>
+ <include>**/*</include>
+ </includes>
+ <!--targetPath>${project.basedir}/target/classes/</targetPath-->
+ </testResource>
+ </testResources>
+ <plugins>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>build-helper-maven-plugin</artifactId>
+ <version>${build-helper-maven-plugin.version}</version>
+ <executions>
+ <execution>
+ <phase>generate-sources</phase>
+ <goals>
+ <goal>add-source</goal>
+ </goals>
+ <configuration>
+ <sources>
+ <source>src/examples/java</source>
+ </sources>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-plugin</artifactId>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.maven.surefire</groupId>
+ <artifactId>surefire-junit47</artifactId>
+ <version>2.20</version>
+ </dependency>
+ </dependencies>
+ <configuration>
+ <forkMode>always</forkMode>
+ <argLine>-Xmx1024m -XX:MaxPermSize=512m</argLine>
+ </configuration>
+ </plugin>
+ </plugins>
+ </build>
+
+ <dependencies>
+
+ <!-- OSX JDK 7 -->
+ <!-- should be removed once fixed in Cassandra -->
+ <dependency>
+ <groupId>org.xerial.snappy</groupId>
--- End diff --
Is this a temporary addition?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: Initial commit gora cassandra cql
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r122808313
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java ---
@@ -0,0 +1,516 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.store;
+
+import com.datastax.driver.core.*;
+import com.datastax.driver.core.policies.*;
+import org.apache.gora.persistency.BeanFactory;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.hadoop.conf.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.List;
+import java.util.Properties;
+
+public class CassandraStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ /**
+ * Consistency property level for Cassandra column families
+ */
+ private static final String COL_FAM_CL = "cf.consistency.level";
+
+ /**
+ * Consistency property level for Cassandra read operations.
+ */
+ private static final String READ_OP_CL = "read.consistency.level";
+
+ /**
+ * Consistency property level for Cassandra write operations.
+ */
+ private static final String WRITE_OP_CL = "write.consistency.level";
+
+ public static final Logger LOG = LoggerFactory.getLogger(CassandraStore.class);
+
+ private Cluster cluster;
+
+ private Session session;
+
+ public CassandraStore() {
+ super();
+ }
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) {
--- End diff --
Document method
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by madhawa-gunasekara <gi...@git.apache.org>.
Github user madhawa-gunasekara commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r126297993
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/test/nativeSerialization/DateAsStringCodec.java ---
@@ -0,0 +1,33 @@
+package org.apache.gora.cassandra.test.nativeSerialization;
--- End diff --
:+1: Added the license header
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r130027243
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/GoraCassandraTestDriver.java ---
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * @author lewismc
+ *
+ */
+
+package org.apache.gora.cassandra;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.apache.cassandra.io.util.FileUtils;
+import org.apache.cassandra.service.CassandraDaemon;
+import org.apache.gora.GoraTestDriver;
+import org.apache.gora.cassandra.store.CassandraStore;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.store.DataStoreFactory;
+import org.apache.gora.util.GoraException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.util.Properties;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+
+// Logging imports
+
+/**
+ * Helper class for third party tests using gora-cassandra backend.
+ * @see GoraTestDriver for test specifics.
+ * This driver is the base for all test cases that require an embedded Cassandra
+ * server. In this case we draw on Hector's @see EmbeddedServerHelper.
+ * It starts (setUp) and stops (tearDown) embedded Cassandra server.
+ *
+ */
+
+public class GoraCassandraTestDriver extends GoraTestDriver {
+ private static Logger log = LoggerFactory.getLogger(GoraCassandraTestDriver.class);
+
+ private static String baseDirectory = "target/test";
+
+ private CassandraDaemon cassandraDaemon;
+
+ private Thread cassandraThread;
+
+ private Properties properties;
+
+ public void setParameters(Properties parameters) {
+ this.properties = parameters;
+ }
+
+ @Override
+ public <K, T extends Persistent> DataStore<K, T> createDataStore(Class<K> keyClass, Class<T> persistentClass) throws GoraException {
+ return DataStoreFactory.createDataStore(CassandraStore.class, keyClass, persistentClass , conf, properties, null);
+ }
+
+ /**
+ * @return temporary base directory of running cassandra instance
+ */
+ public String getBaseDirectory() {
+ return baseDirectory;
+ }
+
+ public GoraCassandraTestDriver() {
+ super(CassandraStore.class);
+ }
+
+ /**
+ * Starts embedded Cassandra server.
+ *
+ * @throws Exception
+ * if an error occurs
+ */
+ @Override
+ public void setUpClass(){
+ log.info("Starting embedded Cassandra Server...");
+ try {
+ cleanupDirectoriesFailover();
+ FileUtils.createDirectory(baseDirectory);
+ System.setProperty("log4j.configuration", "log4j-server.properties");
+ System.setProperty("cassandra.config", "cassandra.yaml");
+
+ cassandraDaemon = new CassandraDaemon();
+ cassandraDaemon.completeSetup();
+ cassandraDaemon.applyConfig();
+ cassandraDaemon.init(null);
+ cassandraThread = new Thread(new Runnable() {
+
+ public void run() {
+ try {
+ cassandraDaemon.start();
+ } catch (Exception e) {
+ log.error("Embedded casandra server run failed!", e);
+ }
+ }
+ });
+
+ cassandraThread.setDaemon(true);
+ cassandraThread.start();
+ } catch (Exception e) {
+ log.error("Embedded casandra server start failed!", e);
+
+ // cleanup
+ tearDownClass();
+ }
+ }
+
+
+ /**
+ * Stops embedded Cassandra server.
+ *
+ * @throws Exception
+ * if an error occurs
+ */
+ @Override
+ public void tearDownClass(){
+ log.info("Shutting down Embedded Cassandra server...");
+ if (cassandraThread != null) {
+ cassandraDaemon.stop();
+ cassandraDaemon.destroy();
+// cassandraThread.interrupt();
+// cassandraThread = null;
+ }
+/* Thread cleanupThread = new Thread(() -> {
+ try {
--- End diff --
If not needed, Please remove this commented code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: Initial commit gora cassandra cql
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r122807988
--- Diff: gora-cassandra-cql/pom.xml ---
@@ -0,0 +1,230 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora</artifactId>
+ <version>0.8-SNAPSHOT</version>
+ <relativePath>../</relativePath>
+ </parent>
+ <artifactId>gora-cassandra-cql</artifactId>
+ <packaging>bundle</packaging>
+
+ <name>Apache Gora :: Cassandra - CQL</name>
+ <url>http://gora.apache.org</url>
+ <description>The Apache Gora open source framework provides an in-memory data model and
+ persistence for big data. Gora supports persisting to column stores, key value stores,
+ document stores and RDBMSs, and analyzing the data with extensive Apache Hadoop MapReduce
+ support.</description>
+ <inceptionYear>2010</inceptionYear>
+ <organization>
+ <name>The Apache Software Foundation</name>
+ <url>http://www.apache.org/</url>
+ </organization>
+ <issueManagement>
+ <system>JIRA</system>
+ <url>https://issues.apache.org/jira/browse/GORA</url>
+ </issueManagement>
+ <ciManagement>
+ <system>Jenkins</system>
+ <url>https://builds.apache.org/job/Gora-trunk/</url>
+ </ciManagement>
+
+ <properties>
+ <osgi.import>*</osgi.import>
+ <osgi.export>org.apache.gora.cassandra*;version="${project.version}";-noimport:=true</osgi.export>
+ </properties>
+
+ <build>
+ <directory>target</directory>
+ <outputDirectory>target/classes</outputDirectory>
+ <finalName>${project.artifactId}-${project.version}</finalName>
+ <testOutputDirectory>target/test-classes</testOutputDirectory>
+ <testSourceDirectory>src/test/java</testSourceDirectory>
+ <sourceDirectory>src/main/java</sourceDirectory>
+ <testResources>
+ <testResource>
+ <directory>${project.basedir}/src/test/conf</directory>
+ <includes>
+ <include>**/*</include>
+ </includes>
+ <!--targetPath>${project.basedir}/target/classes/</targetPath-->
+ </testResource>
+ </testResources>
+ <plugins>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>build-helper-maven-plugin</artifactId>
+ <version>${build-helper-maven-plugin.version}</version>
+ <executions>
+ <execution>
+ <phase>generate-sources</phase>
+ <goals>
+ <goal>add-source</goal>
+ </goals>
+ <configuration>
+ <sources>
+ <source>src/examples/java</source>
+ </sources>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+
+ <dependencies>
+
+ <!-- OSX JDK 7 -->
+ <!-- should be removed once fixed in Cassandra -->
+ <dependency>
+ <groupId>org.xerial.snappy</groupId>
+ <artifactId>snappy-java</artifactId>
+ <version>1.0.5-M3</version>
+ <scope>test</scope>
+ </dependency>
+
+ <!-- Gora Internal Dependencies -->
+ <dependency>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora-core</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora-core</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+
+ <!-- Cassandra Driver Dependencies -->
+ <dependency>
+ <groupId>com.datastax.cassandra</groupId>
+ <artifactId>cassandra-driver-core</artifactId>
+ <version>${cassandra-driver.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <!-- Cassandra Dependencies -->
+ <dependency>
+ <groupId>org.apache.cassandra</groupId>
+ <artifactId>cassandra-all</artifactId>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.cassandra.deps</groupId>
+ <artifactId>avro</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-log4j12</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>log4j-over-slf4j</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <!-- dependency>
--- End diff --
Remove commented out code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora issue #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on the issue:
https://github.com/apache/gora/pull/110
@madhawa-gunasekara This is excellent work :) I have reviewed you PR, I was not able find anything major, however I have added some comments, please check them out. Also code needs bit of a clean up as well, please go through the code again and see cases where you can improve the code .
I agree some of the concerns highlighted by @renato2099 in this thread. So I am waiting @renato2099 @lewismc to have a look and comment on updated PR.
I have also locally tested your PR, I am glad to say I did not encounter any issue. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r132824517
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/store/TestCassandraStore.java ---
@@ -0,0 +1,204 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ * <p>
+ * Testing class for all standard gora-cassandra functionality.
+ * We extend DataStoreTestBase enabling us to run the entire base test
+ * suite for Gora.
+ * <p>
+ * Testing class for all standard gora-cassandra functionality.
+ * We extend DataStoreTestBase enabling us to run the entire base test
+ * suite for Gora.
+ * <p>
+ * Testing class for all standard gora-cassandra functionality.
+ * We extend DataStoreTestBase enabling us to run the entire base test
+ * suite for Gora.
+ * <p>
+ * Testing class for all standard gora-cassandra functionality.
+ * We extend DataStoreTestBase enabling us to run the entire base test
+ * suite for Gora.
+ * <p>
+ * Testing class for all standard gora-cassandra functionality.
+ * We extend DataStoreTestBase enabling us to run the entire base test
+ * suite for Gora.
+ * <p>
+ * Testing class for all standard gora-cassandra functionality.
+ * We extend DataStoreTestBase enabling us to run the entire base test
+ * suite for Gora.
+ * <p>
+ * Testing class for all standard gora-cassandra functionality.
+ * We extend DataStoreTestBase enabling us to run the entire base test
+ * suite for Gora.
+ */
+
+/**
+ * Testing class for all standard gora-cassandra functionality.
+ * We extend DataStoreTestBase enabling us to run the entire base test
+ * suite for Gora.
+ */
+package org.apache.gora.cassandra.store;
+
+import org.apache.gora.cassandra.GoraCassandraTestDriver;
+import org.apache.gora.examples.WebPageDataCreator;
+import org.apache.gora.examples.generated.WebPage;
+import org.apache.gora.query.Query;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.store.DataStoreTestBase;
+import org.apache.gora.store.DataStoreTestUtil;
+import org.junit.Before;
+import org.junit.Ignore;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Properties;
+
+import static org.apache.gora.examples.WebPageDataCreator.SORTED_URLS;
+import static org.apache.gora.examples.WebPageDataCreator.URLS;
+import static org.apache.gora.store.DataStoreTestUtil.assertEmptyResults;
+import static org.apache.gora.store.DataStoreTestUtil.assertNumResults;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
+
+/**
+ * Test for CassandraStore.
+ */
+public class TestCassandraStore extends DataStoreTestBase {
--- End diff --
Thanks for adding AVRO serialization test cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by madhawa-gunasekara <gi...@git.apache.org>.
Github user madhawa-gunasekara commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r134844645
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/store/CassandraMappingBuilder.java ---
@@ -0,0 +1,240 @@
+package org.apache.gora.cassandra.store;
--- End diff --
done :+1:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r130069255
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/store/TestCassandraStoreWithNativeSerialization.java ---
@@ -0,0 +1,305 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.store;
+
+import org.apache.gora.cassandra.GoraCassandraTestDriver;
+import org.apache.gora.cassandra.example.generated.nativeSerialization.ComplexTypes;
+import org.apache.gora.cassandra.example.generated.nativeSerialization.User;
+import org.apache.gora.cassandra.query.CassandraQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.util.GoraException;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+
+/**
+ * This class tests Cassandra Store functionality with Cassandra Native Serialization.
+ */
+public class TestCassandraStoreWithNativeSerialization {
+ private static GoraCassandraTestDriver testDriver = new GoraCassandraTestDriver();
+ private static DataStore<UUID, User> userDataStore;
+ private static Properties parameter;
+
+ @BeforeClass
+ public static void setUpClass() throws Exception {
+ setProperties();
+ testDriver.setParameters(parameter);
+ testDriver.setUpClass();
+ userDataStore = testDriver.createDataStore(UUID.class, User.class);
+ }
+
+ private static void setProperties() {
+ parameter = new Properties();
+ parameter.setProperty(CassandraStoreParameters.CASSANDRA_SERVERS, "localhost");
+ parameter.setProperty(CassandraStoreParameters.PORT, "9042");
+ parameter.setProperty(CassandraStoreParameters.CASSANDRA_SERIALIZATION_TYPE, "native");
+ parameter.setProperty(CassandraStoreParameters.PROTOCOL_VERSION, "3");
+ parameter.setProperty(CassandraStoreParameters.CLUSTER_NAME, "Test Cluster");
+ parameter.setProperty("gora.cassandrastore.mapping.file", "nativeSerialization/gora-cassandra-mapping.xml");
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ testDriver.tearDown();
+ }
+
+ @AfterClass
+ public static void tearDownClass() throws Exception {
+ testDriver.tearDownClass();
+ }
+
+ /**
+ * In this test case, put and get behavior of the data store are testing.
+ */
+ @Test
+ public void testSimplePutAndGet() {
+ UUID id = UUID.randomUUID();
+ User user1 = new User(id, "madhawa", Date.from(Instant.now()));
+ // storing data;
+ userDataStore.put(id, user1);
+ // get data;
+ User olduser = userDataStore.get(id);
+ Assert.assertEquals(olduser.getName(), user1.getName());
+ Assert.assertEquals(olduser.getDateOfBirth(), user1.getDateOfBirth());
+ }
+
+ /**
+ * In this test case, put and delete behavior of the data store are testing.
+ */
+ @Test
+ public void testSimplePutDeleteAndGet() {
+ UUID id = UUID.randomUUID();
+ User user1 = new User(id, "kasun", Date.from(Instant.now()));
+ // storing data;
+ userDataStore.put(id, user1);
+ // get data;
+ User olduser = userDataStore.get(user1.getUserId());
+ Assert.assertEquals(olduser.getName(), user1.getName());
+ Assert.assertEquals(olduser.getDateOfBirth(), user1.getDateOfBirth());
+ // delete data;
+ userDataStore.delete(user1.getUserId());
+ // get data
+ User deletedUser = userDataStore.get(id);
+ Assert.assertNull(deletedUser);
+ }
+
+ /**
+ * In this test case, schema exists method behavior of the data store is testing.
+ */
+ @Test()
+ public void testSchemaExists() {
+ userDataStore.deleteSchema();
+ Assert.assertFalse(userDataStore.schemaExists());
+ userDataStore.createSchema();
+ Assert.assertTrue(userDataStore.schemaExists());
+ }
+
+ /**
+ * In this test case, schema exists method behavior of the data store is testing.
+ */
+ @Test
+ public void testTruncateSchema() {
+ if (!userDataStore.schemaExists()) {
+ userDataStore.createSchema();
+ }
+ UUID id = UUID.randomUUID();
+ User user1 = new User(id, "Madhawa Kasun", Date.from(Instant.now()));
+ userDataStore.put(id, user1);
+ User olduser = userDataStore.get(id);
+ Assert.assertEquals(olduser.getName(), user1.getName());
+ Assert.assertEquals(olduser.getDateOfBirth(), user1.getDateOfBirth());
+ userDataStore.truncateSchema();
+ olduser = userDataStore.get(id);
+ Assert.assertNull(olduser);
+ }
+
+ /**
+ * In this test case, get with fields method behavior of the data store is testing.
+ */
+ @Test
+ public void testGetWithFields() {
+ UUID id = UUID.randomUUID();
+ User user1 = new User(id, "Madhawa Kasun Gunasekara", Date.from(Instant.now()));
+ userDataStore.put(id, user1);
+ // get data;
+ User olduser = userDataStore.get(user1.getUserId());
+ Assert.assertEquals(olduser.getName(), user1.getName());
+ Assert.assertEquals(olduser.getDateOfBirth(), user1.getDateOfBirth());
+ User olduserWithFields = userDataStore.get(id, new String[]{"name"});
+ Assert.assertNull(olduserWithFields.getDateOfBirth());
+ }
+
+ /**
+ * In this test case, get with fields method behavior of the data store is testing.
+ */
+ @Test
+ public void testExecute() throws Exception {
+ userDataStore.truncateSchema();
+ Map<UUID, User> users = new HashMap<>();
+ UUID id1 = UUID.randomUUID();
+ User user1 = new User(id1, "user1", Date.from(Instant.now()));
+ users.put(id1, user1);
+ userDataStore.put(id1, user1);
+ UUID id2 = UUID.randomUUID();
+ User user2 = new User(id2, "user2", Date.from(Instant.now()));
+ users.put(id2, user2);
+ userDataStore.put(id2, user2);
+ UUID id3 = UUID.randomUUID();
+ User user3 = new User(id3, "user3", Date.from(Instant.now()));
+ users.put(id3, user3);
+ userDataStore.put(id3, user3);
+ Query<UUID, User> query1 = userDataStore.newQuery();
+ Result<UUID, User> result1 = userDataStore.execute(query1);
+ int i = 0;
+ Assert.assertEquals(result1.getProgress(),0.0,0.0);
+ while (result1.next()) {
+ // check objects values
+ Assert.assertEquals(result1.get().getName(), users.get(result1.getKey()).getName());
+ Assert.assertEquals(result1.get().getDateOfBirth(), users.get(result1.getKey()).getDateOfBirth());
+ Assert.assertEquals(result1.get().getUserId(), users.get(result1.getKey()).getUserId());
+ i++;
+ }
+ Assert.assertEquals(result1.getProgress(),1.0,0.0);
+ Assert.assertEquals(3, i);
+
+ // Check limit query
+ Query<UUID, User> query2 = userDataStore.newQuery();
+ query2.setLimit(2);
+ Result<UUID, User> result2 = userDataStore.execute(query2);
+ i = 0;
+ while (result2.next()) {
+ Assert.assertEquals(result2.get().getName(), users.get(result2.getKey()).getName());
+ Assert.assertEquals(result2.get().getDateOfBirth(), users.get(result2.getKey()).getDateOfBirth());
+ Assert.assertEquals(result2.get().getUserId(), users.get(result2.getKey()).getUserId());
+ i++;
+ }
+ Assert.assertEquals(2, i);
+
+ // check key element
+ Query<UUID, User> query3 = userDataStore.newQuery();
+ query3.setKey(id1);
+
+ Result<UUID, User> result3 = userDataStore.execute(query3);
+ i = 0;
+ while (result3.next()) {
+ Assert.assertEquals(result3.get().getName(), users.get(result3.getKey()).getName());
+ Assert.assertEquals(result3.get().getDateOfBirth(), users.get(result3.getKey()).getDateOfBirth());
+ Assert.assertEquals(result3.get().getUserId(), users.get(result3.getKey()).getUserId());
+ i++;
+ }
+ Assert.assertEquals(1, i);
+ }
+
+ /**
+ * In this test case, delete by query method behavior of the data store is testing.
+ */
+ @Test
+ public void testDeleteByQuery() throws Exception {
+ userDataStore.truncateSchema();
+ UUID id1 = UUID.randomUUID();
+ User user1 = new User(id1, "user1", Date.from(Instant.now()));
+ userDataStore.put(id1, user1);
+ UUID id2 = UUID.randomUUID();
+ User user2 = new User(id2, "user2", Date.from(Instant.now()));
+ userDataStore.put(id2, user2);
+ Query<UUID, User> query1 = userDataStore.newQuery();
+ query1.setKey(id1);
+ userDataStore.deleteByQuery(query1);
+ User user = userDataStore.get(id1);
+ Assert.assertNull(user);
+
+ //test deleteByFields
+ Query<UUID, User> query2 = userDataStore.newQuery();
+ query2.setKey(id2);
+ query2.setFields("name");
+ userDataStore.deleteByQuery(query2);
+ User partialDeletedUser = userDataStore.get(id2);
+ Assert.assertNull(partialDeletedUser.getName());
+ Assert.assertEquals(partialDeletedUser.getDateOfBirth(),user2.getDateOfBirth());
+ }
+
+ /**
+ * In this test case, update by quert method behavior of the data store is testing.
+ */
+ @Test
+ public void testUpdateByQuery() {
+ userDataStore.truncateSchema();
+ UUID id1 = UUID.randomUUID();
+ User user1 = new User(id1, "user1", Date.from(Instant.now()));
+ userDataStore.put(id1, user1);
+ UUID id2 = UUID.randomUUID();
+ User user2 = new User(id2, "user2", Date.from(Instant.now()));
+ userDataStore.put(id2, user2);
+ Query<UUID, User> query1 = userDataStore.newQuery();
+ if(query1 instanceof CassandraQuery) {
+ ((CassandraQuery) query1).addUpdateField("name", "madhawa");
+ }
+ query1.setKey(id1);
+ if(userDataStore instanceof CassandraStore) {
+ ((CassandraStore) userDataStore).updateByQuery(query1);
+ }
+ User user = userDataStore.get(id1);
+ Assert.assertEquals(user.getName(),"madhawa");
+ }
+
+ @Test
+ public void testComplexTypes() throws GoraException {
+ DataStore<String, ComplexTypes> documentDataStore = testDriver.createDataStore(String.class, ComplexTypes.class);
--- End diff --
It s great if you can extend your native test suit to test nested records.
Please see following tests in DataStoreTestBase.java.
public void testGetRecursive()
public void testGetDoubleRecursive()
public void testGetNested()
Also see possibilities of extending your native test cases compared to base tests we have in Gora.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora issue #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by renato2099 <gi...@git.apache.org>.
Github user renato2099 commented on the issue:
https://github.com/apache/gora/pull/110
Hi @madhawa-gunasekara ,
Sorry for the late reply, but to handle that case why don't just do the mapping manually as we currently do and only serialize differently? So what I had in mind was using our own same old avro java bean for keeping things in-memory, but when pushing this into Cassandra, then (1) store everything as avro binary data (serialize every type as binary data as we did), or (2) use cassandra native serialization so users can use regular CQL for checking their data.
You accomplished the same :) BUT now we have also introduced a different in-memory data bean which will not help in keeping a single entry-point for all no-sql databases (now we need Cassandra java beans). I think we could just go back to use our own avro data beans and use the native serialization you have coded, and then later handling the avro binary serialization. What do you think? how much work would it be?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by madhawa-gunasekara <gi...@git.apache.org>.
Github user madhawa-gunasekara commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r134844868
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/store/TestCassandraStoreWithNativeSerialization.java ---
@@ -0,0 +1,305 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.store;
+
+import org.apache.gora.cassandra.GoraCassandraTestDriver;
+import org.apache.gora.cassandra.example.generated.nativeSerialization.ComplexTypes;
+import org.apache.gora.cassandra.example.generated.nativeSerialization.User;
+import org.apache.gora.cassandra.query.CassandraQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.util.GoraException;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+
+/**
+ * This class tests Cassandra Store functionality with Cassandra Native Serialization.
+ */
+public class TestCassandraStoreWithNativeSerialization {
--- End diff --
Added
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/gora/pull/110
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r130026954
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/GoraCassandraTestDriver.java ---
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * @author lewismc
+ *
+ */
+
+package org.apache.gora.cassandra;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.apache.cassandra.io.util.FileUtils;
+import org.apache.cassandra.service.CassandraDaemon;
+import org.apache.gora.GoraTestDriver;
+import org.apache.gora.cassandra.store.CassandraStore;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.store.DataStoreFactory;
+import org.apache.gora.util.GoraException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.util.Properties;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+
+// Logging imports
+
+/**
+ * Helper class for third party tests using gora-cassandra backend.
+ * @see GoraTestDriver for test specifics.
+ * This driver is the base for all test cases that require an embedded Cassandra
+ * server. In this case we draw on Hector's @see EmbeddedServerHelper.
+ * It starts (setUp) and stops (tearDown) embedded Cassandra server.
+ *
+ */
+
--- End diff --
Unnecessary new line. Please remove this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r132823605
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/store/TestAvroSerializationWithUDT.java ---
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.store;
+
+import org.apache.avro.util.Utf8;
+import org.apache.gora.cassandra.GoraCassandraTestDriver;
+import org.apache.gora.examples.generated.Metadata;
+import org.apache.gora.examples.generated.WebPage;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Map;
+import java.util.Properties;
+
+public class TestAvroSerializationWithUDT {
--- End diff --
Shall we add java doc here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r132824780
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/compiler/GoraCassandraNativeCompiler.java ---
@@ -0,0 +1,319 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.compiler;
+
+import org.apache.commons.io.FilenameUtils;
+import org.apache.gora.cassandra.bean.Field;
+import org.apache.gora.cassandra.store.CassandraMapping;
+import org.apache.gora.cassandra.store.CassandraMappingBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.Writer;
+import java.nio.charset.Charset;
+import java.util.List;
+import java.util.Locale;
+
+public class GoraCassandraNativeCompiler {
+
+ private static final Logger log = LoggerFactory.getLogger(GoraCassandraNativeCompiler.class);
+
+ private Writer out;
+ private File dest;
+
+ GoraCassandraNativeCompiler(File dest) {
+ this.dest = dest;
+ }
+
+ /**
+ * Start point of the compiler program
+ *
+ * @param args the schema file to be compiled and where this should be written
+ */
+ public static void main(String[] args) {
+ try {
+ if (args.length < 2) {
+ log.error("Usage: Compiler <mapping file> <output dir>");
+ System.exit(1);
+ }
+ compileSchema(new File(args[0]), new File(args[1]));
+ } catch (Exception e) {
+ log.error("Something went wrong. Please check the input file.", e.getMessage());
+ throw new RuntimeException(e);
+ }
+ }
+
+ /**
+ * Generates Java classes for a schema.
+ */
+ private static void compileSchema(File src, File dest) throws Exception {
--- End diff --
This is great. Thanks for adding compiler native data bean generation. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: Initial commit gora cassandra cql
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r122808264
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java ---
@@ -0,0 +1,516 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.store;
+
+import com.datastax.driver.core.*;
+import com.datastax.driver.core.policies.*;
+import org.apache.gora.persistency.BeanFactory;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.hadoop.conf.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.List;
+import java.util.Properties;
+
+public class CassandraStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ /**
+ * Consistency property level for Cassandra column families
+ */
+ private static final String COL_FAM_CL = "cf.consistency.level";
+
+ /**
+ * Consistency property level for Cassandra read operations.
+ */
+ private static final String READ_OP_CL = "read.consistency.level";
+
+ /**
+ * Consistency property level for Cassandra write operations.
+ */
+ private static final String WRITE_OP_CL = "write.consistency.level";
+
+ public static final Logger LOG = LoggerFactory.getLogger(CassandraStore.class);
+
+ private Cluster cluster;
+
+ private Session session;
+
+ public CassandraStore() {
+ super();
+ }
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) {
+ LOG.debug("Initializing Cassandra store");
+ super.initialize(keyClass, persistentClass, properties);
+ try {
+ Cluster.Builder builder = Cluster.builder();
+ populateSettings(builder, properties);
+ this.cluster = builder.build();
+ this.session = this.cluster.connect();
+ } catch (Exception e) {
+ LOG.error("Error while initializing Cassandra store: {}",
+ new Object[]{e.getMessage()});
+ throw new RuntimeException(e);
+ }
+ }
+
+ private void populateSettings(Cluster.Builder builder, Properties properties) {
+ String serversParam = properties.getProperty(Constants.CASSANDRA_SERVERS);
+ String[] servers = serversParam.split(",");
+ for (String server : servers) {
+ builder = builder.addContactPoint(server);
+ }
+ String portProp = properties.getProperty(Constants.PORT);
+ if (portProp != null) {
+ builder = builder.withPort(Integer.parseInt(portProp));
+ }
+ String clusterNameProp = properties.getProperty(Constants.CLUSTER_NAME);
+ if (clusterNameProp != null) {
+ builder = builder.withClusterName(clusterNameProp);
+ }
+ String compressionProp = properties.getProperty(Constants.COMPRESSION);
+ if (compressionProp != null) {
+ builder = builder.withCompression(ProtocolOptions.Compression.valueOf(compressionProp));
+ }
+ builder = this.populateCredentials(properties, builder);
+ builder = this.populateLoadBalancingProp(properties, builder);
+ String enableJMXProp = properties.getProperty(Constants.ENABLE_JMX_REPORTING);
+ if (enableJMXProp != null) {
+ if (!Boolean.parseBoolean(enableJMXProp)) {
+ builder = builder.withoutJMXReporting();
+ }
+ }
+ String enableMetricsProp = properties.getProperty(Constants.ENABLE_METRICS);
+ if (enableMetricsProp != null) {
+ if (!Boolean.parseBoolean(enableMetricsProp)) {
+ builder = builder.withoutMetrics();
+ }
+ }
+ builder = this.populatePoolingSettings(properties, builder);
+ String versionProp = properties.getProperty(Constants.PROTOCOL_VERSION);
+ if (versionProp != null) {
+ builder = builder.withProtocolVersion(ProtocolVersion.fromInt(Integer.parseInt(versionProp)));
+ }
+ builder = this.populateQueryOptions(properties, builder);
+ builder = this.populateReconnectPolicy(properties, builder);
+ builder = this.populateRetrytPolicy(properties, builder);
+ builder = this.populateSocketOptions(properties, builder);
+ String enableSSLProp = properties.getProperty(Constants.ENABLE_SSL);
+ if (enableSSLProp != null) {
+ if (Boolean.parseBoolean(enableSSLProp)) {
+ builder = builder.withSSL();
+ }
+ }
+ }
+
--- End diff --
Remove extra line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: Initial commit gora cassandra cql
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r122811939
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/GoraCassandraTestDriver.java ---
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * @author lewismc
+ *
+ */
+
+package org.apache.gora.cassandra;
+
+import org.apache.gora.GoraTestDriver;
+
+import java.io.File;
+
+import org.apache.cassandra.io.util.FileUtils;
+import org.apache.cassandra.service.CassandraDaemon;
+
+// Logging imports
+import org.apache.gora.cassandra.store.CassandraStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Helper class for third party tests using gora-cassandra backend.
+ * @see GoraTestDriver for test specifics.
+ * This driver is the base for all test cases that require an embedded Cassandra
+ * server. In this case we draw on Hector's @see EmbeddedServerHelper.
+ * It starts (setUp) and stops (tearDown) embedded Cassandra server.
+ *
+ * @author lewismc
--- End diff --
Please just remove the author tag here please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r129360056
--- Diff: gora-cassandra-cql/src/examples/java/org/apache/gora/cassandra/example/generated/avroSerialization/CassandraKey.java ---
@@ -0,0 +1,380 @@
+/**
+ *Licensed to the Apache Software Foundation (ASF) under one
+ *or more contributor license agreements. See the NOTICE file
+ *distributed with this work for additional information
+ *regarding copyright ownership. The ASF licenses this file
+ *to you under the Apache License, Version 2.0 (the"
+ *License"); you may not use this file except in compliance
+ *with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *Unless required by applicable law or agreed to in writing, software
+ *distributed under the License is distributed on an "AS IS" BASIS,
+ *WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *See the License for the specific language governing permissions and
+ *limitations under the License.
+ */
+package org.apache.gora.cassandra.example.generated.avroSerialization;
--- End diff --
Please recompile these schemas with updates as described above
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r123814516
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/GoraCassandraTestDriver.java ---
@@ -148,7 +156,7 @@ public void cleanupDirectoriesFailover() {
} catch (Exception e) {
// ignore exception
try {
- Thread.sleep(250);
+ Thread.sleep(2500);
--- End diff --
Why is is longer? Is this for debugging only?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r130026846
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/GoraCassandraTestDriver.java ---
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * @author lewismc
+ *
+ */
+
+package org.apache.gora.cassandra;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.apache.cassandra.io.util.FileUtils;
+import org.apache.cassandra.service.CassandraDaemon;
+import org.apache.gora.GoraTestDriver;
+import org.apache.gora.cassandra.store.CassandraStore;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.store.DataStore;
+import org.apache.gora.store.DataStoreFactory;
+import org.apache.gora.util.GoraException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.util.Properties;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
--- End diff --
Please remove these unused imports.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r124046147
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/bean/CassandraKey.java ---
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.bean;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * This Class represents the Cassandra Key.
+ */
+public class CassandraKey{
+
+ private String name;
+
+ private List<ClusterKeyField> clusterKeyFields;
+
+ private List<PartitionKeyField> partitionKeyFields;
+
+
+ public CassandraKey(String name) {
+ this.name = name;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public List<ClusterKeyField> getClusterKeyFields() {
+ return clusterKeyFields;
+ }
+
+ public List<PartitionKeyField> getPartitionKeyFields() {
+ return partitionKeyFields;
+ }
+
+ public void addPartitionKeyField(PartitionKeyField partitionKeyField) {
--- End diff --
Please document these public methods for all cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by madhawa-gunasekara <gi...@git.apache.org>.
Github user madhawa-gunasekara commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r123604947
--- Diff: gora-cassandra-cql/src/test/conf/gora.properties ---
@@ -0,0 +1,30 @@
+#
--- End diff --
:+1:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r132824244
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/store/TestNativeSerializationWithUDT.java ---
@@ -0,0 +1,84 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.store;
+
+import org.apache.gora.cassandra.GoraCassandraTestDriver;
+import org.apache.gora.cassandra.example.generated.nativeSerialization.Customer;
+import org.apache.gora.cassandra.example.generated.nativeSerialization.Document;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.util.Properties;
+
+public class TestNativeSerializationWithUDT {
+
+ private static GoraCassandraTestDriver testDriver = new GoraCassandraTestDriver();
+ private static CassandraStore<String, Document> documentCassandraStore;
+ private static Properties parameter;
+
+ @BeforeClass
+ public static void setUpClass() throws Exception {
+ setProperties();
+ testDriver.setParameters(parameter);
+ testDriver.setUpClass();
+ documentCassandraStore = (CassandraStore<String, Document>) testDriver.createDataStore(String.class, Document.class);
+ }
+
+ private static void setProperties() {
+ parameter = new Properties();
+ parameter.setProperty(CassandraStoreParameters.CASSANDRA_SERVERS, "localhost");
+ parameter.setProperty(CassandraStoreParameters.PORT, "9042");
+ parameter.setProperty(CassandraStoreParameters.CASSANDRA_SERIALIZATION_TYPE, "native");
+ parameter.setProperty(CassandraStoreParameters.PROTOCOL_VERSION, "3");
+ parameter.setProperty(CassandraStoreParameters.CLUSTER_NAME, "Test Cluster");
+ parameter.setProperty("gora.cassandrastore.mapping.file", "nativeUDT/gora-cassandra-mapping.xml");
+ }
+
+ @AfterClass
+ public static void tearDownClass() throws Exception {
+ testDriver.tearDownClass();
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ testDriver.tearDown();
+ }
+
+ /**
+ * This is for testGetNested() with UDT dataType with native serialization.
+ */
+ @Test
+ public void testSimplePutAndGEt() {
--- End diff --
This is great. Thanks for adding Native nested test cases :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r130070953
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/bean/CassandraKey.java ---
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.bean;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * This Class represents the Cassandra Key.
+ */
+public class CassandraKey{
+
+ private String name;
+
+ private List<ClusterKeyField> clusterKeyFields;
+
+ private List<PartitionKeyField> partitionKeyFields;
+
+ private List<Field> fieldList;
+
+ public CassandraKey(String name) {
+ this.name = name;
+ this.fieldList = new ArrayList<>();
+ this.partitionKeyFields = new ArrayList<>();
+ }
+
+ public String getName() {
+ return this.name;
+ }
+
+ public List<ClusterKeyField> getClusterKeyFields() {
+ return this.clusterKeyFields;
+ }
+
+ public List<PartitionKeyField> getPartitionKeyFields() {
+ return this.partitionKeyFields;
+ }
+
+ public void addPartitionKeyField(PartitionKeyField partitionKeyField) {
--- End diff --
Please document these public methods all possible cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r129360462
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/serializers/AvroCassandraUtils.java ---
@@ -0,0 +1,274 @@
+package org.apache.gora.cassandra.serializers;
--- End diff --
Add license header
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r132824405
--- Diff: gora-cassandra-cql/src/test/conf/gora.properties ---
@@ -0,0 +1,27 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+gora.datastore.default=org.apache.gora.cassandra.CassandraStore
+gora.cassandrastore.cluster=Gora Cassandra Test Cluster
+gora.cassandrastore.host=localhost:9160
+# property is annotated in CassandraClient#checkKeyspace()
+# options are ANY, ONE, TWO, THREE, LOCAL_QUORUM, EACH_QUORUM, QUORUM and ALL.
+
+
--- End diff --
Better if we can remove this unnecessary new lines.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r132824225
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/store/TestNativeSerializationWithUDT.java ---
@@ -0,0 +1,84 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.store;
+
+import org.apache.gora.cassandra.GoraCassandraTestDriver;
+import org.apache.gora.cassandra.example.generated.nativeSerialization.Customer;
+import org.apache.gora.cassandra.example.generated.nativeSerialization.Document;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.util.Properties;
+
+public class TestNativeSerializationWithUDT {
--- End diff --
Please add class level java doc?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r124048997
--- Diff: gora-cassandra-cql/pom.xml ---
@@ -0,0 +1,232 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora</artifactId>
+ <version>0.8-SNAPSHOT</version>
+ <relativePath>../</relativePath>
+ </parent>
+ <artifactId>gora-cassandra-cql</artifactId>
+ <packaging>bundle</packaging>
+
+ <name>Apache Gora :: Cassandra - CQL</name>
+ <url>http://gora.apache.org</url>
+ <description>The Apache Gora open source framework provides an in-memory data model and
+ persistence for big data. Gora supports persisting to column stores, key value stores,
+ document stores and RDBMSs, and analyzing the data with extensive Apache Hadoop MapReduce
+ support.
+ </description>
+ <inceptionYear>2010</inceptionYear>
+ <organization>
+ <name>The Apache Software Foundation</name>
+ <url>http://www.apache.org/</url>
+ </organization>
+ <issueManagement>
+ <system>JIRA</system>
+ <url>https://issues.apache.org/jira/browse/GORA</url>
+ </issueManagement>
+ <ciManagement>
+ <system>Jenkins</system>
+ <url>https://builds.apache.org/job/Gora-trunk/</url>
+ </ciManagement>
+
+ <properties>
+ <guava.version>18.0</guava.version>
+ <osgi.import>*</osgi.import>
+ <osgi.export>org.apache.gora.cassandra*;version="${project.version}";-noimport:=true</osgi.export>
+ </properties>
+
+ <build>
+ <directory>target</directory>
+ <outputDirectory>target/classes</outputDirectory>
+ <finalName>${project.artifactId}-${project.version}</finalName>
+ <testOutputDirectory>target/test-classes</testOutputDirectory>
+ <testSourceDirectory>src/test/java</testSourceDirectory>
+ <sourceDirectory>src/main/java</sourceDirectory>
+ <testResources>
+ <testResource>
+ <directory>${project.basedir}/src/test/conf</directory>
+ <includes>
+ <include>**/*</include>
+ </includes>
+ <!--targetPath>${project.basedir}/target/classes/</targetPath-->
+ </testResource>
+ </testResources>
+ <plugins>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>build-helper-maven-plugin</artifactId>
+ <version>${build-helper-maven-plugin.version}</version>
+ <executions>
+ <execution>
+ <phase>generate-sources</phase>
+ <goals>
+ <goal>add-source</goal>
+ </goals>
+ <configuration>
+ <sources>
+ <source>src/examples/java</source>
+ </sources>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-plugin</artifactId>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.maven.surefire</groupId>
+ <artifactId>surefire-junit47</artifactId>
+ <version>2.20</version>
+ </dependency>
+ </dependencies>
+ <configuration>
+ <forkMode>always</forkMode>
+ <argLine>-Xmx1024m -XX:MaxPermSize=512m</argLine>
+ </configuration>
+ </plugin>
+ </plugins>
+ </build>
+
+ <dependencies>
+
+ <!-- OSX JDK 7 -->
+ <!-- should be removed once fixed in Cassandra -->
+ <dependency>
+ <groupId>org.xerial.snappy</groupId>
+ <artifactId>snappy-java</artifactId>
+ <version>1.0.5-M3</version>
+ <scope>test</scope>
+ </dependency>
+
+ <!-- Gora Internal Dependencies -->
+ <dependency>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora-core</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora-core</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+
+ <!-- Cassandra Driver Dependencies -->
+ <dependency>
+ <groupId>com.datastax.cassandra</groupId>
+ <artifactId>cassandra-driver-core</artifactId>
+ <version>${cassandra-driver.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>com.datastax.cassandra</groupId>
+ <artifactId>cassandra-driver-mapping</artifactId>
+ <version>${cassandra-driver.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>com.datastax.cassandra</groupId>
+ <artifactId>cassandra-driver-extras</artifactId>
+ <version>${cassandra-driver.version}</version>
+ </dependency>
+
+ <!-- Cassandra Dependencies -->
+ <dependency>
+ <groupId>org.apache.cassandra</groupId>
+ <artifactId>cassandra-all</artifactId>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.cassandra.deps</groupId>
+ <artifactId>avro</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-log4j12</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-handler</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>log4j-over-slf4j</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>io.netty</groupId>
--- End diff --
Do we have this Netty dependency defined root parent pom? This Netty versions should be defined there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r130016679
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/store/CassandraClient.java ---
@@ -0,0 +1,501 @@
+package org.apache.gora.cassandra.store;
--- End diff --
Please add License header here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r130026243
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/persistent/CassandraNativePersistent.java ---
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.persistent;
+
+import com.datastax.driver.mapping.annotations.Transient;
+import org.apache.avro.Schema;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.Tombstone;
+import org.apache.gora.persistency.impl.PersistentBase;
+
+import java.util.List;
+
+/**
+ * This class should be used with Native Cassandra Serialization.
+ */
+public abstract class CassandraNativePersistent implements Persistent {
+ @Transient
--- End diff --
Do we have a mechanism to generate these native persistent data beans which extends CassandraNativePersistent?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora issue #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by madhawa-gunasekara <gi...@git.apache.org>.
Github user madhawa-gunasekara commented on the issue:
https://github.com/apache/gora/pull/110
@lewismc , @renato2099
I have used Persistent object for Native Serialization, I have added tests for Native and Avro serialization and also for the composite Cassandra keys as well.
Currently, I'm working on writing a compiler for native serialization objects.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r129359438
--- Diff: gora-cassandra-cql/src/examples/avro/cassandraKey.json ---
@@ -0,0 +1,21 @@
+{
+ "type": "record",
+ "name": "CassandraKey",
+ "default": null,
--- End diff --
Can you provide a 'doc' field as well?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r124047745
--- Diff: gora-cassandra-cql/src/test/java/.gitignore ---
@@ -0,0 +1,15 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
--- End diff --
Is this file added intentionally?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r130027626
--- Diff: gora-cassandra-cql/src/test/conf/nativeSerialization/gora.properties ---
@@ -0,0 +1,26 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+gora.datastore.default=org.apache.gora.cassandra.CassandraStore
+gora.cassandrastore.cluster=Gora Test Cluster
+gora.cassandrastore.host=localhost:9160
+
+
+
--- End diff --
Please remove this unnecessary new lines.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by madhawa-gunasekara <gi...@git.apache.org>.
Github user madhawa-gunasekara commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r126297972
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/bean/CassandraKey.java ---
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.bean;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * This Class represents the Cassandra Key.
+ */
+public class CassandraKey{
+
+ private String name;
+
+ private List<ClusterKeyField> clusterKeyFields;
+
+ private List<PartitionKeyField> partitionKeyFields;
+
+
+ public CassandraKey(String name) {
+ this.name = name;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public List<ClusterKeyField> getClusterKeyFields() {
+ return clusterKeyFields;
+ }
+
+ public List<PartitionKeyField> getPartitionKeyFields() {
+ return partitionKeyFields;
+ }
+
+ public void addPartitionKeyField(PartitionKeyField partitionKeyField) {
--- End diff --
:+1:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by madhawa-gunasekara <gi...@git.apache.org>.
Github user madhawa-gunasekara commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r126297986
--- Diff: gora-cassandra-cql/src/test/java/org/apache/gora/cassandra/store/TestCassandraStoreWithNativeSerialization.java ---
@@ -0,0 +1,117 @@
+package org.apache.gora.cassandra.store;
--- End diff --
:+1: Added the license header
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r124046726
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/serializers/CassandraNativePersistent.java ---
@@ -0,0 +1,91 @@
+package org.apache.gora.cassandra.serializers;
--- End diff --
Please add Apache license header?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: [WIP] GORA-497 : Rewrite Cassandra data store
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r123813981
--- Diff: gora-cassandra-cql/pom.xml ---
@@ -49,7 +49,7 @@
</ciManagement>
<properties>
- <guava.version>17.0</guava.version>
+ <guava.version>18.0</guava.version>
--- End diff --
Please make sure this dependency is present in parent ```pom.xml```. It is an important dependency which is notorious for non backwards API changes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] gora pull request #110: Initial commit gora cassandra cql
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/110#discussion_r122809055
--- Diff: gora-cassandra-cql/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java ---
@@ -0,0 +1,516 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gora.cassandra.store;
+
+import com.datastax.driver.core.*;
+import com.datastax.driver.core.policies.*;
+import org.apache.gora.persistency.BeanFactory;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.hadoop.conf.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.List;
+import java.util.Properties;
+
+public class CassandraStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ /**
+ * Consistency property level for Cassandra column families
+ */
+ private static final String COL_FAM_CL = "cf.consistency.level";
+
+ /**
+ * Consistency property level for Cassandra read operations.
+ */
+ private static final String READ_OP_CL = "read.consistency.level";
+
+ /**
+ * Consistency property level for Cassandra write operations.
+ */
+ private static final String WRITE_OP_CL = "write.consistency.level";
+
+ public static final Logger LOG = LoggerFactory.getLogger(CassandraStore.class);
+
+ private Cluster cluster;
+
+ private Session session;
+
+ public CassandraStore() {
+ super();
+ }
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) {
+ LOG.debug("Initializing Cassandra store");
+ super.initialize(keyClass, persistentClass, properties);
+ try {
+ Cluster.Builder builder = Cluster.builder();
+ populateSettings(builder, properties);
+ this.cluster = builder.build();
+ this.session = this.cluster.connect();
+ } catch (Exception e) {
+ LOG.error("Error while initializing Cassandra store: {}",
+ new Object[]{e.getMessage()});
+ throw new RuntimeException(e);
+ }
+ }
+
+ private void populateSettings(Cluster.Builder builder, Properties properties) {
+ String serversParam = properties.getProperty(Constants.CASSANDRA_SERVERS);
+ String[] servers = serversParam.split(",");
+ for (String server : servers) {
+ builder = builder.addContactPoint(server);
+ }
+ String portProp = properties.getProperty(Constants.PORT);
+ if (portProp != null) {
+ builder = builder.withPort(Integer.parseInt(portProp));
+ }
+ String clusterNameProp = properties.getProperty(Constants.CLUSTER_NAME);
+ if (clusterNameProp != null) {
+ builder = builder.withClusterName(clusterNameProp);
+ }
+ String compressionProp = properties.getProperty(Constants.COMPRESSION);
+ if (compressionProp != null) {
+ builder = builder.withCompression(ProtocolOptions.Compression.valueOf(compressionProp));
+ }
+ builder = this.populateCredentials(properties, builder);
+ builder = this.populateLoadBalancingProp(properties, builder);
+ String enableJMXProp = properties.getProperty(Constants.ENABLE_JMX_REPORTING);
+ if (enableJMXProp != null) {
+ if (!Boolean.parseBoolean(enableJMXProp)) {
+ builder = builder.withoutJMXReporting();
+ }
+ }
+ String enableMetricsProp = properties.getProperty(Constants.ENABLE_METRICS);
+ if (enableMetricsProp != null) {
+ if (!Boolean.parseBoolean(enableMetricsProp)) {
+ builder = builder.withoutMetrics();
+ }
+ }
+ builder = this.populatePoolingSettings(properties, builder);
+ String versionProp = properties.getProperty(Constants.PROTOCOL_VERSION);
+ if (versionProp != null) {
+ builder = builder.withProtocolVersion(ProtocolVersion.fromInt(Integer.parseInt(versionProp)));
+ }
+ builder = this.populateQueryOptions(properties, builder);
+ builder = this.populateReconnectPolicy(properties, builder);
+ builder = this.populateRetrytPolicy(properties, builder);
+ builder = this.populateSocketOptions(properties, builder);
+ String enableSSLProp = properties.getProperty(Constants.ENABLE_SSL);
+ if (enableSSLProp != null) {
+ if (Boolean.parseBoolean(enableSSLProp)) {
+ builder = builder.withSSL();
+ }
+ }
+ }
+
+
+ private Cluster.Builder populateLoadBalancingProp(Properties properties, Cluster.Builder builder) {
+ String loadBalancingProp = properties.getProperty(Constants.LOAD_BALANCING_POLICY);
+ if (loadBalancingProp != null) {
+ switch (loadBalancingProp) {
+ case "LatencyAwareRoundRobinPolicy":
+ builder = builder.withLoadBalancingPolicy(LatencyAwarePolicy.builder(new RoundRobinPolicy()).build());
+ break;
+ case "RoundRobinPolicy":
+ builder = builder.withLoadBalancingPolicy(new RoundRobinPolicy());
+ break;
+ case "DCAwareRoundRobinPolicy": {
+ String dataCenter = properties.getProperty(Constants.DATA_CENTER);
+ boolean allowRemoteDCsForLocalConsistencyLevel = Boolean.parseBoolean(
+ properties.getProperty(Constants.ALLOW_REMOTE_DCS_FOR_LOCAL_CONSISTENCY_LEVEL));
+ if (dataCenter != null && !dataCenter.isEmpty()) {
+ if (allowRemoteDCsForLocalConsistencyLevel) {
+ builder = builder.withLoadBalancingPolicy(
+ DCAwareRoundRobinPolicy.builder().withLocalDc(dataCenter)
+ .allowRemoteDCsForLocalConsistencyLevel().build());
+ } else {
+ builder = builder.withLoadBalancingPolicy(
+ DCAwareRoundRobinPolicy.builder().withLocalDc(dataCenter).build());
+ }
+ } else {
+ if (allowRemoteDCsForLocalConsistencyLevel) {
+ builder = builder.withLoadBalancingPolicy(
+ (DCAwareRoundRobinPolicy.builder().allowRemoteDCsForLocalConsistencyLevel().build()));
+ } else {
+ builder = builder.withLoadBalancingPolicy((DCAwareRoundRobinPolicy.builder().build()));
+ }
+ }
+ break;
+ }
+ case "TokenAwareRoundRobinPolicy":
+ builder = builder.withLoadBalancingPolicy(new TokenAwarePolicy(new RoundRobinPolicy()));
+ break;
+ case "TokenAwareDCAwareRoundRobinPolicy": {
+ String dataCenter = properties.getProperty(Constants.DATA_CENTER);
+ boolean allowRemoteDCsForLocalConsistencyLevel = Boolean.parseBoolean(
+ properties.getProperty(Constants.ALLOW_REMOTE_DCS_FOR_LOCAL_CONSISTENCY_LEVEL));
+ if (dataCenter != null && !dataCenter.isEmpty()) {
+ if (allowRemoteDCsForLocalConsistencyLevel) {
+ builder = builder.withLoadBalancingPolicy(new TokenAwarePolicy(
+ DCAwareRoundRobinPolicy.builder().withLocalDc(dataCenter)
+ .allowRemoteDCsForLocalConsistencyLevel().build()));
+ } else {
+ builder = builder.withLoadBalancingPolicy(new TokenAwarePolicy(
+ DCAwareRoundRobinPolicy.builder().withLocalDc(dataCenter).build()));
+ }
+ } else {
+ if (allowRemoteDCsForLocalConsistencyLevel) {
+ builder = builder.withLoadBalancingPolicy(new TokenAwarePolicy(
+ DCAwareRoundRobinPolicy.builder().allowRemoteDCsForLocalConsistencyLevel().build()));
+ } else {
+ builder = builder.withLoadBalancingPolicy(
+ new TokenAwarePolicy(DCAwareRoundRobinPolicy.builder().build()));
+ }
+ }
+ break;
+ }
+ default:
+ LOG.error("Unsupported Cassandra load balancing " + "policy: " + loadBalancingProp);
+ break;
+ }
+ }
+ return builder;
+ }
+
+ private Cluster.Builder populateCredentials(Properties properties, Cluster.Builder builder) {
+ String usernameProp = properties.getProperty(Constants.USERNAME);
+ String passwordProp = properties.getProperty(Constants.PASSWORD);
+ if (usernameProp != null) {
+ builder = builder.withCredentials(usernameProp, passwordProp);
+ }
+ return builder;
+ }
+
+ private Cluster.Builder populatePoolingSettings(Properties properties, Cluster.Builder builder) {
+ String localCoreConnectionsPerHost = properties.getProperty(Constants.LOCAL_CORE_CONNECTIONS_PER_HOST);
+ String remoteCoreConnectionsPerHost = properties.getProperty(Constants.REMOTE_CORE_CONNECTIONS_PER_HOST);
+ String localMaxConnectionsPerHost = properties.getProperty(Constants.LOCAL_MAX_CONNECTIONS_PER_HOST);
+ String remoteMaxConnectionsPerHost = properties.getProperty(Constants.REMOTE_MAX_CONNECTIONS_PER_HOST);
+ String localNewConnectionThreshold = properties.getProperty(Constants.LOCAL_NEW_CONNECTION_THRESHOLD);
+ String remoteNewConnectionThreshold = properties.getProperty(Constants.REMOTE_NEW_CONNECTION_THRESHOLD);
+ String localMaxRequestsPerConnection = properties.getProperty(Constants.LOCAL_MAX_REQUESTS_PER_CONNECTION);
+ String remoteMaxRequestsPerConnection = properties.getProperty(Constants.REMOTE_MAX_REQUESTS_PER_CONNECTION);
+ PoolingOptions options = new PoolingOptions();
+ if (localCoreConnectionsPerHost != null) {
+ options.setCoreConnectionsPerHost(HostDistance.LOCAL, Integer.parseInt(localCoreConnectionsPerHost));
+ }
+ if (remoteCoreConnectionsPerHost != null) {
+ options.setCoreConnectionsPerHost(HostDistance.REMOTE, Integer.parseInt(remoteCoreConnectionsPerHost));
+ }
+ if (localMaxConnectionsPerHost != null) {
+ options.setMaxConnectionsPerHost(HostDistance.LOCAL, Integer.parseInt(localMaxConnectionsPerHost));
+ }
+ if (remoteMaxConnectionsPerHost != null) {
+ options.setMaxConnectionsPerHost(HostDistance.REMOTE, Integer.parseInt(remoteMaxConnectionsPerHost));
+ }
+ if (localNewConnectionThreshold != null) {
+ options.setNewConnectionThreshold(HostDistance.LOCAL, Integer.parseInt(localNewConnectionThreshold));
+ }
+ if (remoteNewConnectionThreshold != null) {
+ options.setNewConnectionThreshold(HostDistance.REMOTE, Integer.parseInt(remoteNewConnectionThreshold));
+ }
+ if (localMaxRequestsPerConnection != null) {
+ options.setMaxRequestsPerConnection(HostDistance.LOCAL, Integer.parseInt(localMaxRequestsPerConnection));
+ }
+ if (remoteMaxRequestsPerConnection != null) {
+ options.setMaxRequestsPerConnection(HostDistance.REMOTE, Integer.parseInt(remoteMaxRequestsPerConnection));
+ }
+ builder = builder.withPoolingOptions(options);
+ return builder;
+ }
+
+ private Cluster.Builder populateQueryOptions(Properties properties, Cluster.Builder builder) {
+ String consistencyLevelProp = properties.getProperty(Constants.CONSISTENCY_LEVEL);
+ String serialConsistencyLevelProp = properties.getProperty(Constants.SERIAL_CONSISTENCY_LEVEL);
+ String fetchSize = properties.getProperty(Constants.FETCH_SIZE);
+ QueryOptions options = new QueryOptions();
+ if (consistencyLevelProp != null) {
+ options.setConsistencyLevel(ConsistencyLevel.valueOf(consistencyLevelProp));
+ }
+ if (serialConsistencyLevelProp != null) {
+ options.setSerialConsistencyLevel(ConsistencyLevel.valueOf(serialConsistencyLevelProp));
+ }
+ if (fetchSize != null) {
+ options.setFetchSize(Integer.parseInt(fetchSize));
+ }
+ return builder.withQueryOptions(options);
+ }
+
+ private Cluster.Builder populateReconnectPolicy(Properties properties, Cluster.Builder builder) {
+ String reconnectionPolicy = properties.getProperty(Constants.RECONNECTION_POLICY);
+ if (reconnectionPolicy != null) {
+ switch (reconnectionPolicy) {
+ case "ConstantReconnectionPolicy": {
+ String constantReconnectionPolicyDelay = properties.getProperty(Constants.CONSTANT_RECONNECTION_POLICY_DELAY);
+
+ ConstantReconnectionPolicy policy = new ConstantReconnectionPolicy(Long.parseLong(constantReconnectionPolicyDelay));
+ builder = builder.withReconnectionPolicy(policy);
+ break;
+ }
+ case "ExponentialReconnectionPolicy": {
+ String exponentialReconnectionPolicyBaseDelay = properties.getProperty(Constants.EXPONENTIAL_RECONNECTION_POLICY_BASE_DELAY);
+ String exponentialReconnectionPolicyMaxDelay = properties.getProperty(Constants.EXPONENTIAL_RECONNECTION_POLICY_MAX_DELAY);
+
+ ExponentialReconnectionPolicy policy = new ExponentialReconnectionPolicy(Long.parseLong(exponentialReconnectionPolicyBaseDelay),
+ Long.parseLong(exponentialReconnectionPolicyMaxDelay));
+ builder = builder.withReconnectionPolicy(policy);
+ break;
+ }
+ default:
+ LOG.error("Unsupported reconnection policy " + reconnectionPolicy);
--- End diff --
Please use parameterized logging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---