You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gora.apache.org by carlosrmng <gi...@git.apache.org> on 2018/07/07 13:34:37 UTC
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
GitHub user carlosrmng opened a pull request:
https://github.com/apache/gora/pull/134
GORA-535 Add a data store for Apache Ignite
This PR adds the Apache Ignite data store to Gora.
Some issues with this PR:
-All DataStoreTestBase tests are passing.
-Documentation for the new backend is pending.
Code review welcome, thanks
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/carlosrmng/gora GORA-535
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/gora/pull/134.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #134
----
commit 4346216ffbed9a9adef2b90589436de543167c4c
Author: Carlos Rodrigo Muñoz <ca...@...>
Date: 2018-05-08T04:00:44Z
create template gora-ignite module
commit 7d671376a47a7251dad9df2e8176946b4ddb3420
Author: Carlos M <ca...@...>
Date: 2018-06-03T22:35:36Z
Merge branch 'master' of https://github.com/apache/gora into GORA-535
commit 7545170df8047ea6127660c1750e1d0b59044e49
Author: Carlos M <ca...@...>
Date: 2018-06-06T05:03:10Z
Add mapping reading and initialization for Ignite backend
Basic configuration and connection classes were implemented for the
Ignite backend. The basic skeleton of the new DataStore was initialized
using mapping files and some trivial methods were implemented
(#close(), #flush(), #getSchemaName(), ...). Also, an embembed Ignite
instance was included within the test packages in order to be used in
testing.
commit b1e2ae4c902691bbce02a90b76c4bee1c6e58455
Author: Carlos M <ca...@...>
Date: 2018-06-27T04:18:19Z
Implement schema handling methods
The basic schema creation/deletion methods are implemented for the
Ignite backend. In addition, all the SQL generation code was moved to
a separate class.
commit db7540e045fc07ef39d37163a263615c6e580efe
Author: Carlos M <ca...@...>
Date: 2018-07-02T05:50:43Z
Implement put, get, delete
The operations put, get and delete are implemented for the
Ignite backend. In addition, primary key mapping is
improved.
commit 87eddc2495cdeec9b28246ea08708540c365f34e
Author: Carlos M <ca...@...>
Date: 2018-07-06T04:11:38Z
Implement Queries for Ignite
The operations newQuery, execute, deleteByQuery, getPartitions are
implemented for the Ignite backend. The corresponding dependencies
IgniteQuery and IgniteResult are also included.
----
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202526754
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteMappingBuilder.java ---
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import com.google.inject.ConfigurationException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.jdom.Document;
+import org.jdom.Element;
+import org.jdom.JDOMException;
+import org.jdom.input.SAXBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Builder for Mapping definitions of Ignite.
+ */
+public class IgniteMappingBuilder<K, T extends PersistentBase> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IgniteMappingBuilder.class);
+ /**
+ * Mapping instance being built
+ */
+ private IgniteMapping igniteMapping;
+
+ private final IgniteStore<K, T> dataStore;
+
+ public IgniteMappingBuilder(final IgniteStore<K, T> store) {
+ this.igniteMapping = new IgniteMapping();
+ this.dataStore = store;
+ }
+
+ public IgniteMapping getIgniteMapping() {
+ return igniteMapping;
+ }
+
+ public void setIgniteMapping(IgniteMapping igniteMapping) {
+ this.igniteMapping = igniteMapping;
+ }
+
+ public void readMappingFile(String mappingFile) {
+ try {
+ SAXBuilder saxBuilder = new SAXBuilder();
+ InputStream inputStream = getClass().getClassLoader().getResourceAsStream(mappingFile);
+ if (inputStream == null) {
+ LOG.error("Mapping file '{}' could not be found!", mappingFile);
+ throw new IOException("Mapping file '" + mappingFile + "' could not be found!");
+ }
+ Document document = saxBuilder.build(inputStream);
+ if (document == null) {
+ LOG.error("Mapping file '{}' could not be found!", mappingFile);
+ throw new IOException("Mapping file '" + mappingFile + "' could not be found!");
+ }
+ List<Element> classes = document.getRootElement().getChildren("class");
--- End diff --
Type safety: The expression of type List needs unchecked conversion to conform to List < Element >
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202526747
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/utils/IgniteSQLBuilder.java ---
@@ -0,0 +1,238 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.utils;
+
+import avro.shaded.com.google.common.collect.Lists;
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+import java.text.MessageFormat;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
--- End diff --
Remove unused import
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211245661
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/utils/IgniteSQLBuilder.java ---
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.utils;
+
+import avro.shaded.com.google.common.collect.Lists;
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+import java.text.MessageFormat;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.gora.ignite.store.Column;
+import org.apache.gora.ignite.store.IgniteMapping;
+
+/**
+ *
+ * SQL Builder utility for Ignite.
+ */
+public class IgniteSQLBuilder {
--- End diff --
Since Ignite exposes SQL based database, is there any existing query builder available construct Ignite SQL queries? Otherwise writing them by concatenating string parts etc. likely to end up in query issues.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211529507
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteStore.java ---
@@ -0,0 +1,565 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.sql.rowset.CachedRowSet;
+import javax.sql.rowset.RowSetFactory;
+import javax.sql.rowset.RowSetProvider;
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificDatumReader;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.ignite.query.IgniteQuery;
+import org.apache.gora.ignite.query.IgniteResult;
+import org.apache.gora.ignite.utils.IgniteBackendConstants;
+import org.apache.gora.ignite.utils.IgniteSQLBuilder;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.AvroUtils;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of a Ignite data store to be used by gora.
+ *
+ * @param <K> class to be used for the key
+ * @param <T> class to be persisted within the store
+ */
+public class IgniteStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IgniteStore.class);
+ private static final String PARSE_MAPPING_FILE_KEY = "gora.ignite.mapping.file";
+ private static final String DEFAULT_MAPPING_FILE = "gora-ignite-mapping.xml";
+ private IgniteParameters igniteParameters;
+ private IgniteMapping igniteMapping;
+ private Connection connection;
+ private static final ConcurrentHashMap<Schema, SpecificDatumReader<?>> readerMap = new ConcurrentHashMap<>();
+ private static final ConcurrentHashMap<Schema, SpecificDatumWriter<?>> writerMap = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) throws GoraException {
+ try {
+ super.initialize(keyClass, persistentClass, properties);
+ IgniteMappingBuilder<K, T> builder = new IgniteMappingBuilder<K, T>(this);
+ builder.readMappingFile(getConf().get(PARSE_MAPPING_FILE_KEY, DEFAULT_MAPPING_FILE));
+ igniteMapping = builder.getIgniteMapping();
+ igniteParameters = IgniteParameters.load(properties);
+ connection = acquireConnection();
+ LOG.info("Ignite store was successfully initialized");
+ } catch (ClassNotFoundException | SQLException ex) {
+ LOG.error("Error while initializing Ignite store", ex);
+ throw new GoraException(ex);
+ }
+ }
+
+ private Connection acquireConnection() throws ClassNotFoundException, SQLException {
+ Class.forName("org.apache.ignite.IgniteJdbcThinDriver");
+ StringBuilder urlBuilder = new StringBuilder();
+ urlBuilder.append("jdbc:ignite:thin://");
+ urlBuilder.append(igniteParameters.getHost());
+ if (igniteParameters.getPort() != null) {
+ urlBuilder.append(":" + igniteParameters.getPort());
+ }
+ if (igniteParameters.getSchema() != null) {
+ urlBuilder.append("/" + igniteParameters.getSchema());
+ }
+ if (igniteParameters.getUser() != null) {
+ urlBuilder.append(";" + igniteParameters.getUser());
+ }
+ if (igniteParameters.getPassword() != null) {
+ urlBuilder.append(";" + igniteParameters.getPassword());
+ }
+ if (igniteParameters.getAdditionalConfigurations() != null) {
+ urlBuilder.append(igniteParameters.getAdditionalConfigurations());
+ }
+ return DriverManager.getConnection(urlBuilder.toString());
+ }
+
+ @Override
+ public String getSchemaName() {
+ return igniteMapping.getTableName();
+ }
+
+ @Override
+ public String getSchemaName(final String mappingSchemaName,
+ final Class<?> persistentClass) {
+ return super.getSchemaName(mappingSchemaName, persistentClass);
+ }
+
+ @Override
+ public void createSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to create the schema as no connection has been initiated.");
+ }
+ if (schemaExists()) {
+ return;
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String createTableSQL = IgniteSQLBuilder.createTable(igniteMapping);
+ stmt.executeUpdate(createTableSQL);
+ LOG.info("Table {} has been created for Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public void deleteSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to delete the schema as no connection has been initiated.");
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String dropTableSQL = IgniteSQLBuilder.dropTable(igniteMapping.getTableName());
+ stmt.executeUpdate(dropTableSQL);
+ LOG.info("Table {} has been dropped from Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public boolean schemaExists() throws GoraException {
+ try (Statement stmt = connection.createStatement()) {
+ String tableExistsSQL = IgniteSQLBuilder.tableExists(igniteMapping.getTableName());
+ ResultSet executeQuery = stmt.executeQuery(tableExistsSQL);
+ executeQuery.close();
+ return true;
+ } catch (SQLException ex) {
+ if (ex.getSQLState() != null
+ && ex.getSQLState().equals(IgniteBackendConstants.DEFAULT_IGNITE_TABLE_NOT_EXISTS_CODE)) {
+ return false;
+ } else {
+ throw new GoraException(ex);
+ }
+ }
+ }
+
+ @Override
+ public T get(K key, String[] fields) throws GoraException {
+ String[] avFields = getFieldsToQuery(fields);
+ Object[] keyl = null;
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ keyl = new Object[]{key};
+ } else {
+ //Composite key pending
+ }
+ //Avro fields to Ignite fields
+ List<String> dbFields = new ArrayList<>();
+ for (String af : avFields) {
+ dbFields.add(igniteMapping.getFields().get(af).getName());
+ }
+ String selectQuery = IgniteSQLBuilder.selectGet(igniteMapping, dbFields);
+ try (PreparedStatement stmt = connection.prepareStatement(selectQuery)) {
+ IgniteSQLBuilder.fillSelectStatement(stmt, igniteMapping, keyl);
+ ResultSet rs = stmt.executeQuery();
+ boolean data = rs.next();
+ T resp = null;
+ if (data) {
+ resp = newInstance(rs, fields);
+ if (rs.next()) {
+ LOG.warn("Multiple results for primary key {} in the schema {}, ignoring additional rows.", keyl, igniteMapping.getTableName());
+ }
+ }
+ rs.close();
+ return resp;
+ } catch (SQLException | IOException ex) {
+ throw new GoraException(ex);
+ }
+
+ }
+
+ public T newInstance(ResultSet rs, String[] fields) throws GoraException, SQLException, IOException {
+ fields = getFieldsToQuery(fields);
+ T persistent = newPersistent();
+ for (String f : fields) {
+ Schema.Field field = fieldMap.get(f);
+ Schema fieldSchema = field.schema();
+ String dbField = igniteMapping.getFields().get(f).getName();
+ Object sv = rs.getObject(dbField);
+ if (sv == null) {
+ continue;
+ }
+ Object v = deserializeFieldValue(field, fieldSchema, sv, persistent);
+ persistent.put(field.pos(), v);
+ persistent.setDirty(field.pos());
+ }
+ return persistent;
+ }
+
+ @SuppressWarnings("unchecked")
+ private Object deserializeFieldValue(Schema.Field field, Schema fieldSchema,
+ Object igniteValue, T persistent) throws IOException {
+ Object fieldValue = null;
+ switch (fieldSchema.getType()) {
+ case MAP:
+ case ARRAY:
+ case RECORD:
+ @SuppressWarnings("rawtypes") SpecificDatumReader reader = getDatumReader(fieldSchema);
+ fieldValue = IOUtils.deserialize((byte[]) igniteValue, reader,
+ persistent.get(field.pos()));
+ break;
+ case ENUM:
+ fieldValue = AvroUtils.getEnumValue(fieldSchema, igniteValue.toString());
+ break;
+ case FIXED:
+ break;
+ case BYTES:
+ fieldValue = ByteBuffer.wrap((byte[]) igniteValue);
+ break;
+ case STRING:
+ fieldValue = new Utf8(igniteValue.toString());
+ break;
+ case UNION:
+ if (fieldSchema.getTypes().size() == 2 && isNullable(fieldSchema)) {
+ int schemaPos = getUnionSchema(igniteValue, fieldSchema);
+ Schema unionSchema = fieldSchema.getTypes().get(schemaPos);
+ fieldValue = deserializeFieldValue(field, unionSchema, igniteValue, persistent);
+ } else {
+ reader = getDatumReader(fieldSchema);
+ fieldValue = IOUtils.deserialize((byte[]) igniteValue, reader,
+ persistent.get(field.pos()));
+ }
+ break;
+ default:
+ fieldValue = igniteValue;
+ }
+ return fieldValue;
+
+ }
+
+ @Override
+ public void put(K key, T obj) throws GoraException {
+ try {
+ if (obj.isDirty()) {
+ Schema schema = obj.getSchema();
+ List<Schema.Field> fields = schema.getFields();
+ Map<Column, Object> data = new HashMap<>();
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ Column getKey = igniteMapping.getPrimaryKey().get(0);
+ data.put(getKey, key);
+ } else {
+ //Composite keys pending..
+ }
+ for (Schema.Field field : fields) {
+ Column get = igniteMapping.getFields().get(field.name());
+ Object v = obj.get(field.pos());
+ if (get != null && v != null) {
+ Schema fieldSchema = field.schema();
+ Object serializedObj = serializeFieldValue(fieldSchema, v);
+ data.put(get, serializedObj);
+ }
+ }
+ String baseInsertStatement = IgniteSQLBuilder.baseInsertStatement(igniteMapping, data);
+ try (PreparedStatement stmt = connection.prepareStatement(baseInsertStatement)) {
+ IgniteSQLBuilder.fillInsertStatement(stmt, data);
+ stmt.executeUpdate();
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ } else {
+ LOG.info("Ignored putting object {} in the store as it is neither "
+ + "new, neither dirty.", new Object[]{obj});
+ }
+ } catch (Exception e) {
+ throw new GoraException(e);
+ }
+ }
+
+ @Override
+ public boolean delete(K key) throws GoraException {
+ String deleteQuery = null;
+ Object[] keyl = null;
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ deleteQuery = IgniteSQLBuilder.delete(igniteMapping);
+ keyl = new Object[]{key};
+ } else {
+ //Composite key pending
+ }
+ try (PreparedStatement stmt = connection.prepareStatement(deleteQuery)) {
+ IgniteSQLBuilder.fillDeleteStatement(stmt, igniteMapping, keyl);
+ stmt.executeUpdate();
+ return true;
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public long deleteByQuery(Query<K, T> query) throws GoraException {
+
+ String deleteQuery;
+ if (query.getFields() != null && query.getFields().length < igniteMapping.getFields().size()) {
+ List<String> dbFields = new ArrayList<>();
+ for (String af : query.getFields()) {
+ dbFields.add(igniteMapping.getFields().get(af).getName());
+ }
+ deleteQuery = IgniteSQLBuilder.deleteQueryFields(igniteMapping, dbFields);
+ } else {
+ deleteQuery = IgniteSQLBuilder.deleteQuery(igniteMapping);
+ }
+ String selectQueryWhere = IgniteSQLBuilder.queryWhere(igniteMapping, query.getStartKey(), query.getEndKey(), query.getLimit());
+ try (PreparedStatement stmt = connection.prepareStatement(deleteQuery + selectQueryWhere)) {
+ IgniteSQLBuilder.fillSelectQuery(stmt, query.getStartKey(), query.getEndKey());
+ stmt.executeUpdate();
+ return 0;
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public Result<K, T> execute(Query<K, T> query) throws GoraException {
+ String[] fields = getFieldsToQuery(query.getFields());
+ //Avro fields to Ignite fields
+ List<String> dbFields = new ArrayList<>();
+ for (String af : fields) {
+ dbFields.add(igniteMapping.getFields().get(af).getName());
+ }
+ String selectQuery = IgniteSQLBuilder.selectQuery(igniteMapping, dbFields);
+ String selectQueryWhere = IgniteSQLBuilder.queryWhere(igniteMapping, query.getStartKey(), query.getEndKey(), query.getLimit());
+ try (PreparedStatement stmt = connection.prepareStatement(selectQuery + selectQueryWhere)) {
+ RowSetFactory factory = RowSetProvider.newFactory();
+ CachedRowSet rowset = factory.createCachedRowSet();
+ IgniteSQLBuilder.fillSelectQuery(stmt, query.getStartKey(), query.getEndKey());
+ ResultSet executeQuery = stmt.executeQuery();
+ rowset.populate(executeQuery);
+ IgniteResult<K, T> igniteResult = new IgniteResult<>(this, query);
+ igniteResult.setResultSet(rowset);
+ return igniteResult;
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ public K extractKey(ResultSet r) throws SQLException {
+ assert igniteMapping.getPrimaryKey().size() == 1;
+ return (K) r.getObject(igniteMapping.getPrimaryKey().get(0).getName());
+ }
+
+ @Override
+ public Query<K, T> newQuery() {
+ IgniteQuery<K, T> query = new IgniteQuery<>(this);
+ query.setFields(getFieldsToQuery(null));
+ return query;
+ }
+
+ @Override
+ public List<PartitionQuery<K, T>> getPartitions(Query<K, T> query) throws IOException {
+ List<PartitionQuery<K, T>> partitions = new ArrayList<>();
+ PartitionQueryImpl<K, T> partitionQuery = new PartitionQueryImpl<>(
+ query);
+ partitionQuery.setConf(getConf());
+ partitions.add(partitionQuery);
+ return partitions;
+ }
+
+ @Override
+ public void flush() throws GoraException {
+ //Auto-commit mode by default
+ }
+
+ @Override
+ public void close() {
+ try {
+ connection.close();
+ LOG.info("Ignite datastore destroyed successfully.");
+ } catch (Exception ex) {
+ LOG.error(ex.getMessage(), ex);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ private Object serializeFieldValue(Schema fieldSchema, Object fieldValue) {
+ Object output = fieldValue;
+ switch (fieldSchema.getType()) {
+ case ARRAY:
+ case MAP:
+ case RECORD:
+ byte[] data = null;
+ try {
+ @SuppressWarnings("rawtypes")
+ SpecificDatumWriter writer = getDatumWriter(fieldSchema);
+ data = IOUtils.serialize(writer, fieldValue);
+ } catch (IOException e) {
+ LOG.error(e.getMessage(), e);
+ }
+ output = data;
+ break;
+ case UNION:
+ if (fieldSchema.getTypes().size() == 2 && isNullable(fieldSchema)) {
+ int schemaPos = getUnionSchema(fieldValue, fieldSchema);
+ Schema unionSchema = fieldSchema.getTypes().get(schemaPos);
+ output = serializeFieldValue(unionSchema, fieldValue);
+ } else {
+ data = null;
+ try {
+ @SuppressWarnings("rawtypes")
+ SpecificDatumWriter writer = getDatumWriter(fieldSchema);
+ data = IOUtils.serialize(writer, fieldValue);
+ } catch (IOException e) {
+ LOG.error(e.getMessage(), e);
+ }
+ output = data;
+ }
+ break;
+ case FIXED:
+ break;
+ case ENUM:
+ case STRING:
+ output = fieldValue.toString();
+ break;
+ case BYTES:
+ output = ((ByteBuffer) fieldValue).array();
+ break;
+ case INT:
+ case LONG:
+ case FLOAT:
+ case DOUBLE:
+ case BOOLEAN:
+ output = fieldValue;
+ break;
+ case NULL:
+ break;
+ default:
+ throw new AssertionError(fieldSchema.getType().name());
+
+ }
+ return output;
+ }
+
+ private boolean isNullable(Schema unionSchema) {
+ for (Schema innerSchema : unionSchema.getTypes()) {
+ if (innerSchema.getType().equals(Schema.Type.NULL)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Method to retrieve the corresponding schema type index of a particular
+ * object having UNION schema. As UNION type can have one or more types and at
+ * a given instance, it holds an object of only one type of the defined types,
+ * this method is used to figure out the corresponding instance's schema type
+ * index.
+ *
+ * @param instanceValue value that the object holds
+ * @param unionSchema union schema containing all of the data types
+ * @return the unionSchemaPosition corresponding schema position
+ */
+ private int getUnionSchema(Object instanceValue, Schema unionSchema) {
+ int unionSchemaPos = 0;
+ for (Schema currentSchema : unionSchema.getTypes()) {
+ Schema.Type schemaType = currentSchema.getType();
+ if (instanceValue instanceof CharSequence && schemaType.equals(Schema.Type.STRING)) {
+ return unionSchemaPos;
+ }
+ if (instanceValue instanceof ByteBuffer && schemaType.equals(Schema.Type.BYTES)) {
+ return unionSchemaPos;
+ }
+ if (instanceValue instanceof byte[] && schemaType.equals(Schema.Type.BYTES)) {
+ return unionSchemaPos;
+ }
+ if (instanceValue instanceof Integer && schemaType.equals(Schema.Type.INT)) {
+ return unionSchemaPos;
+ }
+ if (instanceValue instanceof Long && schemaType.equals(Schema.Type.LONG)) {
+ return unionSchemaPos;
+ }
+ if (instanceValue instanceof Double && schemaType.equals(Schema.Type.DOUBLE)) {
+ return unionSchemaPos;
+ }
+ if (instanceValue instanceof Float && schemaType.equals(Schema.Type.FLOAT)) {
+ return unionSchemaPos;
+ }
+ if (instanceValue instanceof Boolean && schemaType.equals(Schema.Type.BOOLEAN)) {
+ return unionSchemaPos;
+ }
+ if (instanceValue instanceof Map && schemaType.equals(Schema.Type.MAP)) {
+ return unionSchemaPos;
+ }
+ if (instanceValue instanceof List && schemaType.equals(Schema.Type.ARRAY)) {
+ return unionSchemaPos;
+ }
+ if (instanceValue instanceof Persistent && schemaType.equals(Schema.Type.RECORD)) {
+ return unionSchemaPos;
+ }
+ if (instanceValue instanceof byte[] && schemaType.equals(Schema.Type.MAP)) {
+ return unionSchemaPos;
+ }
+ if (instanceValue instanceof byte[] && schemaType.equals(Schema.Type.RECORD)) {
+ return unionSchemaPos;
+ }
+ if (instanceValue instanceof byte[] && schemaType.equals(Schema.Type.ARRAY)) {
+ return unionSchemaPos;
+ }
+ unionSchemaPos++;
+ }
+ return 0;
+ }
+
+ @SuppressWarnings("rawtypes")
+ private SpecificDatumReader getDatumReader(Schema fieldSchema) {
+ SpecificDatumReader<?> reader = readerMap.get(fieldSchema);
+ if (reader == null) {
+ reader = new SpecificDatumReader(fieldSchema);// ignore dirty bits
+ SpecificDatumReader localReader = null;
+ if ((localReader = readerMap.putIfAbsent(fieldSchema, reader)) != null) {
+ reader = localReader;
+ }
+ }
+ return reader;
+ }
+
+ @SuppressWarnings("rawtypes")
+ private SpecificDatumWriter getDatumWriter(Schema fieldSchema) {
+ SpecificDatumWriter writer = writerMap.get(fieldSchema);
+ if (writer == null) {
+ writer = new SpecificDatumWriter(fieldSchema);// ignore dirty bits
+ writerMap.put(fieldSchema, writer);
+ }
+
--- End diff --
Please remove unnecessary new lines.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by nishadi <gi...@git.apache.org>.
Github user nishadi commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202522672
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteParameters.java ---
@@ -0,0 +1,139 @@
+/*
+ * Copyright 2018 The Apache Software Foundation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.util.Properties;
+import org.apache.hadoop.conf.Configuration;
+
+/**
+ * Parameters definitions for Ignite.
+ */
+public class IgniteParameters {
+
+ /**
+ * Property indicating the Ignite Schema to be used
+ */
+ public static final String PROP_SCHEMA = "gora.datastore.ignite.schema";
+
+ /**
+ * Property indicating the Ignite Cluster Node IP
+ */
+ public static final String PROP_HOST = "gora.datastore.ignite.host";
+
+ /**
+ * Property indicating the port used by the Ignite Server
+ */
+ public static final String PROP_PORT = "gora.datastore.ignite.port";
+
+ /**
+ * Property indicating the username to connect to the server
+ */
+ public static final String PROP_USER = "gora.datastore.ignite.user";
+
+ /**
+ * Property indicating the password to connect to the server
+ */
+ public static final String PROP_PASSWORD = "gora.datastore.ignite.password";
+
+ /**
+ * Property indicating additional JDBC options
+ */
+ public static final String PROP_ADDITIONALS = "gora.datastore.ignite.additionalConfigurations";
+
+ private String host;
+ private String port;
+ private String schema;
+ private String user;
+ private String password;
+ private String additionalConfigurations;
+
+ /**
+ *
+ * @param host
+ * @param port Optional port for Ignite Server
+ * @param user Optional username for Ignite
--- End diff --
Shall we complete the java docs?
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202526796
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteStore.java ---
@@ -0,0 +1,578 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.sql.rowset.CachedRowSet;
+import javax.sql.rowset.RowSetFactory;
+import javax.sql.rowset.RowSetProvider;
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificDatumReader;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.ignite.query.IgniteQuery;
+import org.apache.gora.ignite.query.IgniteResult;
+import org.apache.gora.ignite.utils.IgniteSQLBuilder;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.AvroUtils;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of a Ignite data store to be used by gora.
+ *
+ * @param <K> class to be used for the key
+ * @param <T> class to be persisted within the store
+ */
+public class IgniteStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ public static final Logger LOG = LoggerFactory.getLogger(IgniteStore.class);
+ private static final String PARSE_MAPPING_FILE_KEY = "gora.ignite.mapping.file";
+ private static final String DEFAULT_MAPPING_FILE = "gora-ignite-mapping.xml";
+ private IgniteParameters igniteParameters;
+ private IgniteMapping igniteMapping;
+ private Connection connection;
+
+ /*
+ * Create a threadlocal map for the datum readers and writers, because they
+ * are not thread safe, at least not before Avro 1.4.0 (See AVRO-650). When
+ * they are thread safe, it is possible to maintain a single reader and writer
+ * pair for every schema, instead of one for every thread.
+ */
+ public static final ConcurrentHashMap<Schema, SpecificDatumReader<?>> readerMap = new ConcurrentHashMap<>();
+
+ public static final ConcurrentHashMap<Schema, SpecificDatumWriter<?>> writerMap = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) throws GoraException {
+
+ try {
+ super.initialize(keyClass, persistentClass, properties);
+ IgniteMappingBuilder builder = new IgniteMappingBuilder(this);
+ builder.readMappingFile(getConf().get(PARSE_MAPPING_FILE_KEY, DEFAULT_MAPPING_FILE));
+ igniteMapping = builder.getIgniteMapping();
+ igniteParameters = IgniteParameters.load(properties, conf);
+ connection = acquiereConnection();
+ LOG.info("Ignite store was successfully initialized");
+ } catch (ClassNotFoundException | SQLException ex) {
+ LOG.error("Error while initializing Ignite store", ex);
+ throw new GoraException(ex);
+ }
+ }
+
+ private Connection acquiereConnection() throws ClassNotFoundException, SQLException {
+ Class.forName("org.apache.ignite.IgniteJdbcThinDriver");
+ StringBuilder urlBuilder = new StringBuilder();
+ urlBuilder.append("jdbc:ignite:thin://");
+ urlBuilder.append(igniteParameters.getHost());
+ if (igniteParameters.getPort() != null) {
+ urlBuilder.append(":" + igniteParameters.getPort());
+ }
+ if (igniteParameters.getSchema() != null) {
+ urlBuilder.append("/" + igniteParameters.getSchema());
+ }
+ if (igniteParameters.getUser() != null) {
+ urlBuilder.append(";" + igniteParameters.getUser());
+ }
+ if (igniteParameters.getPassword() != null) {
+ urlBuilder.append(";" + igniteParameters.getPassword());
+ }
+ if (igniteParameters.getAdditionalConfigurations() != null) {
+ urlBuilder.append(igniteParameters.getAdditionalConfigurations());
+ }
+ Connection conn = DriverManager.getConnection(urlBuilder.toString());
+ return conn;
+ }
+
+ @Override
+ public String getSchemaName() {
+ return igniteMapping.getTableName();
+ }
+
+ @Override
+ public String getSchemaName(final String mappingSchemaName,
+ final Class<?> persistentClass) {
+ return super.getSchemaName(mappingSchemaName, persistentClass);
+ }
+
+ @Override
+ public void createSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to create the schema as no connection has been initiated.");
+ }
+ if (schemaExists()) {
+ return;
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String createTableSQL = IgniteSQLBuilder.createTable(igniteMapping);
+ stmt.executeUpdate(createTableSQL);
+ LOG.info("Table {} has been created for Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public void deleteSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to delete the schema as no connection has been initiated.");
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String dropTableSQL = IgniteSQLBuilder.dropTable(igniteMapping.getTableName());
+ stmt.executeUpdate(dropTableSQL);
+ LOG.info("Table {} has been dropped from Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public boolean schemaExists() throws GoraException {
+ boolean exists = false;
+ try (Statement stmt = connection.createStatement()) {
+ String tableExistsSQL = IgniteSQLBuilder.tableExists(igniteMapping.getTableName());
+ ResultSet executeQuery = stmt.executeQuery(tableExistsSQL);
+ executeQuery.close();
+ exists = true;
+ } catch (SQLException ex) {
+ /**
+ * a 42000 error code is thrown by Ignite when a non-existent table
+ * queried. More details:
+ * https://apacheignite-sql.readme.io/docs/jdbc-error-codes
+ */
+ if (ex.getSQLState() != null && ex.getSQLState().equals("42000")) {
+ exists = false;
+ } else {
+ throw new GoraException(ex);
+ }
+ }
+ return exists;
+ }
+
+ @Override
+ public T get(K key, String[] fields) throws GoraException {
+ String[] avFields = getFieldsToQuery(fields);
+ Object[] keyl = null;
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ keyl = new Object[]{key};
+ } else {
+ //Composite key pending
+ }
+ //Avro fields to Ignite fields
+ List<String> dbFields = new ArrayList<>();
+ for (String af : avFields) {
+ dbFields.add(igniteMapping.getFields().get(af).getName());
+ }
+ String selectQuery = IgniteSQLBuilder.selectGet(igniteMapping, dbFields);
+ try (PreparedStatement stmt = connection.prepareStatement(selectQuery)) {
+ IgniteSQLBuilder.fillSelectStatement(stmt, igniteMapping, keyl);
+ ResultSet rs = stmt.executeQuery();
+ boolean data = rs.next();
+ T resp = null;
+ if (data) {
+ resp = newInstance(rs, fields);
+ if (rs.next()) {
+ LOG.warn("Multiple results for primary key {} in the schema {}, ignoring additional rows.", keyl, igniteMapping.getTableName());
+ }
+ }
+ rs.close();
+ return resp;
+ } catch (SQLException | IOException ex) {
+ throw new GoraException(ex);
+ }
+
+ }
+
+ public T newInstance(ResultSet rs, String[] fields) throws GoraException, SQLException, IOException {
+ fields = getFieldsToQuery(fields);
+ T persistent = newPersistent();
+ for (String f : fields) {
+ Schema.Field field = fieldMap.get(f);
+ Schema fieldSchema = field.schema();
+ String dbField = igniteMapping.getFields().get(f).getName();
+ Object sv = rs.getObject(dbField);
+ if (sv == null) {
+ continue;
+ }
+ Object v = deserializeFieldValue(field, fieldSchema, sv, persistent);
+ persistent.put(field.pos(), v);
+ persistent.setDirty(field.pos());
+ }
+ return persistent;
+ }
+
+ private Object deserializeFieldValue(Schema.Field field, Schema fieldSchema,
+ Object igniteValue, T persistent) throws IOException {
+ Object fieldValue = null;
+ switch (fieldSchema.getType()) {
+ case MAP:
+ case ARRAY:
+ case RECORD:
+ @SuppressWarnings("rawtypes") SpecificDatumReader reader = getDatumReader(fieldSchema);
+ fieldValue = IOUtils.deserialize((byte[]) igniteValue, reader,
+ persistent.get(field.pos()));
+ break;
+ case ENUM:
+ fieldValue = AvroUtils.getEnumValue(fieldSchema, igniteValue.toString());
+ break;
+ case FIXED:
+ break;
+ case BYTES:
+ fieldValue = ByteBuffer.wrap((byte[]) igniteValue);
+ break;
+ case STRING:
+ fieldValue = new Utf8(igniteValue.toString());
+ break;
+ case UNION:
+ if (fieldSchema.getTypes().size() == 2 && isNullable(fieldSchema)) {
+ int schemaPos = getUnionSchema(igniteValue, fieldSchema);
+ Schema unionSchema = fieldSchema.getTypes().get(schemaPos);
+ fieldValue = deserializeFieldValue(field, unionSchema, igniteValue, persistent);
+ } else {
+ reader = getDatumReader(fieldSchema);
+ fieldValue = IOUtils.deserialize((byte[]) igniteValue, reader,
--- End diff --
Type safety: Unchecked invocation deserialize(byte[], SpecificDatumReader, Object) of the generic method deserialize(byte[], SpecificDatumReader< T >, T) of type IOUtils
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211538913
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteStore.java ---
@@ -0,0 +1,565 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.sql.rowset.CachedRowSet;
+import javax.sql.rowset.RowSetFactory;
+import javax.sql.rowset.RowSetProvider;
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificDatumReader;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.ignite.query.IgniteQuery;
+import org.apache.gora.ignite.query.IgniteResult;
+import org.apache.gora.ignite.utils.IgniteBackendConstants;
+import org.apache.gora.ignite.utils.IgniteSQLBuilder;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.AvroUtils;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of a Ignite data store to be used by gora.
+ *
+ * @param <K> class to be used for the key
+ * @param <T> class to be persisted within the store
+ */
+public class IgniteStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IgniteStore.class);
+ private static final String PARSE_MAPPING_FILE_KEY = "gora.ignite.mapping.file";
+ private static final String DEFAULT_MAPPING_FILE = "gora-ignite-mapping.xml";
+ private IgniteParameters igniteParameters;
+ private IgniteMapping igniteMapping;
+ private Connection connection;
+ private static final ConcurrentHashMap<Schema, SpecificDatumReader<?>> readerMap = new ConcurrentHashMap<>();
+ private static final ConcurrentHashMap<Schema, SpecificDatumWriter<?>> writerMap = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) throws GoraException {
+ try {
+ super.initialize(keyClass, persistentClass, properties);
+ IgniteMappingBuilder<K, T> builder = new IgniteMappingBuilder<K, T>(this);
+ builder.readMappingFile(getConf().get(PARSE_MAPPING_FILE_KEY, DEFAULT_MAPPING_FILE));
+ igniteMapping = builder.getIgniteMapping();
+ igniteParameters = IgniteParameters.load(properties);
+ connection = acquireConnection();
+ LOG.info("Ignite store was successfully initialized");
+ } catch (ClassNotFoundException | SQLException ex) {
+ LOG.error("Error while initializing Ignite store", ex);
+ throw new GoraException(ex);
+ }
+ }
+
+ private Connection acquireConnection() throws ClassNotFoundException, SQLException {
+ Class.forName("org.apache.ignite.IgniteJdbcThinDriver");
+ StringBuilder urlBuilder = new StringBuilder();
+ urlBuilder.append("jdbc:ignite:thin://");
+ urlBuilder.append(igniteParameters.getHost());
+ if (igniteParameters.getPort() != null) {
+ urlBuilder.append(":" + igniteParameters.getPort());
+ }
+ if (igniteParameters.getSchema() != null) {
+ urlBuilder.append("/" + igniteParameters.getSchema());
+ }
+ if (igniteParameters.getUser() != null) {
+ urlBuilder.append(";" + igniteParameters.getUser());
+ }
+ if (igniteParameters.getPassword() != null) {
+ urlBuilder.append(";" + igniteParameters.getPassword());
+ }
+ if (igniteParameters.getAdditionalConfigurations() != null) {
+ urlBuilder.append(igniteParameters.getAdditionalConfigurations());
+ }
+ return DriverManager.getConnection(urlBuilder.toString());
+ }
+
+ @Override
+ public String getSchemaName() {
+ return igniteMapping.getTableName();
+ }
+
+ @Override
+ public String getSchemaName(final String mappingSchemaName,
+ final Class<?> persistentClass) {
+ return super.getSchemaName(mappingSchemaName, persistentClass);
+ }
+
+ @Override
+ public void createSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to create the schema as no connection has been initiated.");
+ }
+ if (schemaExists()) {
+ return;
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String createTableSQL = IgniteSQLBuilder.createTable(igniteMapping);
+ stmt.executeUpdate(createTableSQL);
+ LOG.info("Table {} has been created for Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public void deleteSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to delete the schema as no connection has been initiated.");
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String dropTableSQL = IgniteSQLBuilder.dropTable(igniteMapping.getTableName());
+ stmt.executeUpdate(dropTableSQL);
+ LOG.info("Table {} has been dropped from Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public boolean schemaExists() throws GoraException {
+ try (Statement stmt = connection.createStatement()) {
+ String tableExistsSQL = IgniteSQLBuilder.tableExists(igniteMapping.getTableName());
+ ResultSet executeQuery = stmt.executeQuery(tableExistsSQL);
+ executeQuery.close();
+ return true;
+ } catch (SQLException ex) {
+ if (ex.getSQLState() != null
+ && ex.getSQLState().equals(IgniteBackendConstants.DEFAULT_IGNITE_TABLE_NOT_EXISTS_CODE)) {
+ return false;
+ } else {
+ throw new GoraException(ex);
+ }
+ }
+ }
+
+ @Override
+ public T get(K key, String[] fields) throws GoraException {
+ String[] avFields = getFieldsToQuery(fields);
+ Object[] keyl = null;
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ keyl = new Object[]{key};
+ } else {
+ //Composite key pending
+ }
+ //Avro fields to Ignite fields
+ List<String> dbFields = new ArrayList<>();
+ for (String af : avFields) {
+ dbFields.add(igniteMapping.getFields().get(af).getName());
+ }
+ String selectQuery = IgniteSQLBuilder.selectGet(igniteMapping, dbFields);
+ try (PreparedStatement stmt = connection.prepareStatement(selectQuery)) {
+ IgniteSQLBuilder.fillSelectStatement(stmt, igniteMapping, keyl);
+ ResultSet rs = stmt.executeQuery();
+ boolean data = rs.next();
+ T resp = null;
+ if (data) {
+ resp = newInstance(rs, fields);
+ if (rs.next()) {
+ LOG.warn("Multiple results for primary key {} in the schema {}, ignoring additional rows.", keyl, igniteMapping.getTableName());
+ }
+ }
+ rs.close();
+ return resp;
+ } catch (SQLException | IOException ex) {
+ throw new GoraException(ex);
+ }
+
+ }
+
+ public T newInstance(ResultSet rs, String[] fields) throws GoraException, SQLException, IOException {
+ fields = getFieldsToQuery(fields);
+ T persistent = newPersistent();
+ for (String f : fields) {
+ Schema.Field field = fieldMap.get(f);
+ Schema fieldSchema = field.schema();
+ String dbField = igniteMapping.getFields().get(f).getName();
+ Object sv = rs.getObject(dbField);
+ if (sv == null) {
+ continue;
+ }
+ Object v = deserializeFieldValue(field, fieldSchema, sv, persistent);
+ persistent.put(field.pos(), v);
+ persistent.setDirty(field.pos());
+ }
+ return persistent;
+ }
+
+ @SuppressWarnings("unchecked")
+ private Object deserializeFieldValue(Schema.Field field, Schema fieldSchema,
+ Object igniteValue, T persistent) throws IOException {
+ Object fieldValue = null;
+ switch (fieldSchema.getType()) {
+ case MAP:
+ case ARRAY:
+ case RECORD:
+ @SuppressWarnings("rawtypes") SpecificDatumReader reader = getDatumReader(fieldSchema);
+ fieldValue = IOUtils.deserialize((byte[]) igniteValue, reader,
+ persistent.get(field.pos()));
+ break;
+ case ENUM:
+ fieldValue = AvroUtils.getEnumValue(fieldSchema, igniteValue.toString());
+ break;
+ case FIXED:
+ break;
+ case BYTES:
+ fieldValue = ByteBuffer.wrap((byte[]) igniteValue);
+ break;
+ case STRING:
+ fieldValue = new Utf8(igniteValue.toString());
+ break;
+ case UNION:
+ if (fieldSchema.getTypes().size() == 2 && isNullable(fieldSchema)) {
+ int schemaPos = getUnionSchema(igniteValue, fieldSchema);
+ Schema unionSchema = fieldSchema.getTypes().get(schemaPos);
+ fieldValue = deserializeFieldValue(field, unionSchema, igniteValue, persistent);
+ } else {
+ reader = getDatumReader(fieldSchema);
+ fieldValue = IOUtils.deserialize((byte[]) igniteValue, reader,
+ persistent.get(field.pos()));
+ }
+ break;
+ default:
+ fieldValue = igniteValue;
+ }
+ return fieldValue;
+
+ }
+
+ @Override
+ public void put(K key, T obj) throws GoraException {
+ try {
+ if (obj.isDirty()) {
+ Schema schema = obj.getSchema();
+ List<Schema.Field> fields = schema.getFields();
+ Map<Column, Object> data = new HashMap<>();
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ Column getKey = igniteMapping.getPrimaryKey().get(0);
+ data.put(getKey, key);
+ } else {
+ //Composite keys pending..
+ }
+ for (Schema.Field field : fields) {
+ Column get = igniteMapping.getFields().get(field.name());
--- End diff --
Please use intuitive names for variables Eg:- mappedColumn
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202526785
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteStore.java ---
@@ -0,0 +1,578 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.sql.rowset.CachedRowSet;
+import javax.sql.rowset.RowSetFactory;
+import javax.sql.rowset.RowSetProvider;
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificDatumReader;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.ignite.query.IgniteQuery;
+import org.apache.gora.ignite.query.IgniteResult;
+import org.apache.gora.ignite.utils.IgniteSQLBuilder;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.AvroUtils;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of a Ignite data store to be used by gora.
+ *
+ * @param <K> class to be used for the key
+ * @param <T> class to be persisted within the store
+ */
+public class IgniteStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ public static final Logger LOG = LoggerFactory.getLogger(IgniteStore.class);
+ private static final String PARSE_MAPPING_FILE_KEY = "gora.ignite.mapping.file";
+ private static final String DEFAULT_MAPPING_FILE = "gora-ignite-mapping.xml";
+ private IgniteParameters igniteParameters;
+ private IgniteMapping igniteMapping;
+ private Connection connection;
+
+ /*
+ * Create a threadlocal map for the datum readers and writers, because they
+ * are not thread safe, at least not before Avro 1.4.0 (See AVRO-650). When
+ * they are thread safe, it is possible to maintain a single reader and writer
+ * pair for every schema, instead of one for every thread.
+ */
+ public static final ConcurrentHashMap<Schema, SpecificDatumReader<?>> readerMap = new ConcurrentHashMap<>();
+
+ public static final ConcurrentHashMap<Schema, SpecificDatumWriter<?>> writerMap = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) throws GoraException {
+
+ try {
+ super.initialize(keyClass, persistentClass, properties);
+ IgniteMappingBuilder builder = new IgniteMappingBuilder(this);
--- End diff --
Type safety: The constructor IgniteMappingBuilder(IgniteStore) belongs to the raw type IgniteMappingBuilder. References to generic type IgniteMappingBuilder< K,T > should be parameterized
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211537029
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteStore.java ---
@@ -0,0 +1,565 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.sql.rowset.CachedRowSet;
+import javax.sql.rowset.RowSetFactory;
+import javax.sql.rowset.RowSetProvider;
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificDatumReader;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.ignite.query.IgniteQuery;
+import org.apache.gora.ignite.query.IgniteResult;
+import org.apache.gora.ignite.utils.IgniteBackendConstants;
+import org.apache.gora.ignite.utils.IgniteSQLBuilder;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.AvroUtils;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of a Ignite data store to be used by gora.
+ *
+ * @param <K> class to be used for the key
+ * @param <T> class to be persisted within the store
+ */
+public class IgniteStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IgniteStore.class);
+ private static final String PARSE_MAPPING_FILE_KEY = "gora.ignite.mapping.file";
+ private static final String DEFAULT_MAPPING_FILE = "gora-ignite-mapping.xml";
+ private IgniteParameters igniteParameters;
+ private IgniteMapping igniteMapping;
+ private Connection connection;
+ private static final ConcurrentHashMap<Schema, SpecificDatumReader<?>> readerMap = new ConcurrentHashMap<>();
+ private static final ConcurrentHashMap<Schema, SpecificDatumWriter<?>> writerMap = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) throws GoraException {
+ try {
+ super.initialize(keyClass, persistentClass, properties);
+ IgniteMappingBuilder<K, T> builder = new IgniteMappingBuilder<K, T>(this);
+ builder.readMappingFile(getConf().get(PARSE_MAPPING_FILE_KEY, DEFAULT_MAPPING_FILE));
+ igniteMapping = builder.getIgniteMapping();
+ igniteParameters = IgniteParameters.load(properties);
+ connection = acquireConnection();
+ LOG.info("Ignite store was successfully initialized");
+ } catch (ClassNotFoundException | SQLException ex) {
+ LOG.error("Error while initializing Ignite store", ex);
+ throw new GoraException(ex);
+ }
+ }
+
+ private Connection acquireConnection() throws ClassNotFoundException, SQLException {
+ Class.forName("org.apache.ignite.IgniteJdbcThinDriver");
+ StringBuilder urlBuilder = new StringBuilder();
+ urlBuilder.append("jdbc:ignite:thin://");
+ urlBuilder.append(igniteParameters.getHost());
+ if (igniteParameters.getPort() != null) {
+ urlBuilder.append(":" + igniteParameters.getPort());
+ }
+ if (igniteParameters.getSchema() != null) {
+ urlBuilder.append("/" + igniteParameters.getSchema());
+ }
+ if (igniteParameters.getUser() != null) {
+ urlBuilder.append(";" + igniteParameters.getUser());
+ }
+ if (igniteParameters.getPassword() != null) {
+ urlBuilder.append(";" + igniteParameters.getPassword());
+ }
+ if (igniteParameters.getAdditionalConfigurations() != null) {
+ urlBuilder.append(igniteParameters.getAdditionalConfigurations());
+ }
+ return DriverManager.getConnection(urlBuilder.toString());
+ }
+
+ @Override
+ public String getSchemaName() {
+ return igniteMapping.getTableName();
+ }
+
+ @Override
+ public String getSchemaName(final String mappingSchemaName,
+ final Class<?> persistentClass) {
+ return super.getSchemaName(mappingSchemaName, persistentClass);
+ }
+
+ @Override
+ public void createSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to create the schema as no connection has been initiated.");
+ }
+ if (schemaExists()) {
+ return;
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String createTableSQL = IgniteSQLBuilder.createTable(igniteMapping);
+ stmt.executeUpdate(createTableSQL);
+ LOG.info("Table {} has been created for Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public void deleteSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to delete the schema as no connection has been initiated.");
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String dropTableSQL = IgniteSQLBuilder.dropTable(igniteMapping.getTableName());
+ stmt.executeUpdate(dropTableSQL);
+ LOG.info("Table {} has been dropped from Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public boolean schemaExists() throws GoraException {
+ try (Statement stmt = connection.createStatement()) {
+ String tableExistsSQL = IgniteSQLBuilder.tableExists(igniteMapping.getTableName());
+ ResultSet executeQuery = stmt.executeQuery(tableExistsSQL);
+ executeQuery.close();
+ return true;
+ } catch (SQLException ex) {
+ if (ex.getSQLState() != null
+ && ex.getSQLState().equals(IgniteBackendConstants.DEFAULT_IGNITE_TABLE_NOT_EXISTS_CODE)) {
+ return false;
+ } else {
+ throw new GoraException(ex);
+ }
+ }
+ }
+
+ @Override
+ public T get(K key, String[] fields) throws GoraException {
+ String[] avFields = getFieldsToQuery(fields);
+ Object[] keyl = null;
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ keyl = new Object[]{key};
+ } else {
+ //Composite key pending
+ }
+ //Avro fields to Ignite fields
+ List<String> dbFields = new ArrayList<>();
+ for (String af : avFields) {
+ dbFields.add(igniteMapping.getFields().get(af).getName());
+ }
+ String selectQuery = IgniteSQLBuilder.selectGet(igniteMapping, dbFields);
+ try (PreparedStatement stmt = connection.prepareStatement(selectQuery)) {
+ IgniteSQLBuilder.fillSelectStatement(stmt, igniteMapping, keyl);
+ ResultSet rs = stmt.executeQuery();
+ boolean data = rs.next();
+ T resp = null;
+ if (data) {
+ resp = newInstance(rs, fields);
+ if (rs.next()) {
+ LOG.warn("Multiple results for primary key {} in the schema {}, ignoring additional rows.", keyl, igniteMapping.getTableName());
+ }
+ }
+ rs.close();
+ return resp;
+ } catch (SQLException | IOException ex) {
+ throw new GoraException(ex);
+ }
+
+ }
+
+ public T newInstance(ResultSet rs, String[] fields) throws GoraException, SQLException, IOException {
+ fields = getFieldsToQuery(fields);
+ T persistent = newPersistent();
+ for (String f : fields) {
+ Schema.Field field = fieldMap.get(f);
+ Schema fieldSchema = field.schema();
+ String dbField = igniteMapping.getFields().get(f).getName();
+ Object sv = rs.getObject(dbField);
+ if (sv == null) {
+ continue;
+ }
+ Object v = deserializeFieldValue(field, fieldSchema, sv, persistent);
+ persistent.put(field.pos(), v);
+ persistent.setDirty(field.pos());
+ }
+ return persistent;
+ }
+
+ @SuppressWarnings("unchecked")
+ private Object deserializeFieldValue(Schema.Field field, Schema fieldSchema,
+ Object igniteValue, T persistent) throws IOException {
+ Object fieldValue = null;
+ switch (fieldSchema.getType()) {
+ case MAP:
+ case ARRAY:
+ case RECORD:
+ @SuppressWarnings("rawtypes") SpecificDatumReader reader = getDatumReader(fieldSchema);
+ fieldValue = IOUtils.deserialize((byte[]) igniteValue, reader,
+ persistent.get(field.pos()));
+ break;
+ case ENUM:
+ fieldValue = AvroUtils.getEnumValue(fieldSchema, igniteValue.toString());
+ break;
+ case FIXED:
+ break;
+ case BYTES:
+ fieldValue = ByteBuffer.wrap((byte[]) igniteValue);
+ break;
+ case STRING:
+ fieldValue = new Utf8(igniteValue.toString());
+ break;
+ case UNION:
+ if (fieldSchema.getTypes().size() == 2 && isNullable(fieldSchema)) {
+ int schemaPos = getUnionSchema(igniteValue, fieldSchema);
+ Schema unionSchema = fieldSchema.getTypes().get(schemaPos);
+ fieldValue = deserializeFieldValue(field, unionSchema, igniteValue, persistent);
+ } else {
+ reader = getDatumReader(fieldSchema);
+ fieldValue = IOUtils.deserialize((byte[]) igniteValue, reader,
+ persistent.get(field.pos()));
+ }
+ break;
+ default:
+ fieldValue = igniteValue;
+ }
+ return fieldValue;
+
+ }
+
+ @Override
+ public void put(K key, T obj) throws GoraException {
+ try {
+ if (obj.isDirty()) {
+ Schema schema = obj.getSchema();
+ List<Schema.Field> fields = schema.getFields();
+ Map<Column, Object> data = new HashMap<>();
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ Column getKey = igniteMapping.getPrimaryKey().get(0);
+ data.put(getKey, key);
+ } else {
+ //Composite keys pending..
+ }
+ for (Schema.Field field : fields) {
+ Column get = igniteMapping.getFields().get(field.name());
+ Object v = obj.get(field.pos());
+ if (get != null && v != null) {
+ Schema fieldSchema = field.schema();
+ Object serializedObj = serializeFieldValue(fieldSchema, v);
+ data.put(get, serializedObj);
+ }
+ }
+ String baseInsertStatement = IgniteSQLBuilder.baseInsertStatement(igniteMapping, data);
+ try (PreparedStatement stmt = connection.prepareStatement(baseInsertStatement)) {
+ IgniteSQLBuilder.fillInsertStatement(stmt, data);
+ stmt.executeUpdate();
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ } else {
+ LOG.info("Ignored putting object {} in the store as it is neither "
+ + "new, neither dirty.", new Object[]{obj});
+ }
+ } catch (Exception e) {
+ throw new GoraException(e);
+ }
+ }
+
+ @Override
+ public boolean delete(K key) throws GoraException {
+ String deleteQuery = null;
+ Object[] keyl = null;
--- End diff --
Please use intuitive names for variables. Eg:- keyArray, keyValues
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by carlosrmng <gi...@git.apache.org>.
Github user carlosrmng commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211491705
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteParameters.java ---
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.util.Properties;
+import org.apache.gora.ignite.utils.IgniteBackendConstants;
+import org.apache.hadoop.conf.Configuration;
+
+/**
+ * Parameters definitions for Ignite.
+ */
+public class IgniteParameters {
+
+ /**
+ * Property indicating the Ignite Schema to be used
+ */
+ public static final String PROP_SCHEMA = "gora.datastore.ignite.schema";
+
+ /**
+ * Property indicating the Ignite Cluster Node IP
+ */
+ public static final String PROP_HOST = "gora.datastore.ignite.host";
+
+ /**
+ * Property indicating the port used by the Ignite Server
+ */
+ public static final String PROP_PORT = "gora.datastore.ignite.port";
+
+ /**
+ * Property indicating the username to connect to the server
+ */
+ public static final String PROP_USER = "gora.datastore.ignite.user";
+
+ /**
+ * Property indicating the password to connect to the server
+ */
+ public static final String PROP_PASSWORD = "gora.datastore.ignite.password";
+
+ /**
+ * Property indicating additional JDBC options
+ */
+ public static final String PROP_ADDITIONALS = "gora.datastore.ignite.additionalConfigurations";
+
+ private String host;
+ private String port;
+ private String schema;
+ private String user;
+ private String password;
+ private String additionalConfigurations;
+
+ /**
+ * @param host Hostname/IP of the Ignite Server
+ * @param port Optional port for Ignite Server
+ * @param user Optional username for Ignite
+ * @param password Optional password for Ignite
+ * @param additionalConfigurations Optional additional configurations for
+ * Ignite
+ */
+ private IgniteParameters(String host, String port, String schema, String user, String password, String additionalConfigurations) {
+ this.host = host;
+ this.port = port;
+ this.schema = schema;
+ this.user = user;
+ this.password = password;
+ this.additionalConfigurations = additionalConfigurations;
+ }
+
+ public String getHost() {
+ return host;
+ }
+
+ public void setHost(String host) {
+ this.host = host;
+ }
+
+ public String getPort() {
+ return port;
+ }
+
+ public void setPort(String port) {
+ this.port = port;
+ }
+
+ public String getUser() {
+ return user;
+ }
+
+ public void setUser(String user) {
+ this.user = user;
+ }
+
+ public String getPassword() {
+ return password;
+ }
+
+ public void setPassword(String password) {
+ this.password = password;
+ }
+
+ public String getAdditionalConfigurations() {
+ return additionalConfigurations;
+ }
+
+ public void setAdditionalConfigurations(String additionalConfigurations) {
+ this.additionalConfigurations = additionalConfigurations;
+ }
+
+ public String getSchema() {
+ return schema;
+ }
+
+ public void setSchema(String schema) {
+ this.schema = schema;
+ }
+
+ public static IgniteParameters load(Properties properties) {
+ return new IgniteParameters(
+ properties.getProperty(PROP_HOST, IgniteBackendConstants.DEFAULT_IGNITE_HOST),
+ properties.getProperty(PROP_PORT, IgniteBackendConstants.DEFAULT_IGNITE_PORT),
+ properties.getProperty(PROP_SCHEMA, null),
+ properties.getProperty(PROP_USER, null),
+ properties.getProperty(PROP_PASSWORD, null),
--- End diff --
I will remove the nulls, I did not realize that if the key does not exist it just will return null anyway. Also, all the attributes of IgniteParameters are validated in the DataStore, so it does not end up in null pointer exceptions.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by nishadi <gi...@git.apache.org>.
Github user nishadi commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202522061
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteStore.java ---
@@ -0,0 +1,578 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.sql.rowset.CachedRowSet;
+import javax.sql.rowset.RowSetFactory;
+import javax.sql.rowset.RowSetProvider;
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificDatumReader;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.ignite.query.IgniteQuery;
+import org.apache.gora.ignite.query.IgniteResult;
+import org.apache.gora.ignite.utils.IgniteSQLBuilder;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.AvroUtils;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of a Ignite data store to be used by gora.
+ *
+ * @param <K> class to be used for the key
+ * @param <T> class to be persisted within the store
+ */
+public class IgniteStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ public static final Logger LOG = LoggerFactory.getLogger(IgniteStore.class);
--- End diff --
Access can be private in here.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211230897
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteMappingBuilder.java ---
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import com.google.inject.ConfigurationException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.jdom.Document;
+import org.jdom.Element;
+import org.jdom.JDOMException;
+import org.jdom.input.SAXBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Builder for Mapping definitions of Ignite.
+ */
+public class IgniteMappingBuilder<K, T extends PersistentBase> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IgniteMappingBuilder.class);
+ /**
+ * Mapping instance being built
+ */
+ private IgniteMapping igniteMapping;
+
+ private final IgniteStore<K, T> dataStore;
+
+ public IgniteMappingBuilder(final IgniteStore<K, T> store) {
+ this.igniteMapping = new IgniteMapping();
+ this.dataStore = store;
+ }
+
+ public IgniteMapping getIgniteMapping() {
+ return igniteMapping;
+ }
+
+ public void setIgniteMapping(IgniteMapping igniteMapping) {
+ this.igniteMapping = igniteMapping;
+ }
+
+ public void readMappingFile(String mappingFile) {
+ try {
+ SAXBuilder saxBuilder = new SAXBuilder();
+ InputStream inputStream = getClass().getClassLoader().getResourceAsStream(mappingFile);
+ if (inputStream == null) {
+ LOG.error("Mapping file '{}' could not be found!", mappingFile);
+ throw new IOException("Mapping file '" + mappingFile + "' could not be found!");
+ }
+ Document document = saxBuilder.build(inputStream);
+ if (document == null) {
+ LOG.error("Mapping file '{}' could not be found!", mappingFile);
+ throw new IOException("Mapping file '" + mappingFile + "' could not be found!");
+ }
+ @SuppressWarnings("unchecked")
+ List<Element> classes = document.getRootElement().getChildren("class");
+ for (Element classElement : classes) {
+ if (classElement.getAttributeValue("keyClass").equals(
+ dataStore.getKeyClass().getCanonicalName())
+ && classElement.getAttributeValue("name").equals(
+ dataStore.getPersistentClass().getCanonicalName())) {
+ final String tableNameFromMapping = classElement.getAttributeValue("table");
+ String tableName = dataStore.getSchemaName(tableNameFromMapping, dataStore.getPersistentClass());
+ igniteMapping.setTableName(tableName);
+ @SuppressWarnings("unchecked")
+ List<Element> prColumns = classElement.getChildren("primarykey");
+ List<Column> prFields = new ArrayList<>();
+ for (Element aPrimaryKey : prColumns) {
+ String name = aPrimaryKey.getAttributeValue("column");
+ String type = aPrimaryKey.getAttributeValue("type");
+ prFields.add(new Column(name, Column.FieldType.valueOf(type)));
+ }
+ igniteMapping.setPrimaryKey(prFields);
+ @SuppressWarnings("unchecked")
+ List<Element> fields = classElement.getChildren("field");
+ Map<String, Column> mp = new HashMap<>();
+ for (Element field : fields) {
+ String fieldName = field.getAttributeValue("name");
+ String columnName = field.getAttributeValue("column");
+ String columnType = field.getAttributeValue("type");
+ mp.put(fieldName, new Column(columnName, Column.FieldType.valueOf(columnType)));
+ }
+ igniteMapping.setFields(mp);
+ break;
+ }
+ }
+
+ } catch (IOException | JDOMException | ConfigurationException e) {
+ throw new RuntimeException(e);
+ }
+ LOG.info("Gora Ignite mapping file was read successfully.");
+
--- End diff --
Please remove new lines.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202523585
--- Diff: nbactions.xml ---
@@ -0,0 +1,13 @@
+<?xml version="1.0" encoding="UTF-8"?>
--- End diff --
Add Apache License header.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211220963
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/Column.java ---
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+public class Column {
+
+ private String name;
+ private FieldType dataType;
+
+ public Column(String name, FieldType dataType) {
+ this.name = name;
+ this.dataType = dataType;
+ }
+
+ public String getName() {
--- End diff --
Please document these public methods.
---
[GitHub] gora issue #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on the issue:
https://github.com/apache/gora/pull/134
@carlosrmng This is excellent work Carlos. Thank you for all hard work you have put into this in summer. I hope you enjoyed your work. Please do address my comments, Let's get this thing merged. :)
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211246685
--- Diff: nbactions.xml ---
@@ -0,0 +1,13 @@
+<?xml version="1.0" encoding="UTF-8"?>
--- End diff --
+1 please remove nbactions.xml.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by nishadi <gi...@git.apache.org>.
Github user nishadi commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202522015
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteStore.java ---
@@ -0,0 +1,578 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.sql.rowset.CachedRowSet;
+import javax.sql.rowset.RowSetFactory;
+import javax.sql.rowset.RowSetProvider;
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificDatumReader;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.ignite.query.IgniteQuery;
+import org.apache.gora.ignite.query.IgniteResult;
+import org.apache.gora.ignite.utils.IgniteSQLBuilder;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.AvroUtils;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of a Ignite data store to be used by gora.
+ *
+ * @param <K> class to be used for the key
+ * @param <T> class to be persisted within the store
+ */
+public class IgniteStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ public static final Logger LOG = LoggerFactory.getLogger(IgniteStore.class);
+ private static final String PARSE_MAPPING_FILE_KEY = "gora.ignite.mapping.file";
+ private static final String DEFAULT_MAPPING_FILE = "gora-ignite-mapping.xml";
+ private IgniteParameters igniteParameters;
+ private IgniteMapping igniteMapping;
+ private Connection connection;
+
+ /*
+ * Create a threadlocal map for the datum readers and writers, because they
+ * are not thread safe, at least not before Avro 1.4.0 (See AVRO-650). When
+ * they are thread safe, it is possible to maintain a single reader and writer
+ * pair for every schema, instead of one for every thread.
+ */
+ public static final ConcurrentHashMap<Schema, SpecificDatumReader<?>> readerMap = new ConcurrentHashMap<>();
+
+ public static final ConcurrentHashMap<Schema, SpecificDatumWriter<?>> writerMap = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) throws GoraException {
+
+ try {
+ super.initialize(keyClass, persistentClass, properties);
+ IgniteMappingBuilder builder = new IgniteMappingBuilder(this);
+ builder.readMappingFile(getConf().get(PARSE_MAPPING_FILE_KEY, DEFAULT_MAPPING_FILE));
+ igniteMapping = builder.getIgniteMapping();
+ igniteParameters = IgniteParameters.load(properties, conf);
+ connection = acquiereConnection();
+ LOG.info("Ignite store was successfully initialized");
+ } catch (ClassNotFoundException | SQLException ex) {
+ LOG.error("Error while initializing Ignite store", ex);
+ throw new GoraException(ex);
+ }
+ }
+
+ private Connection acquiereConnection() throws ClassNotFoundException, SQLException {
+ Class.forName("org.apache.ignite.IgniteJdbcThinDriver");
+ StringBuilder urlBuilder = new StringBuilder();
+ urlBuilder.append("jdbc:ignite:thin://");
+ urlBuilder.append(igniteParameters.getHost());
+ if (igniteParameters.getPort() != null) {
+ urlBuilder.append(":" + igniteParameters.getPort());
+ }
+ if (igniteParameters.getSchema() != null) {
+ urlBuilder.append("/" + igniteParameters.getSchema());
+ }
+ if (igniteParameters.getUser() != null) {
+ urlBuilder.append(";" + igniteParameters.getUser());
+ }
+ if (igniteParameters.getPassword() != null) {
+ urlBuilder.append(";" + igniteParameters.getPassword());
+ }
+ if (igniteParameters.getAdditionalConfigurations() != null) {
+ urlBuilder.append(igniteParameters.getAdditionalConfigurations());
+ }
+ Connection conn = DriverManager.getConnection(urlBuilder.toString());
+ return conn;
+ }
+
+ @Override
+ public String getSchemaName() {
+ return igniteMapping.getTableName();
+ }
+
+ @Override
+ public String getSchemaName(final String mappingSchemaName,
+ final Class<?> persistentClass) {
+ return super.getSchemaName(mappingSchemaName, persistentClass);
+ }
+
+ @Override
+ public void createSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to create the schema as no connection has been initiated.");
+ }
+ if (schemaExists()) {
+ return;
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String createTableSQL = IgniteSQLBuilder.createTable(igniteMapping);
+ stmt.executeUpdate(createTableSQL);
+ LOG.info("Table {} has been created for Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public void deleteSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to delete the schema as no connection has been initiated.");
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String dropTableSQL = IgniteSQLBuilder.dropTable(igniteMapping.getTableName());
+ stmt.executeUpdate(dropTableSQL);
+ LOG.info("Table {} has been dropped from Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public boolean schemaExists() throws GoraException {
+ boolean exists = false;
+ try (Statement stmt = connection.createStatement()) {
+ String tableExistsSQL = IgniteSQLBuilder.tableExists(igniteMapping.getTableName());
+ ResultSet executeQuery = stmt.executeQuery(tableExistsSQL);
+ executeQuery.close();
+ exists = true;
+ } catch (SQLException ex) {
+ /**
+ * a 42000 error code is thrown by Ignite when a non-existent table
+ * queried. More details:
+ * https://apacheignite-sql.readme.io/docs/jdbc-error-codes
+ */
+ if (ex.getSQLState() != null && ex.getSQLState().equals("42000")) {
--- End diff --
Shall we move the constants to a separate constant file?
---
[GitHub] gora issue #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on the issue:
https://github.com/apache/gora/pull/134
Overall PR looks great Carloz :+1: I will go through PR in detail.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by nishadi <gi...@git.apache.org>.
Github user nishadi commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202522595
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteParameters.java ---
@@ -0,0 +1,139 @@
+/*
+ * Copyright 2018 The Apache Software Foundation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.util.Properties;
+import org.apache.hadoop.conf.Configuration;
+
+/**
+ * Parameters definitions for Ignite.
+ */
+public class IgniteParameters {
+
+ /**
+ * Property indicating the Ignite Schema to be used
+ */
+ public static final String PROP_SCHEMA = "gora.datastore.ignite.schema";
+
+ /**
+ * Property indicating the Ignite Cluster Node IP
+ */
+ public static final String PROP_HOST = "gora.datastore.ignite.host";
+
+ /**
+ * Property indicating the port used by the Ignite Server
+ */
+ public static final String PROP_PORT = "gora.datastore.ignite.port";
+
+ /**
+ * Property indicating the username to connect to the server
+ */
+ public static final String PROP_USER = "gora.datastore.ignite.user";
+
+ /**
+ * Property indicating the password to connect to the server
+ */
+ public static final String PROP_PASSWORD = "gora.datastore.ignite.password";
+
+ /**
+ * Property indicating additional JDBC options
+ */
+ public static final String PROP_ADDITIONALS = "gora.datastore.ignite.additionalConfigurations";
+
+ private String host;
+ private String port;
+ private String schema;
+ private String user;
+ private String password;
+ private String additionalConfigurations;
+
+ /**
+ *
+ * @param host
+ * @param port Optional port for Ignite Server
+ * @param user Optional username for Ignite
+ * @param password Optional password for Ignite
+ * @param additionalConfigurations Optional additional configurations for
+ * Ignite
+ */
+ private IgniteParameters(String host, String port, String schema, String user, String password, String additionalConfigurations) {
+ this.host = host;
+ this.port = port;
+ this.schema = schema;
+ this.user = user;
+ this.password = password;
+ this.additionalConfigurations = additionalConfigurations;
+ }
+
+ public String getHost() {
+ return host;
+ }
+
+ public void setHost(String host) {
+ this.host = host;
+ }
+
+ public String getPort() {
+ return port;
+ }
+
+ public void setPort(String port) {
+ this.port = port;
+ }
+
+ public String getUser() {
+ return user;
+ }
+
+ public void setUser(String user) {
+ this.user = user;
+ }
+
+ public String getPassword() {
+ return password;
+ }
+
+ public void setPassword(String password) {
+ this.password = password;
+ }
+
+ public String getAdditionalConfigurations() {
+ return additionalConfigurations;
+ }
+
+ public void setAdditionalConfigurations(String additionalConfigurations) {
+ this.additionalConfigurations = additionalConfigurations;
+ }
+
+ public String getSchema() {
+ return schema;
+ }
+
+ public void setSchema(String schema) {
+ this.schema = schema;
+ }
+
+ public static IgniteParameters load(Properties properties, Configuration conf) {
+ return new IgniteParameters(
--- End diff --
Why are we not using the configuration, when returning the Ignite parameters?
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211539734
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteStore.java ---
@@ -0,0 +1,565 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.sql.rowset.CachedRowSet;
+import javax.sql.rowset.RowSetFactory;
+import javax.sql.rowset.RowSetProvider;
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificDatumReader;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.ignite.query.IgniteQuery;
+import org.apache.gora.ignite.query.IgniteResult;
+import org.apache.gora.ignite.utils.IgniteBackendConstants;
+import org.apache.gora.ignite.utils.IgniteSQLBuilder;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.AvroUtils;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of a Ignite data store to be used by gora.
+ *
+ * @param <K> class to be used for the key
+ * @param <T> class to be persisted within the store
+ */
+public class IgniteStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IgniteStore.class);
+ private static final String PARSE_MAPPING_FILE_KEY = "gora.ignite.mapping.file";
+ private static final String DEFAULT_MAPPING_FILE = "gora-ignite-mapping.xml";
+ private IgniteParameters igniteParameters;
+ private IgniteMapping igniteMapping;
+ private Connection connection;
+ private static final ConcurrentHashMap<Schema, SpecificDatumReader<?>> readerMap = new ConcurrentHashMap<>();
+ private static final ConcurrentHashMap<Schema, SpecificDatumWriter<?>> writerMap = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) throws GoraException {
+ try {
+ super.initialize(keyClass, persistentClass, properties);
+ IgniteMappingBuilder<K, T> builder = new IgniteMappingBuilder<K, T>(this);
+ builder.readMappingFile(getConf().get(PARSE_MAPPING_FILE_KEY, DEFAULT_MAPPING_FILE));
+ igniteMapping = builder.getIgniteMapping();
+ igniteParameters = IgniteParameters.load(properties);
+ connection = acquireConnection();
+ LOG.info("Ignite store was successfully initialized");
+ } catch (ClassNotFoundException | SQLException ex) {
+ LOG.error("Error while initializing Ignite store", ex);
+ throw new GoraException(ex);
+ }
+ }
+
+ private Connection acquireConnection() throws ClassNotFoundException, SQLException {
+ Class.forName("org.apache.ignite.IgniteJdbcThinDriver");
+ StringBuilder urlBuilder = new StringBuilder();
+ urlBuilder.append("jdbc:ignite:thin://");
+ urlBuilder.append(igniteParameters.getHost());
+ if (igniteParameters.getPort() != null) {
+ urlBuilder.append(":" + igniteParameters.getPort());
+ }
+ if (igniteParameters.getSchema() != null) {
+ urlBuilder.append("/" + igniteParameters.getSchema());
+ }
+ if (igniteParameters.getUser() != null) {
+ urlBuilder.append(";" + igniteParameters.getUser());
+ }
+ if (igniteParameters.getPassword() != null) {
+ urlBuilder.append(";" + igniteParameters.getPassword());
+ }
+ if (igniteParameters.getAdditionalConfigurations() != null) {
+ urlBuilder.append(igniteParameters.getAdditionalConfigurations());
+ }
+ return DriverManager.getConnection(urlBuilder.toString());
+ }
+
+ @Override
+ public String getSchemaName() {
+ return igniteMapping.getTableName();
+ }
+
+ @Override
+ public String getSchemaName(final String mappingSchemaName,
+ final Class<?> persistentClass) {
+ return super.getSchemaName(mappingSchemaName, persistentClass);
+ }
+
+ @Override
+ public void createSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to create the schema as no connection has been initiated.");
+ }
+ if (schemaExists()) {
+ return;
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String createTableSQL = IgniteSQLBuilder.createTable(igniteMapping);
+ stmt.executeUpdate(createTableSQL);
+ LOG.info("Table {} has been created for Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public void deleteSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to delete the schema as no connection has been initiated.");
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String dropTableSQL = IgniteSQLBuilder.dropTable(igniteMapping.getTableName());
+ stmt.executeUpdate(dropTableSQL);
+ LOG.info("Table {} has been dropped from Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public boolean schemaExists() throws GoraException {
+ try (Statement stmt = connection.createStatement()) {
+ String tableExistsSQL = IgniteSQLBuilder.tableExists(igniteMapping.getTableName());
+ ResultSet executeQuery = stmt.executeQuery(tableExistsSQL);
+ executeQuery.close();
+ return true;
+ } catch (SQLException ex) {
+ if (ex.getSQLState() != null
+ && ex.getSQLState().equals(IgniteBackendConstants.DEFAULT_IGNITE_TABLE_NOT_EXISTS_CODE)) {
+ return false;
+ } else {
+ throw new GoraException(ex);
+ }
+ }
+ }
+
+ @Override
+ public T get(K key, String[] fields) throws GoraException {
+ String[] avFields = getFieldsToQuery(fields);
+ Object[] keyl = null;
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ keyl = new Object[]{key};
+ } else {
+ //Composite key pending
+ }
+ //Avro fields to Ignite fields
+ List<String> dbFields = new ArrayList<>();
+ for (String af : avFields) {
+ dbFields.add(igniteMapping.getFields().get(af).getName());
+ }
+ String selectQuery = IgniteSQLBuilder.selectGet(igniteMapping, dbFields);
+ try (PreparedStatement stmt = connection.prepareStatement(selectQuery)) {
+ IgniteSQLBuilder.fillSelectStatement(stmt, igniteMapping, keyl);
+ ResultSet rs = stmt.executeQuery();
+ boolean data = rs.next();
+ T resp = null;
+ if (data) {
+ resp = newInstance(rs, fields);
+ if (rs.next()) {
+ LOG.warn("Multiple results for primary key {} in the schema {}, ignoring additional rows.", keyl, igniteMapping.getTableName());
+ }
+ }
+ rs.close();
+ return resp;
+ } catch (SQLException | IOException ex) {
+ throw new GoraException(ex);
+ }
+
+ }
+
+ public T newInstance(ResultSet rs, String[] fields) throws GoraException, SQLException, IOException {
+ fields = getFieldsToQuery(fields);
+ T persistent = newPersistent();
+ for (String f : fields) {
+ Schema.Field field = fieldMap.get(f);
+ Schema fieldSchema = field.schema();
+ String dbField = igniteMapping.getFields().get(f).getName();
+ Object sv = rs.getObject(dbField);
+ if (sv == null) {
+ continue;
+ }
+ Object v = deserializeFieldValue(field, fieldSchema, sv, persistent);
+ persistent.put(field.pos(), v);
+ persistent.setDirty(field.pos());
+ }
+ return persistent;
+ }
+
+ @SuppressWarnings("unchecked")
+ private Object deserializeFieldValue(Schema.Field field, Schema fieldSchema,
+ Object igniteValue, T persistent) throws IOException {
+ Object fieldValue = null;
+ switch (fieldSchema.getType()) {
+ case MAP:
+ case ARRAY:
+ case RECORD:
+ @SuppressWarnings("rawtypes") SpecificDatumReader reader = getDatumReader(fieldSchema);
+ fieldValue = IOUtils.deserialize((byte[]) igniteValue, reader,
+ persistent.get(field.pos()));
+ break;
+ case ENUM:
+ fieldValue = AvroUtils.getEnumValue(fieldSchema, igniteValue.toString());
+ break;
+ case FIXED:
+ break;
+ case BYTES:
+ fieldValue = ByteBuffer.wrap((byte[]) igniteValue);
+ break;
+ case STRING:
+ fieldValue = new Utf8(igniteValue.toString());
+ break;
+ case UNION:
+ if (fieldSchema.getTypes().size() == 2 && isNullable(fieldSchema)) {
+ int schemaPos = getUnionSchema(igniteValue, fieldSchema);
+ Schema unionSchema = fieldSchema.getTypes().get(schemaPos);
+ fieldValue = deserializeFieldValue(field, unionSchema, igniteValue, persistent);
+ } else {
+ reader = getDatumReader(fieldSchema);
+ fieldValue = IOUtils.deserialize((byte[]) igniteValue, reader,
+ persistent.get(field.pos()));
+ }
+ break;
+ default:
+ fieldValue = igniteValue;
+ }
+ return fieldValue;
+
--- End diff --
Please remove unnecessary new lines.
---
[GitHub] gora issue #134: GORA-535 Add a data store for Apache Ignite
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on the issue:
https://github.com/apache/gora/pull/134
np @djkevincr I just wanted to make sure we got it there as well :)
Excellent job on this one folks.. really.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211522585
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/utils/IgniteSQLBuilder.java ---
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.utils;
+
+import avro.shaded.com.google.common.collect.Lists;
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+import java.text.MessageFormat;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.gora.ignite.store.Column;
+import org.apache.gora.ignite.store.IgniteMapping;
+
+/**
+ *
+ * SQL Builder utility for Ignite.
+ */
+public class IgniteSQLBuilder {
+
+ private static String format(String pattern, Object... args) {
+ MessageFormat messageFormat = new MessageFormat(pattern, Locale.getDefault());
+ return messageFormat.format(args);
+ }
+
+ /**
+ * Returns a SQL query for determine whether a table exists or not.
+ *
+ * @param tableName The name of the table to be check.
+ * @return SQL query
+ */
+ public static String tableExists(String tableName) {
+ return format("SELECT * FROM {0} LIMIT 0", tableName);
+ }
+
+ /**
+ * Returns a SQL create table statement for initializing a datastore based
+ * upon a Ignite Mapping definition.
+ *
+ * @param mapping The ignite mapping definition of the data store
+ * @return SQL create query (DDL).
+ */
+ public static String createTable(IgniteMapping mapping) {
+ StringBuilder sqlBuilder = new StringBuilder();
+ sqlBuilder.append("CREATE TABLE ");
+ sqlBuilder.append(mapping.getTableName());
+ sqlBuilder.append("(");
+ ArrayList<Column> fieldsList = Lists.newArrayList(mapping.getPrimaryKey());
+ fieldsList.addAll(Lists.newArrayList(mapping.getFields().values()));
+ for (Column aColumn : fieldsList) {
+ String name = aColumn.getName();
+ Column.FieldType dataType = aColumn.getDataType();
+ sqlBuilder.append(name).append(" ").append(dataType.toString()).append(",");
+ }
+ sqlBuilder.append("PRIMARY KEY ");
+ sqlBuilder.append("(");
+ for (int i = 0; i < mapping.getPrimaryKey().size(); i++) {
+ sqlBuilder.append(mapping.getPrimaryKey().get(i).getName());
+ sqlBuilder.append(i == mapping.getPrimaryKey().size() - 1 ? "" : ",");
+ }
+ sqlBuilder.append(")");
+ sqlBuilder.append(");");
+ return sqlBuilder.toString();
+ }
+
+ /**
+ * Returns a SQL drop table statement for deleting a datastore instance within
+ * ignite.
+ *
+ * @param tableName The name of the table to be dropped.
+ * @return SQL drop query (DDL).
+ */
+ public static String dropTable(String tableName) {
+ return format("DROP TABLE IF EXISTS {0} ;", tableName);
+ }
+
+ /**
+ * Returns a bare SQL insert statement for adding a new record on a Ignite
+ * data store.
+ *
+ * @param mapping The ignite mapping definition of the data store
+ * @param data A map containing the Column-Value pairs of the new record.
+ * @return SQL insert statement
+ */
+ public static String baseInsertStatement(IgniteMapping mapping, Map<Column, Object> data) {
+ StringBuilder sqlBuilder = new StringBuilder();
+ sqlBuilder.append("MERGE INTO ");
+ sqlBuilder.append(mapping.getTableName());
+ sqlBuilder.append(" (");
+ List<Entry<Column, Object>> list = new ArrayList<>(data.entrySet());
+ for (int i = 0; i < list.size(); i++) {
+ sqlBuilder.append(list.get(i).getKey().getName());
+ sqlBuilder.append(i == list.size() - 1 ? "" : ",");
+ }
+ sqlBuilder.append(")");
+ sqlBuilder.append(" VALUES ");
+ sqlBuilder.append(" (");
+ for (int i = 0; i < list.size(); i++) {
+ sqlBuilder.append("?");
+ sqlBuilder.append(i == list.size() - 1 ? "" : ",");
+ }
+ sqlBuilder.append(" )");
+ return sqlBuilder.toString();
+ }
+
+ /**
+ * Fills a SQL PreparedStatement of a insert operation with the actual data to
+ * be inserted.
+ *
+ * @param statement The insertion PreparedStatement to be filled.
+ * @param insertData A map containing the Column-Value pairs of the new
+ * record.
+ * @throws SQLException When invalid values are provided as parameters for the
+ * insert statement.
+ */
+ public static void fillInsertStatement(PreparedStatement statement, Map<Column, Object> insertData) throws SQLException {
+ List<Entry<Column, Object>> list = new ArrayList<>(insertData.entrySet());
+ for (int i = 0; i < list.size(); i++) {
+ int j = i + 1;
+ statement.setObject(j, list.get(i).getValue());
+ }
+ }
+
+ /**
+ * Returns a bare SQL statement for deleting a record from the Ignite data
+ * store.
+ *
+ * @param mapping The ignite mapping definition of the data store
+ * @return SQL delete statement
+ */
+ public static String delete(IgniteMapping mapping) {
+ StringBuilder sqlBuilder = new StringBuilder();
+ sqlBuilder.append("DELETE FROM ");
+ sqlBuilder.append(mapping.getTableName());
+ sqlBuilder.append(" WHERE ");
+ for (int i = 0; i < mapping.getPrimaryKey().size(); i++) {
+ sqlBuilder.append(mapping.getPrimaryKey().get(i).getName());
+ sqlBuilder.append("= ? ");
+ sqlBuilder.append(i == mapping.getPrimaryKey().size() - 1 ? "" : " AND ");
+ }
+ return sqlBuilder.toString();
+ }
+
+ /**
+ * Fills a SQL PreparedStatement of a delete operation with the actual key of
+ * the record to be deleted
+ *
+ * @param statement The deletion PreparedStatement to be filled.
+ * @param mapping The ignite mapping definition of the data store
+ * @param deleteData An Object array containing the primary key values of the
+ * record to be deleted
+ * @throws SQLException When invalid keys' values are provided as parameters
+ */
+ public static void fillDeleteStatement(PreparedStatement statement, IgniteMapping mapping, Object... deleteData) throws SQLException {
+ assert mapping.getPrimaryKey().size() == deleteData.length;
+ for (int i = 0; i < mapping.getPrimaryKey().size(); i++) {
+ int j = i + 1;
+ statement.setObject(j, deleteData[i]);
+ }
+ }
+
+ /**
+ * Returns a bare SQL statement for retrieving a record from the ignite data
+ * store
+ *
+ * @param mapping The ignite mapping definition of the data store
+ * @param columns A list of columns to be retrieved within the select query
+ * @return SQL select statement
+ */
+ public static String selectGet(IgniteMapping mapping, List<String> columns) {
+ StringBuilder sqlBuilder = new StringBuilder();
+ sqlBuilder.append("SELECT ");
+ for (int i = 0; i < columns.size(); i++) {
+ sqlBuilder.append(columns.get(i));
+ sqlBuilder.append(i == columns.size() - 1 ? "" : " , ");
+ }
+ sqlBuilder.append(" FROM ");
+ sqlBuilder.append(mapping.getTableName());
+ sqlBuilder.append(" WHERE ");
+ for (int i = 0; i < mapping.getPrimaryKey().size(); i++) {
+ sqlBuilder.append(mapping.getPrimaryKey().get(i).getName());
+ sqlBuilder.append("= ? ");
+ sqlBuilder.append(i == mapping.getPrimaryKey().size() - 1 ? "" : " AND ");
+ }
+ return sqlBuilder.toString();
+ }
+
+ /**
+ * Fills a SQL PreparedStatement of a select operation with the actual keys of
+ * the record to be retrieved
+ *
+ * @param statement The select PreparedStatement to be filled.
+ * @param mapping The ignite mapping definition of the data store
+ * @param selectData An Object array containing the primary key values of the
+ * record to be retrieved
+ * @throws SQLException When invalid keys' values are provided as parameters
+ */
+ public static void fillSelectStatement(PreparedStatement statement, IgniteMapping mapping, Object... selectData) throws SQLException {
+ assert mapping.getPrimaryKey().size() == selectData.length;
+ for (int i = 0; i < mapping.getPrimaryKey().size(); i++) {
+ int j = i + 1;
+ statement.setObject(j, selectData[i]);
+ }
+ }
+
+ /**
+ * Returns a base SQL statement for retrieving multiple records from the
+ * ignite data store
+ *
+ * @param mapping The ignite mapping definition of the data store
+ * @param selectFields A list of columns to be retrieved within the select
+ * query
+ * @return SQL select statement
+ */
+ public static String selectQuery(IgniteMapping mapping, List<String> selectFields) {
+ List<String> fields = new ArrayList<>();
+ for (Column c : mapping.getPrimaryKey()) {
+ fields.add(c.getName());
+ }
+ fields.addAll(selectFields);
+ StringBuilder sqlBuilder = new StringBuilder();
+ sqlBuilder.append("SELECT ");
+ for (int i = 0; i < fields.size(); i++) {
+ sqlBuilder.append(fields.get(i));
+ sqlBuilder.append(i == fields.size() - 1 ? "" : " , ");
+ }
+ sqlBuilder.append(" FROM ");
+ sqlBuilder.append(mapping.getTableName());
+ return sqlBuilder.toString();
+ }
+
+ /**
+ * Returns a base SQL statement for deleting multiple records from the ignite
+ * data store
+ *
+ * @param mapping The ignite mapping definition of the data store
+ * @return SQL delete statement
+ */
+ public static String deleteQuery(IgniteMapping mapping) {
+ StringBuilder sqlBuilder = new StringBuilder();
+ sqlBuilder.append("DELETE FROM ");
+ sqlBuilder.append(mapping.getTableName());
+ return sqlBuilder.toString();
+ }
+
+ /**
+ * Returns a base SQL statement for deleting fields from records of the ignite
+ * data store
+ *
+ * @param mapping The ignite mapping definition of the data store
+ * @param deleteFields A list of columns to be deleted (set to null)
+ * @return SQL update statement
+ */
+ public static String deleteQueryFields(IgniteMapping mapping, List<String> deleteFields) {
+ StringBuilder sqlBuilder = new StringBuilder();
+ sqlBuilder.append("UPDATE ");
+ sqlBuilder.append(mapping.getTableName());
+ if (!deleteFields.isEmpty()) {
+ sqlBuilder.append(" SET ");
+ }
+ for (int i = 0; i < deleteFields.size(); i++) {
+ sqlBuilder.append(deleteFields.get(i));
+ sqlBuilder.append(" = null");
+ sqlBuilder.append(i == deleteFields.size() - 1 ? "" : " , ");
+ }
+ return sqlBuilder.toString();
+ }
+
+ /**
+ * Returns a SQL's WHERE segment with proper conditions set for
+ * Querying/Deleting/Updating multiple records of a ignite data store
+ *
+ * @param mapping The ignite mapping definition of the data store
+ * @param startKey Start key of the WHERE condition
+ * @param endKey End key of the WHERE condition
+ * @param limit The maximum number of records to be consider
+ * @return SQL WHERE segment
+ */
+ public static String queryWhere(IgniteMapping mapping, Object startKey, Object endKey, long limit) {
--- End diff --
Please rename these methods to suitable names Eg:- createWhereQuery() Please do this for all the methods in the class.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211214999
--- Diff: gora-ignite/pom.xml ---
@@ -0,0 +1,172 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora</artifactId>
+ <version>0.9-SNAPSHOT</version>
+ <relativePath>../</relativePath>
+ </parent>
+ <artifactId>gora-ignite</artifactId>
+ <packaging>bundle</packaging>
+
+ <name>Apache Gora :: Ignite</name>
+ <url>http://gora.apache.org</url>
+ <description>The Apache Gora open source framework provides an in-memory data model and
+ persistence for big data. Gora supports persisting to column stores, key value stores,
+ document stores and RDBMSs, and analyzing the data with extensive Apache Hadoop MapReduce
+ support.</description>
+ <inceptionYear>2010</inceptionYear>
+ <organization>
+ <name>The Apache Software Foundation</name>
+ <url>http://www.apache.org/</url>
+ </organization>
+ <issueManagement>
+ <system>JIRA</system>
+ <url>https://issues.apache.org/jira/browse/GORA</url>
+ </issueManagement>
+ <ciManagement>
+ <system>Jenkins</system>
+ <url>https://builds.apache.org/job/Gora-trunk/</url>
+ </ciManagement>
+
+ <properties>
+ <osgi.import>*</osgi.import>
+ <osgi.export>org.apache.gora.ignite*;version="${project.version}";-noimport:=true</osgi.export>
+ </properties>
+
+ <build>
+ <directory>target</directory>
+ <outputDirectory>target/classes</outputDirectory>
+ <finalName>${project.artifactId}-${project.version}</finalName>
+ <testOutputDirectory>target/test-classes</testOutputDirectory>
+ <testSourceDirectory>src/test/java</testSourceDirectory>
+ <sourceDirectory>src/main/java</sourceDirectory>
+ <testResources>
+ <testResource>
+ <directory>${project.basedir}/src/test/resources</directory>
+ <includes>
+ <include>**/*</include>
+ </includes>
+ <!--targetPath>${project.basedir}/target/classes/</targetPath -->
+ </testResource>
+ </testResources>
+ <plugins>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>build-helper-maven-plugin</artifactId>
+ <version>${build-helper-maven-plugin.version}</version>
+ <executions>
+ <execution>
+ <phase>generate-sources</phase>
+ <goals>
+ <goal>add-source</goal>
+ </goals>
+ <configuration>
+ <sources>
+ <source>src/examples/java</source>
+ </sources>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+
+ <dependencies>
+ <!-- Gora Internal Dependencies -->
+ <dependency>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora-core</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora-core</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+
+ <!--Ignite Dependency -->
+ <dependency>
+ <groupId>org.apache.ignite</groupId>
+ <artifactId>ignite-core</artifactId>
+ <version>${ignite.version}</version>
--- End diff --
Please move this dependency to Parent POM, then you just need to refer without
```<version>${ignite.version}</version>``` tag.
Please check other dependencies here, which have been referred from parent pom.
---
[GitHub] gora issue #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on the issue:
https://github.com/apache/gora/pull/134
@lewismc I will have a look Lewis. Sorry about the delays. If it all fine i will proceed merging to master.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202523513
--- Diff: gora-ignite/pom.xml ---
@@ -0,0 +1,182 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora</artifactId>
+ <version>0.9-SNAPSHOT</version>
+ <relativePath>../</relativePath>
+ </parent>
+ <artifactId>gora-ignite</artifactId>
+ <packaging>bundle</packaging>
+
+ <name>Apache Gora :: Ignite</name>
+ <url>http://gora.apache.org</url>
+ <description>The Apache Gora open source framework provides an in-memory data model and
+ persistence for big data. Gora supports persisting to column stores, key value stores,
+ document stores and RDBMSs, and analyzing the data with extensive Apache Hadoop MapReduce
+ support.</description>
+ <inceptionYear>2010</inceptionYear>
+ <organization>
+ <name>The Apache Software Foundation</name>
+ <url>http://www.apache.org/</url>
+ </organization>
+ <issueManagement>
+ <system>JIRA</system>
+ <url>https://issues.apache.org/jira/browse/GORA</url>
+ </issueManagement>
+ <ciManagement>
+ <system>Jenkins</system>
+ <url>https://builds.apache.org/job/Gora-trunk/</url>
+ </ciManagement>
+
+ <properties>
+ <ignite.version>2.4.0</ignite.version>
+ <osgi.import>*</osgi.import>
+ <osgi.export>org.apache.gora.ignite*;version="${project.version}";-noimport:=true</osgi.export>
+ </properties>
+
+ <build>
+ <directory>target</directory>
+ <outputDirectory>target/classes</outputDirectory>
+ <finalName>${project.artifactId}-${project.version}</finalName>
+ <testOutputDirectory>target/test-classes</testOutputDirectory>
+ <testSourceDirectory>src/test/java</testSourceDirectory>
+ <sourceDirectory>src/main/java</sourceDirectory>
+ <testResources>
+ <testResource>
+ <directory>${project.basedir}/src/test/resources</directory>
+ <includes>
+ <include>**/*</include>
+ </includes>
+ <!--targetPath>${project.basedir}/target/classes/</targetPath -->
+ </testResource>
+ </testResources>
+ <plugins>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>build-helper-maven-plugin</artifactId>
+ <version>${build-helper-maven-plugin.version}</version>
+ <executions>
+ <execution>
+ <phase>generate-sources</phase>
+ <goals>
+ <goal>add-source</goal>
+ </goals>
+ <configuration>
+ <sources>
+ <source>src/examples/java</source>
+ </sources>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+
+ <dependencies>
+ <!-- Gora Internal Dependencies -->
+ <dependency>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora-core</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora-core</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+
+ <!--Ignite Dependency -->
+ <dependency>
+ <groupId>org.apache.ignite</groupId>
+ <artifactId>ignite-core</artifactId>
+ <version>${ignite.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ </dependency>
+
+ <dependency>
+ <!-- Ignite mini was trying to load the Shell, which failed with the version of jline in the parent pom -->
+ <groupId>jline</groupId>
+ <artifactId>jline</artifactId>
+ <version>2.11</version>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.zookeeper</groupId>
+ <artifactId>zookeeper</artifactId>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.avro</groupId>
+ <artifactId>avro</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.jdom</groupId>
+ <artifactId>jdom</artifactId>
+ <scope>compile</scope>
+ </dependency>
+
+ <!-- Logging Dependencies -->
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-log4j12</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>log4j</groupId>
+ <artifactId>log4j</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>javax.jms</groupId>
+ <artifactId>jms</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <!-- Testing Dependencies -->
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-minicluster</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.ignite</groupId>
+ <artifactId>ignite-indexing</artifactId>
+ <version>2.4.0</version>
--- End diff --
Please move this up to the other ignite dependency. Also consider defining the version as a constant somewhere.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211237792
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteParameters.java ---
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.util.Properties;
+import org.apache.gora.ignite.utils.IgniteBackendConstants;
+import org.apache.hadoop.conf.Configuration;
+
+/**
+ * Parameters definitions for Ignite.
+ */
+public class IgniteParameters {
+
+ /**
+ * Property indicating the Ignite Schema to be used
+ */
+ public static final String PROP_SCHEMA = "gora.datastore.ignite.schema";
+
+ /**
+ * Property indicating the Ignite Cluster Node IP
+ */
+ public static final String PROP_HOST = "gora.datastore.ignite.host";
+
+ /**
+ * Property indicating the port used by the Ignite Server
+ */
+ public static final String PROP_PORT = "gora.datastore.ignite.port";
+
+ /**
+ * Property indicating the username to connect to the server
+ */
+ public static final String PROP_USER = "gora.datastore.ignite.user";
+
+ /**
+ * Property indicating the password to connect to the server
+ */
+ public static final String PROP_PASSWORD = "gora.datastore.ignite.password";
+
+ /**
+ * Property indicating additional JDBC options
+ */
+ public static final String PROP_ADDITIONALS = "gora.datastore.ignite.additionalConfigurations";
+
+ private String host;
+ private String port;
+ private String schema;
+ private String user;
+ private String password;
+ private String additionalConfigurations;
+
+ /**
+ * @param host Hostname/IP of the Ignite Server
+ * @param port Optional port for Ignite Server
+ * @param user Optional username for Ignite
+ * @param password Optional password for Ignite
+ * @param additionalConfigurations Optional additional configurations for
+ * Ignite
+ */
+ private IgniteParameters(String host, String port, String schema, String user, String password, String additionalConfigurations) {
+ this.host = host;
+ this.port = port;
+ this.schema = schema;
+ this.user = user;
+ this.password = password;
+ this.additionalConfigurations = additionalConfigurations;
+ }
+
+ public String getHost() {
+ return host;
+ }
+
+ public void setHost(String host) {
+ this.host = host;
+ }
+
+ public String getPort() {
+ return port;
+ }
+
+ public void setPort(String port) {
+ this.port = port;
+ }
+
+ public String getUser() {
+ return user;
+ }
+
+ public void setUser(String user) {
+ this.user = user;
+ }
+
+ public String getPassword() {
+ return password;
+ }
+
+ public void setPassword(String password) {
+ this.password = password;
+ }
+
+ public String getAdditionalConfigurations() {
+ return additionalConfigurations;
+ }
+
+ public void setAdditionalConfigurations(String additionalConfigurations) {
+ this.additionalConfigurations = additionalConfigurations;
+ }
+
+ public String getSchema() {
+ return schema;
+ }
+
+ public void setSchema(String schema) {
+ this.schema = schema;
+ }
+
+ public static IgniteParameters load(Properties properties) {
+ return new IgniteParameters(
+ properties.getProperty(PROP_HOST, IgniteBackendConstants.DEFAULT_IGNITE_HOST),
+ properties.getProperty(PROP_PORT, IgniteBackendConstants.DEFAULT_IGNITE_PORT),
+ properties.getProperty(PROP_SCHEMA, null),
+ properties.getProperty(PROP_USER, null),
+ properties.getProperty(PROP_PASSWORD, null),
--- End diff --
Is it required to add null value as the default value, since it will return null if key is not exist in the properties. Another point is, have you handled the null value for a non existing key in higher layer when they are referred from IgniteParameters. Check whether you end up in null pointer exception if one of these properties are not included in properties file.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211215194
--- Diff: gora-ignite/pom.xml ---
@@ -0,0 +1,182 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora</artifactId>
+ <version>0.9-SNAPSHOT</version>
+ <relativePath>../</relativePath>
+ </parent>
+ <artifactId>gora-ignite</artifactId>
+ <packaging>bundle</packaging>
+
+ <name>Apache Gora :: Ignite</name>
+ <url>http://gora.apache.org</url>
+ <description>The Apache Gora open source framework provides an in-memory data model and
+ persistence for big data. Gora supports persisting to column stores, key value stores,
+ document stores and RDBMSs, and analyzing the data with extensive Apache Hadoop MapReduce
+ support.</description>
+ <inceptionYear>2010</inceptionYear>
+ <organization>
+ <name>The Apache Software Foundation</name>
+ <url>http://www.apache.org/</url>
+ </organization>
+ <issueManagement>
+ <system>JIRA</system>
+ <url>https://issues.apache.org/jira/browse/GORA</url>
+ </issueManagement>
+ <ciManagement>
+ <system>Jenkins</system>
+ <url>https://builds.apache.org/job/Gora-trunk/</url>
+ </ciManagement>
+
+ <properties>
+ <ignite.version>2.4.0</ignite.version>
+ <osgi.import>*</osgi.import>
+ <osgi.export>org.apache.gora.ignite*;version="${project.version}";-noimport:=true</osgi.export>
+ </properties>
+
+ <build>
+ <directory>target</directory>
+ <outputDirectory>target/classes</outputDirectory>
+ <finalName>${project.artifactId}-${project.version}</finalName>
+ <testOutputDirectory>target/test-classes</testOutputDirectory>
+ <testSourceDirectory>src/test/java</testSourceDirectory>
+ <sourceDirectory>src/main/java</sourceDirectory>
+ <testResources>
+ <testResource>
+ <directory>${project.basedir}/src/test/resources</directory>
+ <includes>
+ <include>**/*</include>
+ </includes>
+ <!--targetPath>${project.basedir}/target/classes/</targetPath -->
+ </testResource>
+ </testResources>
+ <plugins>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>build-helper-maven-plugin</artifactId>
+ <version>${build-helper-maven-plugin.version}</version>
+ <executions>
+ <execution>
+ <phase>generate-sources</phase>
+ <goals>
+ <goal>add-source</goal>
+ </goals>
+ <configuration>
+ <sources>
+ <source>src/examples/java</source>
+ </sources>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+
+ <dependencies>
+ <!-- Gora Internal Dependencies -->
+ <dependency>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora-core</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora-core</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+
+ <!--Ignite Dependency -->
+ <dependency>
+ <groupId>org.apache.ignite</groupId>
+ <artifactId>ignite-core</artifactId>
+ <version>${ignite.version}</version>
+ </dependency>
+
--- End diff --
+1
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211249639
--- Diff: gora-ignite/src/test/java/org/apache/gora/ignite/store/TestIgniteStore.java ---
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import org.apache.gora.ignite.GoraIgniteTestDriver;
+import org.apache.gora.store.DataStoreTestBase;
+
+/**
+ * Test case for IgniteStore.
+ */
+public class TestIgniteStore extends DataStoreTestBase {
--- End diff --
Please add some map reduce tests for the datastore, Please follow orientdb, hbase datastore test cases.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202526799
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteStore.java ---
@@ -0,0 +1,578 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.sql.rowset.CachedRowSet;
+import javax.sql.rowset.RowSetFactory;
+import javax.sql.rowset.RowSetProvider;
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificDatumReader;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.ignite.query.IgniteQuery;
+import org.apache.gora.ignite.query.IgniteResult;
+import org.apache.gora.ignite.utils.IgniteSQLBuilder;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.AvroUtils;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of a Ignite data store to be used by gora.
+ *
+ * @param <K> class to be used for the key
+ * @param <T> class to be persisted within the store
+ */
+public class IgniteStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ public static final Logger LOG = LoggerFactory.getLogger(IgniteStore.class);
+ private static final String PARSE_MAPPING_FILE_KEY = "gora.ignite.mapping.file";
+ private static final String DEFAULT_MAPPING_FILE = "gora-ignite-mapping.xml";
+ private IgniteParameters igniteParameters;
+ private IgniteMapping igniteMapping;
+ private Connection connection;
+
+ /*
+ * Create a threadlocal map for the datum readers and writers, because they
+ * are not thread safe, at least not before Avro 1.4.0 (See AVRO-650). When
+ * they are thread safe, it is possible to maintain a single reader and writer
+ * pair for every schema, instead of one for every thread.
+ */
+ public static final ConcurrentHashMap<Schema, SpecificDatumReader<?>> readerMap = new ConcurrentHashMap<>();
+
+ public static final ConcurrentHashMap<Schema, SpecificDatumWriter<?>> writerMap = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) throws GoraException {
+
+ try {
+ super.initialize(keyClass, persistentClass, properties);
+ IgniteMappingBuilder builder = new IgniteMappingBuilder(this);
+ builder.readMappingFile(getConf().get(PARSE_MAPPING_FILE_KEY, DEFAULT_MAPPING_FILE));
+ igniteMapping = builder.getIgniteMapping();
+ igniteParameters = IgniteParameters.load(properties, conf);
+ connection = acquiereConnection();
+ LOG.info("Ignite store was successfully initialized");
+ } catch (ClassNotFoundException | SQLException ex) {
+ LOG.error("Error while initializing Ignite store", ex);
+ throw new GoraException(ex);
+ }
+ }
+
+ private Connection acquiereConnection() throws ClassNotFoundException, SQLException {
+ Class.forName("org.apache.ignite.IgniteJdbcThinDriver");
+ StringBuilder urlBuilder = new StringBuilder();
+ urlBuilder.append("jdbc:ignite:thin://");
+ urlBuilder.append(igniteParameters.getHost());
+ if (igniteParameters.getPort() != null) {
+ urlBuilder.append(":" + igniteParameters.getPort());
+ }
+ if (igniteParameters.getSchema() != null) {
+ urlBuilder.append("/" + igniteParameters.getSchema());
+ }
+ if (igniteParameters.getUser() != null) {
+ urlBuilder.append(";" + igniteParameters.getUser());
+ }
+ if (igniteParameters.getPassword() != null) {
+ urlBuilder.append(";" + igniteParameters.getPassword());
+ }
+ if (igniteParameters.getAdditionalConfigurations() != null) {
+ urlBuilder.append(igniteParameters.getAdditionalConfigurations());
+ }
+ Connection conn = DriverManager.getConnection(urlBuilder.toString());
+ return conn;
+ }
+
+ @Override
+ public String getSchemaName() {
+ return igniteMapping.getTableName();
+ }
+
+ @Override
+ public String getSchemaName(final String mappingSchemaName,
+ final Class<?> persistentClass) {
+ return super.getSchemaName(mappingSchemaName, persistentClass);
+ }
+
+ @Override
+ public void createSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to create the schema as no connection has been initiated.");
+ }
+ if (schemaExists()) {
+ return;
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String createTableSQL = IgniteSQLBuilder.createTable(igniteMapping);
+ stmt.executeUpdate(createTableSQL);
+ LOG.info("Table {} has been created for Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public void deleteSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to delete the schema as no connection has been initiated.");
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String dropTableSQL = IgniteSQLBuilder.dropTable(igniteMapping.getTableName());
+ stmt.executeUpdate(dropTableSQL);
+ LOG.info("Table {} has been dropped from Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public boolean schemaExists() throws GoraException {
+ boolean exists = false;
+ try (Statement stmt = connection.createStatement()) {
+ String tableExistsSQL = IgniteSQLBuilder.tableExists(igniteMapping.getTableName());
+ ResultSet executeQuery = stmt.executeQuery(tableExistsSQL);
+ executeQuery.close();
+ exists = true;
+ } catch (SQLException ex) {
+ /**
+ * a 42000 error code is thrown by Ignite when a non-existent table
+ * queried. More details:
+ * https://apacheignite-sql.readme.io/docs/jdbc-error-codes
+ */
+ if (ex.getSQLState() != null && ex.getSQLState().equals("42000")) {
+ exists = false;
+ } else {
+ throw new GoraException(ex);
+ }
+ }
+ return exists;
+ }
+
+ @Override
+ public T get(K key, String[] fields) throws GoraException {
+ String[] avFields = getFieldsToQuery(fields);
+ Object[] keyl = null;
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ keyl = new Object[]{key};
+ } else {
+ //Composite key pending
+ }
+ //Avro fields to Ignite fields
+ List<String> dbFields = new ArrayList<>();
+ for (String af : avFields) {
+ dbFields.add(igniteMapping.getFields().get(af).getName());
+ }
+ String selectQuery = IgniteSQLBuilder.selectGet(igniteMapping, dbFields);
+ try (PreparedStatement stmt = connection.prepareStatement(selectQuery)) {
+ IgniteSQLBuilder.fillSelectStatement(stmt, igniteMapping, keyl);
+ ResultSet rs = stmt.executeQuery();
+ boolean data = rs.next();
+ T resp = null;
+ if (data) {
+ resp = newInstance(rs, fields);
+ if (rs.next()) {
+ LOG.warn("Multiple results for primary key {} in the schema {}, ignoring additional rows.", keyl, igniteMapping.getTableName());
+ }
+ }
+ rs.close();
+ return resp;
+ } catch (SQLException | IOException ex) {
+ throw new GoraException(ex);
+ }
+
+ }
+
+ public T newInstance(ResultSet rs, String[] fields) throws GoraException, SQLException, IOException {
+ fields = getFieldsToQuery(fields);
+ T persistent = newPersistent();
+ for (String f : fields) {
+ Schema.Field field = fieldMap.get(f);
+ Schema fieldSchema = field.schema();
+ String dbField = igniteMapping.getFields().get(f).getName();
+ Object sv = rs.getObject(dbField);
+ if (sv == null) {
+ continue;
+ }
+ Object v = deserializeFieldValue(field, fieldSchema, sv, persistent);
+ persistent.put(field.pos(), v);
+ persistent.setDirty(field.pos());
+ }
+ return persistent;
+ }
+
+ private Object deserializeFieldValue(Schema.Field field, Schema fieldSchema,
+ Object igniteValue, T persistent) throws IOException {
+ Object fieldValue = null;
+ switch (fieldSchema.getType()) {
+ case MAP:
+ case ARRAY:
+ case RECORD:
+ @SuppressWarnings("rawtypes") SpecificDatumReader reader = getDatumReader(fieldSchema);
+ fieldValue = IOUtils.deserialize((byte[]) igniteValue, reader,
+ persistent.get(field.pos()));
+ break;
+ case ENUM:
+ fieldValue = AvroUtils.getEnumValue(fieldSchema, igniteValue.toString());
+ break;
+ case FIXED:
+ break;
+ case BYTES:
+ fieldValue = ByteBuffer.wrap((byte[]) igniteValue);
+ break;
+ case STRING:
+ fieldValue = new Utf8(igniteValue.toString());
+ break;
+ case UNION:
+ if (fieldSchema.getTypes().size() == 2 && isNullable(fieldSchema)) {
+ int schemaPos = getUnionSchema(igniteValue, fieldSchema);
+ Schema unionSchema = fieldSchema.getTypes().get(schemaPos);
+ fieldValue = deserializeFieldValue(field, unionSchema, igniteValue, persistent);
+ } else {
+ reader = getDatumReader(fieldSchema);
+ fieldValue = IOUtils.deserialize((byte[]) igniteValue, reader,
+ persistent.get(field.pos()));
+ }
+ break;
+ default:
+ fieldValue = igniteValue;
+ }
+ return fieldValue;
+
+ }
+
+ @Override
+ public void put(K key, T obj) throws GoraException {
+ try {
+ if (obj.isDirty()) {
+ Schema schema = obj.getSchema();
+ List<Schema.Field> fields = schema.getFields();
+ Map<Column, Object> data = new HashMap<>();
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ Column getKey = igniteMapping.getPrimaryKey().get(0);
+ data.put(getKey, key);
+ } else {
+ //Composite keys pending..
+ }
+ for (Schema.Field field : fields) {
+ Column get = igniteMapping.getFields().get(field.name());
+ Object v = obj.get(field.pos());
+ if (get != null && v != null) {
+ Schema fieldSchema = field.schema();
+ Object serializedObj = serializeFieldValue(get, fieldSchema, v);
+ data.put(get, serializedObj);
+ }
+ }
+ String baseInsertStatement = IgniteSQLBuilder.baseInsertStatement(igniteMapping, data);
+ try (PreparedStatement stmt = connection.prepareStatement(baseInsertStatement)) {
+ IgniteSQLBuilder.fillInsertStatement(stmt, data);
+ stmt.executeUpdate();
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ } else {
+ LOG.info("Ignored putting object {} in the store as it is neither "
+ + "new, neither dirty.", new Object[]{obj});
+ }
+ } catch (Exception e) {
+ throw new GoraException(e);
+ }
+ }
+
+ @Override
+ public boolean delete(K key) throws GoraException {
+ String deleteQuery = null;
+ Object[] keyl = null;
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ deleteQuery = IgniteSQLBuilder.delete(igniteMapping);
+ keyl = new Object[]{key};
+ } else {
+ //Composite key pending
+ }
+ try (PreparedStatement stmt = connection.prepareStatement(deleteQuery)) {
+ IgniteSQLBuilder.fillDeleteStatement(stmt, igniteMapping, keyl);
+ stmt.executeUpdate();
+ return true;
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public long deleteByQuery(Query<K, T> query) throws GoraException {
+
+ String deleteQuery;
+ if (query.getFields() != null && query.getFields().length < igniteMapping.getFields().size()) {
+ List<String> dbFields = new ArrayList<>();
+ for (String af : query.getFields()) {
+ dbFields.add(igniteMapping.getFields().get(af).getName());
+ }
+ deleteQuery = IgniteSQLBuilder.deleteQueryFields(igniteMapping, dbFields);
+ } else {
+ deleteQuery = IgniteSQLBuilder.deleteQuery(igniteMapping);
+ }
+ String selectQueryWhere = IgniteSQLBuilder.selectQueryWhere(igniteMapping, query.getStartKey(), query.getEndKey(), query.getLimit());
+ try (PreparedStatement stmt = connection.prepareStatement(deleteQuery + selectQueryWhere)) {
+ IgniteSQLBuilder.fillSelectQuery(stmt, query.getStartKey(), query.getEndKey());
+ stmt.executeUpdate();
+ return 0;
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public Result<K, T> execute(Query<K, T> query) throws GoraException {
+ String[] fields = getFieldsToQuery(query.getFields());
+ //Avro fields to Ignite fields
+ List<String> dbFields = new ArrayList<>();
+ for (String af : fields) {
+ dbFields.add(igniteMapping.getFields().get(af).getName());
+ }
+ String selectQuery = IgniteSQLBuilder.selectQuery(igniteMapping, dbFields);
+ String selectQueryWhere = IgniteSQLBuilder.selectQueryWhere(igniteMapping, query.getStartKey(), query.getEndKey(), query.getLimit());
+ try {
+ PreparedStatement stmt = connection.prepareStatement(selectQuery + selectQueryWhere);
+ RowSetFactory factory = RowSetProvider.newFactory();
+ CachedRowSet rowset = factory.createCachedRowSet();
+ IgniteSQLBuilder.fillSelectQuery(stmt, query.getStartKey(), query.getEndKey());
+ ResultSet executeQuery = stmt.executeQuery();
+ rowset.populate(executeQuery);
+ IgniteResult<K, T> igniteResult = new IgniteResult<>(this, query);
+ igniteResult.setResultSet(rowset);
+ return igniteResult;
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ public K extractKey(ResultSet r) throws SQLException {
+ assert igniteMapping.getPrimaryKey().size() == 1;
+ return (K) r.getObject(igniteMapping.getPrimaryKey().get(0).getName());
--- End diff --
Type safety: Unchecked cast from Object to K
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by nishadi <gi...@git.apache.org>.
Github user nishadi commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202521872
--- Diff: nbactions.xml ---
@@ -0,0 +1,13 @@
+<?xml version="1.0" encoding="UTF-8"?>
--- End diff --
I think this NetBeans related file can be removed.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211527142
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/utils/package-info.java ---
@@ -0,0 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+/**
+ * This package contains Ignite datastore related all classes.
--- End diff --
Please use suitable descriptions for these packages Eg:- This package contains all the utility classes.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211247029
--- Diff: .gitignore ---
@@ -37,3 +37,4 @@ ivy/ivy*.jar
**/gora-solr-5/src/test/conf/solr/collection1/data/
**/_rest_managed.json
**/gora-solr/src/test/conf/solr-managed-schema/collection1/data/
+**/nbactions.xml
--- End diff --
Please remove nbactions.xml as suggested, you don't need this additions to .gitignore file.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by nishadi <gi...@git.apache.org>.
Github user nishadi commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202522746
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteStore.java ---
@@ -0,0 +1,578 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.sql.rowset.CachedRowSet;
+import javax.sql.rowset.RowSetFactory;
+import javax.sql.rowset.RowSetProvider;
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificDatumReader;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.ignite.query.IgniteQuery;
+import org.apache.gora.ignite.query.IgniteResult;
+import org.apache.gora.ignite.utils.IgniteSQLBuilder;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.AvroUtils;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of a Ignite data store to be used by gora.
+ *
+ * @param <K> class to be used for the key
+ * @param <T> class to be persisted within the store
+ */
+public class IgniteStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ public static final Logger LOG = LoggerFactory.getLogger(IgniteStore.class);
+ private static final String PARSE_MAPPING_FILE_KEY = "gora.ignite.mapping.file";
+ private static final String DEFAULT_MAPPING_FILE = "gora-ignite-mapping.xml";
+ private IgniteParameters igniteParameters;
+ private IgniteMapping igniteMapping;
+ private Connection connection;
+
+ /*
+ * Create a threadlocal map for the datum readers and writers, because they
+ * are not thread safe, at least not before Avro 1.4.0 (See AVRO-650). When
+ * they are thread safe, it is possible to maintain a single reader and writer
+ * pair for every schema, instead of one for every thread.
+ */
+ public static final ConcurrentHashMap<Schema, SpecificDatumReader<?>> readerMap = new ConcurrentHashMap<>();
+
+ public static final ConcurrentHashMap<Schema, SpecificDatumWriter<?>> writerMap = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) throws GoraException {
+
+ try {
+ super.initialize(keyClass, persistentClass, properties);
+ IgniteMappingBuilder builder = new IgniteMappingBuilder(this);
+ builder.readMappingFile(getConf().get(PARSE_MAPPING_FILE_KEY, DEFAULT_MAPPING_FILE));
+ igniteMapping = builder.getIgniteMapping();
+ igniteParameters = IgniteParameters.load(properties, conf);
+ connection = acquiereConnection();
+ LOG.info("Ignite store was successfully initialized");
+ } catch (ClassNotFoundException | SQLException ex) {
+ LOG.error("Error while initializing Ignite store", ex);
+ throw new GoraException(ex);
+ }
+ }
+
+ private Connection acquiereConnection() throws ClassNotFoundException, SQLException {
+ Class.forName("org.apache.ignite.IgniteJdbcThinDriver");
+ StringBuilder urlBuilder = new StringBuilder();
+ urlBuilder.append("jdbc:ignite:thin://");
+ urlBuilder.append(igniteParameters.getHost());
+ if (igniteParameters.getPort() != null) {
+ urlBuilder.append(":" + igniteParameters.getPort());
+ }
+ if (igniteParameters.getSchema() != null) {
+ urlBuilder.append("/" + igniteParameters.getSchema());
+ }
+ if (igniteParameters.getUser() != null) {
+ urlBuilder.append(";" + igniteParameters.getUser());
+ }
+ if (igniteParameters.getPassword() != null) {
+ urlBuilder.append(";" + igniteParameters.getPassword());
+ }
+ if (igniteParameters.getAdditionalConfigurations() != null) {
+ urlBuilder.append(igniteParameters.getAdditionalConfigurations());
+ }
+ Connection conn = DriverManager.getConnection(urlBuilder.toString());
--- End diff --
We can directly return the connection rather assigning it to a separate variable.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by nishadi <gi...@git.apache.org>.
Github user nishadi commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202521431
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/package-info.java ---
@@ -0,0 +1,20 @@
+/**
--- End diff --
Shall we make sure that all the license headers are only a comment like below
/*
*
*
*/
rather being in the Java doc comment style
/**
*
*/
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by nishadi <gi...@git.apache.org>.
Github user nishadi commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r205691535
--- Diff: gora-ignite/pom.xml ---
@@ -0,0 +1,182 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora</artifactId>
+ <version>0.9-SNAPSHOT</version>
+ <relativePath>../</relativePath>
+ </parent>
+ <artifactId>gora-ignite</artifactId>
+ <packaging>bundle</packaging>
+
+ <name>Apache Gora :: Ignite</name>
+ <url>http://gora.apache.org</url>
+ <description>The Apache Gora open source framework provides an in-memory data model and
+ persistence for big data. Gora supports persisting to column stores, key value stores,
+ document stores and RDBMSs, and analyzing the data with extensive Apache Hadoop MapReduce
+ support.</description>
+ <inceptionYear>2010</inceptionYear>
+ <organization>
+ <name>The Apache Software Foundation</name>
+ <url>http://www.apache.org/</url>
+ </organization>
+ <issueManagement>
+ <system>JIRA</system>
+ <url>https://issues.apache.org/jira/browse/GORA</url>
+ </issueManagement>
+ <ciManagement>
+ <system>Jenkins</system>
+ <url>https://builds.apache.org/job/Gora-trunk/</url>
+ </ciManagement>
+
+ <properties>
+ <ignite.version>2.4.0</ignite.version>
+ <osgi.import>*</osgi.import>
+ <osgi.export>org.apache.gora.ignite*;version="${project.version}";-noimport:=true</osgi.export>
+ </properties>
+
+ <build>
+ <directory>target</directory>
+ <outputDirectory>target/classes</outputDirectory>
+ <finalName>${project.artifactId}-${project.version}</finalName>
+ <testOutputDirectory>target/test-classes</testOutputDirectory>
+ <testSourceDirectory>src/test/java</testSourceDirectory>
+ <sourceDirectory>src/main/java</sourceDirectory>
+ <testResources>
+ <testResource>
+ <directory>${project.basedir}/src/test/resources</directory>
+ <includes>
+ <include>**/*</include>
+ </includes>
+ <!--targetPath>${project.basedir}/target/classes/</targetPath -->
+ </testResource>
+ </testResources>
+ <plugins>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>build-helper-maven-plugin</artifactId>
+ <version>${build-helper-maven-plugin.version}</version>
+ <executions>
+ <execution>
+ <phase>generate-sources</phase>
+ <goals>
+ <goal>add-source</goal>
+ </goals>
+ <configuration>
+ <sources>
+ <source>src/examples/java</source>
+ </sources>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+
+ <dependencies>
+ <!-- Gora Internal Dependencies -->
+ <dependency>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora-core</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora-core</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+
+ <!--Ignite Dependency -->
+ <dependency>
+ <groupId>org.apache.ignite</groupId>
+ <artifactId>ignite-core</artifactId>
+ <version>${ignite.version}</version>
+ </dependency>
+
--- End diff --
Hi Carlos, Not just the ignite version should be added to the parent POM. This dependency needs to be moved to the dependency management section and referred from there. Have a look at how the dependencies are added in the other data stores as well.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211546949
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/query/IgniteResult.java ---
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.query;
+
+import java.io.IOException;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import org.apache.gora.ignite.store.IgniteStore;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.impl.ResultBase;
+import org.apache.gora.store.DataStore;
+
+/**
+ * IgniteResult specific implementation of the
+ * {@link org.apache.gora.query.Result} interface.
+ */
+public class IgniteResult<K, T extends PersistentBase> extends ResultBase<K, T> {
+
+ private ResultSet resultSet;
+ private int size;
+
+ public IgniteResult(DataStore<K, T> dataStore, Query<K, T> query) {
+ super(dataStore, query);
+ }
+
+ @Override
+ protected boolean nextInner() throws IOException {
+ try {
+ if (!resultSet.next()) {
+ return false;
+ }
+ key = ((IgniteStore<K, T>) getDataStore()).extractKey(resultSet);
+ persistent = ((IgniteStore<K, T>) getDataStore()).newInstance(resultSet, getQuery().getFields());
+ return persistent != null;
+ } catch (SQLException ex) {
+ throw new IOException(ex);
+ }
+ }
+
+ @Override
+ public float getProgress() throws IOException, InterruptedException {
+ if (resultSet == null) {
+ return 0;
+ } else if (size == 0) {
+ return 1;
+ } else {
+ return offset / (float) size;
+ }
+ }
+
+ @Override
+ public int size() {
+ return size;
+ }
+
+ @Override
+ public void close() throws IOException {
+ if (resultSet != null) {
+ try {
+ resultSet.close();
+ } catch (SQLException ex) {
+ throw new IOException(ex);
+ }
+ }
+ }
+
+ public void setResultSet(ResultSet resultSet) throws SQLException {
--- End diff --
Please write a overloaded constructor for IgniteResult(DataStore<K, T> dataStore, Query<K, T> query, ResultSet resultSet) then you can get rid of this public method, resultSet is passed as constructor argument. You can also move the size initialization logic to constructor itself.
I don't this we need expose this public method/api, WDYT? This behavior/method is not exposed from ResultBase, added as public method for IgniteResult. That means this method is not visible for the people who use base class ResultBase methods in their code, you have already extended required base class methods according to Ignite store needs.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by nishadi <gi...@git.apache.org>.
Github user nishadi commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202521966
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteStore.java ---
@@ -0,0 +1,578 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.sql.rowset.CachedRowSet;
+import javax.sql.rowset.RowSetFactory;
+import javax.sql.rowset.RowSetProvider;
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificDatumReader;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.ignite.query.IgniteQuery;
+import org.apache.gora.ignite.query.IgniteResult;
+import org.apache.gora.ignite.utils.IgniteSQLBuilder;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.AvroUtils;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of a Ignite data store to be used by gora.
+ *
+ * @param <K> class to be used for the key
+ * @param <T> class to be persisted within the store
+ */
+public class IgniteStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ public static final Logger LOG = LoggerFactory.getLogger(IgniteStore.class);
+ private static final String PARSE_MAPPING_FILE_KEY = "gora.ignite.mapping.file";
+ private static final String DEFAULT_MAPPING_FILE = "gora-ignite-mapping.xml";
+ private IgniteParameters igniteParameters;
+ private IgniteMapping igniteMapping;
+ private Connection connection;
+
+ /*
+ * Create a threadlocal map for the datum readers and writers, because they
+ * are not thread safe, at least not before Avro 1.4.0 (See AVRO-650). When
+ * they are thread safe, it is possible to maintain a single reader and writer
+ * pair for every schema, instead of one for every thread.
+ */
+ public static final ConcurrentHashMap<Schema, SpecificDatumReader<?>> readerMap = new ConcurrentHashMap<>();
+
+ public static final ConcurrentHashMap<Schema, SpecificDatumWriter<?>> writerMap = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) throws GoraException {
+
+ try {
+ super.initialize(keyClass, persistentClass, properties);
+ IgniteMappingBuilder builder = new IgniteMappingBuilder(this);
+ builder.readMappingFile(getConf().get(PARSE_MAPPING_FILE_KEY, DEFAULT_MAPPING_FILE));
+ igniteMapping = builder.getIgniteMapping();
+ igniteParameters = IgniteParameters.load(properties, conf);
+ connection = acquiereConnection();
+ LOG.info("Ignite store was successfully initialized");
+ } catch (ClassNotFoundException | SQLException ex) {
+ LOG.error("Error while initializing Ignite store", ex);
+ throw new GoraException(ex);
+ }
+ }
+
+ private Connection acquiereConnection() throws ClassNotFoundException, SQLException {
--- End diff --
Need to correct the spellings in acquire
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by alfonsonishikawa <gi...@git.apache.org>.
Github user alfonsonishikawa commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202935976
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteStore.java ---
@@ -0,0 +1,578 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.sql.rowset.CachedRowSet;
+import javax.sql.rowset.RowSetFactory;
+import javax.sql.rowset.RowSetProvider;
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificDatumReader;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.ignite.query.IgniteQuery;
+import org.apache.gora.ignite.query.IgniteResult;
+import org.apache.gora.ignite.utils.IgniteSQLBuilder;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.AvroUtils;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of a Ignite data store to be used by gora.
+ *
+ * @param <K> class to be used for the key
+ * @param <T> class to be persisted within the store
+ */
+public class IgniteStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ public static final Logger LOG = LoggerFactory.getLogger(IgniteStore.class);
+ private static final String PARSE_MAPPING_FILE_KEY = "gora.ignite.mapping.file";
+ private static final String DEFAULT_MAPPING_FILE = "gora-ignite-mapping.xml";
+ private IgniteParameters igniteParameters;
+ private IgniteMapping igniteMapping;
+ private Connection connection;
+
+ /*
+ * Create a threadlocal map for the datum readers and writers, because they
+ * are not thread safe, at least not before Avro 1.4.0 (See AVRO-650). When
+ * they are thread safe, it is possible to maintain a single reader and writer
+ * pair for every schema, instead of one for every thread.
+ */
+ public static final ConcurrentHashMap<Schema, SpecificDatumReader<?>> readerMap = new ConcurrentHashMap<>();
+
+ public static final ConcurrentHashMap<Schema, SpecificDatumWriter<?>> writerMap = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) throws GoraException {
+
+ try {
+ super.initialize(keyClass, persistentClass, properties);
+ IgniteMappingBuilder builder = new IgniteMappingBuilder(this);
+ builder.readMappingFile(getConf().get(PARSE_MAPPING_FILE_KEY, DEFAULT_MAPPING_FILE));
+ igniteMapping = builder.getIgniteMapping();
+ igniteParameters = IgniteParameters.load(properties, conf);
+ connection = acquiereConnection();
+ LOG.info("Ignite store was successfully initialized");
+ } catch (ClassNotFoundException | SQLException ex) {
+ LOG.error("Error while initializing Ignite store", ex);
+ throw new GoraException(ex);
+ }
+ }
+
+ private Connection acquiereConnection() throws ClassNotFoundException, SQLException {
+ Class.forName("org.apache.ignite.IgniteJdbcThinDriver");
+ StringBuilder urlBuilder = new StringBuilder();
+ urlBuilder.append("jdbc:ignite:thin://");
+ urlBuilder.append(igniteParameters.getHost());
+ if (igniteParameters.getPort() != null) {
+ urlBuilder.append(":" + igniteParameters.getPort());
+ }
+ if (igniteParameters.getSchema() != null) {
+ urlBuilder.append("/" + igniteParameters.getSchema());
+ }
+ if (igniteParameters.getUser() != null) {
+ urlBuilder.append(";" + igniteParameters.getUser());
+ }
+ if (igniteParameters.getPassword() != null) {
+ urlBuilder.append(";" + igniteParameters.getPassword());
+ }
+ if (igniteParameters.getAdditionalConfigurations() != null) {
+ urlBuilder.append(igniteParameters.getAdditionalConfigurations());
+ }
+ Connection conn = DriverManager.getConnection(urlBuilder.toString());
+ return conn;
+ }
+
+ @Override
+ public String getSchemaName() {
+ return igniteMapping.getTableName();
+ }
+
+ @Override
+ public String getSchemaName(final String mappingSchemaName,
+ final Class<?> persistentClass) {
+ return super.getSchemaName(mappingSchemaName, persistentClass);
+ }
+
+ @Override
+ public void createSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to create the schema as no connection has been initiated.");
+ }
+ if (schemaExists()) {
+ return;
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String createTableSQL = IgniteSQLBuilder.createTable(igniteMapping);
+ stmt.executeUpdate(createTableSQL);
+ LOG.info("Table {} has been created for Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public void deleteSchema() throws GoraException {
--- End diff --
There is no need to check it here because in IgniteSQLBuilder#dropTable is the guard `IF EXISTS` (https://github.com/apache/gora/pull/134/files/87eddc2495cdeec9b28246ea08708540c365f34e#diff-b3ec8d37ec4251ad56065e9091607199R66)
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211522113
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/utils/IgniteSQLBuilder.java ---
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.utils;
+
+import avro.shaded.com.google.common.collect.Lists;
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+import java.text.MessageFormat;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.gora.ignite.store.Column;
+import org.apache.gora.ignite.store.IgniteMapping;
+
+/**
+ *
+ * SQL Builder utility for Ignite.
+ */
+public class IgniteSQLBuilder {
+
+ private static String format(String pattern, Object... args) {
+ MessageFormat messageFormat = new MessageFormat(pattern, Locale.getDefault());
+ return messageFormat.format(args);
+ }
+
+ /**
+ * Returns a SQL query for determine whether a table exists or not.
+ *
+ * @param tableName The name of the table to be check.
+ * @return SQL query
+ */
+ public static String tableExists(String tableName) {
+ return format("SELECT * FROM {0} LIMIT 0", tableName);
+ }
+
+ /**
+ * Returns a SQL create table statement for initializing a datastore based
+ * upon a Ignite Mapping definition.
+ *
+ * @param mapping The ignite mapping definition of the data store
+ * @return SQL create query (DDL).
+ */
+ public static String createTable(IgniteMapping mapping) {
+ StringBuilder sqlBuilder = new StringBuilder();
+ sqlBuilder.append("CREATE TABLE ");
+ sqlBuilder.append(mapping.getTableName());
+ sqlBuilder.append("(");
+ ArrayList<Column> fieldsList = Lists.newArrayList(mapping.getPrimaryKey());
+ fieldsList.addAll(Lists.newArrayList(mapping.getFields().values()));
+ for (Column aColumn : fieldsList) {
+ String name = aColumn.getName();
+ Column.FieldType dataType = aColumn.getDataType();
+ sqlBuilder.append(name).append(" ").append(dataType.toString()).append(",");
+ }
+ sqlBuilder.append("PRIMARY KEY ");
+ sqlBuilder.append("(");
+ for (int i = 0; i < mapping.getPrimaryKey().size(); i++) {
+ sqlBuilder.append(mapping.getPrimaryKey().get(i).getName());
+ sqlBuilder.append(i == mapping.getPrimaryKey().size() - 1 ? "" : ",");
+ }
+ sqlBuilder.append(")");
+ sqlBuilder.append(");");
+ return sqlBuilder.toString();
+ }
+
+ /**
+ * Returns a SQL drop table statement for deleting a datastore instance within
+ * ignite.
+ *
+ * @param tableName The name of the table to be dropped.
+ * @return SQL drop query (DDL).
+ */
+ public static String dropTable(String tableName) {
+ return format("DROP TABLE IF EXISTS {0} ;", tableName);
+ }
+
+ /**
+ * Returns a bare SQL insert statement for adding a new record on a Ignite
+ * data store.
+ *
+ * @param mapping The ignite mapping definition of the data store
+ * @param data A map containing the Column-Value pairs of the new record.
+ * @return SQL insert statement
+ */
+ public static String baseInsertStatement(IgniteMapping mapping, Map<Column, Object> data) {
+ StringBuilder sqlBuilder = new StringBuilder();
+ sqlBuilder.append("MERGE INTO ");
+ sqlBuilder.append(mapping.getTableName());
+ sqlBuilder.append(" (");
+ List<Entry<Column, Object>> list = new ArrayList<>(data.entrySet());
+ for (int i = 0; i < list.size(); i++) {
+ sqlBuilder.append(list.get(i).getKey().getName());
+ sqlBuilder.append(i == list.size() - 1 ? "" : ",");
+ }
+ sqlBuilder.append(")");
+ sqlBuilder.append(" VALUES ");
+ sqlBuilder.append(" (");
+ for (int i = 0; i < list.size(); i++) {
+ sqlBuilder.append("?");
+ sqlBuilder.append(i == list.size() - 1 ? "" : ",");
+ }
+ sqlBuilder.append(" )");
+ return sqlBuilder.toString();
+ }
+
+ /**
+ * Fills a SQL PreparedStatement of a insert operation with the actual data to
+ * be inserted.
+ *
+ * @param statement The insertion PreparedStatement to be filled.
+ * @param insertData A map containing the Column-Value pairs of the new
+ * record.
+ * @throws SQLException When invalid values are provided as parameters for the
+ * insert statement.
+ */
+ public static void fillInsertStatement(PreparedStatement statement, Map<Column, Object> insertData) throws SQLException {
+ List<Entry<Column, Object>> list = new ArrayList<>(insertData.entrySet());
+ for (int i = 0; i < list.size(); i++) {
+ int j = i + 1;
+ statement.setObject(j, list.get(i).getValue());
+ }
+ }
+
+ /**
+ * Returns a bare SQL statement for deleting a record from the Ignite data
+ * store.
+ *
+ * @param mapping The ignite mapping definition of the data store
+ * @return SQL delete statement
+ */
+ public static String delete(IgniteMapping mapping) {
+ StringBuilder sqlBuilder = new StringBuilder();
+ sqlBuilder.append("DELETE FROM ");
+ sqlBuilder.append(mapping.getTableName());
+ sqlBuilder.append(" WHERE ");
+ for (int i = 0; i < mapping.getPrimaryKey().size(); i++) {
+ sqlBuilder.append(mapping.getPrimaryKey().get(i).getName());
+ sqlBuilder.append("= ? ");
+ sqlBuilder.append(i == mapping.getPrimaryKey().size() - 1 ? "" : " AND ");
+ }
+ return sqlBuilder.toString();
+ }
+
+ /**
+ * Fills a SQL PreparedStatement of a delete operation with the actual key of
+ * the record to be deleted
+ *
+ * @param statement The deletion PreparedStatement to be filled.
+ * @param mapping The ignite mapping definition of the data store
+ * @param deleteData An Object array containing the primary key values of the
+ * record to be deleted
+ * @throws SQLException When invalid keys' values are provided as parameters
+ */
+ public static void fillDeleteStatement(PreparedStatement statement, IgniteMapping mapping, Object... deleteData) throws SQLException {
+ assert mapping.getPrimaryKey().size() == deleteData.length;
+ for (int i = 0; i < mapping.getPrimaryKey().size(); i++) {
+ int j = i + 1;
+ statement.setObject(j, deleteData[i]);
+ }
+ }
+
+ /**
+ * Returns a bare SQL statement for retrieving a record from the ignite data
+ * store
+ *
+ * @param mapping The ignite mapping definition of the data store
+ * @param columns A list of columns to be retrieved within the select query
+ * @return SQL select statement
+ */
+ public static String selectGet(IgniteMapping mapping, List<String> columns) {
+ StringBuilder sqlBuilder = new StringBuilder();
+ sqlBuilder.append("SELECT ");
+ for (int i = 0; i < columns.size(); i++) {
+ sqlBuilder.append(columns.get(i));
+ sqlBuilder.append(i == columns.size() - 1 ? "" : " , ");
+ }
+ sqlBuilder.append(" FROM ");
+ sqlBuilder.append(mapping.getTableName());
+ sqlBuilder.append(" WHERE ");
+ for (int i = 0; i < mapping.getPrimaryKey().size(); i++) {
+ sqlBuilder.append(mapping.getPrimaryKey().get(i).getName());
+ sqlBuilder.append("= ? ");
+ sqlBuilder.append(i == mapping.getPrimaryKey().size() - 1 ? "" : " AND ");
+ }
+ return sqlBuilder.toString();
+ }
+
+ /**
+ * Fills a SQL PreparedStatement of a select operation with the actual keys of
+ * the record to be retrieved
+ *
+ * @param statement The select PreparedStatement to be filled.
+ * @param mapping The ignite mapping definition of the data store
+ * @param selectData An Object array containing the primary key values of the
+ * record to be retrieved
+ * @throws SQLException When invalid keys' values are provided as parameters
+ */
+ public static void fillSelectStatement(PreparedStatement statement, IgniteMapping mapping, Object... selectData) throws SQLException {
+ assert mapping.getPrimaryKey().size() == selectData.length;
+ for (int i = 0; i < mapping.getPrimaryKey().size(); i++) {
+ int j = i + 1;
+ statement.setObject(j, selectData[i]);
+ }
+ }
+
+ /**
+ * Returns a base SQL statement for retrieving multiple records from the
+ * ignite data store
+ *
+ * @param mapping The ignite mapping definition of the data store
+ * @param selectFields A list of columns to be retrieved within the select
+ * query
+ * @return SQL select statement
+ */
+ public static String selectQuery(IgniteMapping mapping, List<String> selectFields) {
+ List<String> fields = new ArrayList<>();
+ for (Column c : mapping.getPrimaryKey()) {
+ fields.add(c.getName());
+ }
+ fields.addAll(selectFields);
+ StringBuilder sqlBuilder = new StringBuilder();
+ sqlBuilder.append("SELECT ");
+ for (int i = 0; i < fields.size(); i++) {
+ sqlBuilder.append(fields.get(i));
+ sqlBuilder.append(i == fields.size() - 1 ? "" : " , ");
+ }
+ sqlBuilder.append(" FROM ");
+ sqlBuilder.append(mapping.getTableName());
+ return sqlBuilder.toString();
+ }
+
+ /**
+ * Returns a base SQL statement for deleting multiple records from the ignite
+ * data store
+ *
+ * @param mapping The ignite mapping definition of the data store
+ * @return SQL delete statement
+ */
+ public static String deleteQuery(IgniteMapping mapping) {
+ StringBuilder sqlBuilder = new StringBuilder();
+ sqlBuilder.append("DELETE FROM ");
+ sqlBuilder.append(mapping.getTableName());
+ return sqlBuilder.toString();
+ }
+
+ /**
+ * Returns a base SQL statement for deleting fields from records of the ignite
+ * data store
+ *
+ * @param mapping The ignite mapping definition of the data store
+ * @param deleteFields A list of columns to be deleted (set to null)
+ * @return SQL update statement
+ */
+ public static String deleteQueryFields(IgniteMapping mapping, List<String> deleteFields) {
+ StringBuilder sqlBuilder = new StringBuilder();
+ sqlBuilder.append("UPDATE ");
+ sqlBuilder.append(mapping.getTableName());
+ if (!deleteFields.isEmpty()) {
+ sqlBuilder.append(" SET ");
+ }
+ for (int i = 0; i < deleteFields.size(); i++) {
+ sqlBuilder.append(deleteFields.get(i));
+ sqlBuilder.append(" = null");
+ sqlBuilder.append(i == deleteFields.size() - 1 ? "" : " , ");
+ }
+ return sqlBuilder.toString();
+ }
+
+ /**
+ * Returns a SQL's WHERE segment with proper conditions set for
+ * Querying/Deleting/Updating multiple records of a ignite data store
+ *
+ * @param mapping The ignite mapping definition of the data store
+ * @param startKey Start key of the WHERE condition
+ * @param endKey End key of the WHERE condition
+ * @param limit The maximum number of records to be consider
+ * @return SQL WHERE segment
+ */
+ public static String queryWhere(IgniteMapping mapping, Object startKey, Object endKey, long limit) {
+ //composite keys pending
+ assert mapping.getPrimaryKey().size() == 1;
+ String keycolumn = mapping.getPrimaryKey().get(0).getName();
+ StringBuilder sqlBuilder = new StringBuilder();
+ if (startKey != null || endKey != null) {
+ sqlBuilder.append(" WHERE ");
+ if (startKey != null && endKey != null && startKey.equals(endKey)) {
+ sqlBuilder.append(keycolumn);
+ sqlBuilder.append("= ?");
+ } else {
+ if (startKey != null) {
+ sqlBuilder.append(keycolumn);
+ sqlBuilder.append(">= ?");
+ }
+ if (startKey != null && endKey != null) {
+ sqlBuilder.append(" AND ");
+ }
+ if (endKey != null) {
+ sqlBuilder.append(keycolumn);
+ sqlBuilder.append("<= ?");
+ }
+ }
+ }
+ if (limit > 0) {
+ sqlBuilder.append(" LIMIT ").append(limit);
+ }
+ return sqlBuilder.toString();
+ }
+
+ /**
+ * Fills a SQL PreparedStatement's WHERE segment of a select/delete/update
+ * operation with proper key values
+ *
+ * @param statement The select PreparedStatement to be filled.
+ * @param startKey Start key of the WHERE condition
+ * @param endKey End key of the WHERE condition
+ * @throws SQLException When invalid keys' values are provided as parameters
+ */
+ public static void fillSelectQuery(PreparedStatement statement, Object startKey, Object endKey) throws SQLException {
--- End diff --
Please rename this method to suitable name - This method is not only being used for filling select queries it is also used for delete and update as well.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202526805
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteStore.java ---
@@ -0,0 +1,578 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.sql.rowset.CachedRowSet;
+import javax.sql.rowset.RowSetFactory;
+import javax.sql.rowset.RowSetProvider;
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificDatumReader;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.ignite.query.IgniteQuery;
+import org.apache.gora.ignite.query.IgniteResult;
+import org.apache.gora.ignite.utils.IgniteSQLBuilder;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.AvroUtils;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of a Ignite data store to be used by gora.
+ *
+ * @param <K> class to be used for the key
+ * @param <T> class to be persisted within the store
+ */
+public class IgniteStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ public static final Logger LOG = LoggerFactory.getLogger(IgniteStore.class);
+ private static final String PARSE_MAPPING_FILE_KEY = "gora.ignite.mapping.file";
+ private static final String DEFAULT_MAPPING_FILE = "gora-ignite-mapping.xml";
+ private IgniteParameters igniteParameters;
+ private IgniteMapping igniteMapping;
+ private Connection connection;
+
+ /*
+ * Create a threadlocal map for the datum readers and writers, because they
+ * are not thread safe, at least not before Avro 1.4.0 (See AVRO-650). When
+ * they are thread safe, it is possible to maintain a single reader and writer
+ * pair for every schema, instead of one for every thread.
+ */
+ public static final ConcurrentHashMap<Schema, SpecificDatumReader<?>> readerMap = new ConcurrentHashMap<>();
+
+ public static final ConcurrentHashMap<Schema, SpecificDatumWriter<?>> writerMap = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) throws GoraException {
+
+ try {
+ super.initialize(keyClass, persistentClass, properties);
+ IgniteMappingBuilder builder = new IgniteMappingBuilder(this);
+ builder.readMappingFile(getConf().get(PARSE_MAPPING_FILE_KEY, DEFAULT_MAPPING_FILE));
+ igniteMapping = builder.getIgniteMapping();
+ igniteParameters = IgniteParameters.load(properties, conf);
+ connection = acquiereConnection();
+ LOG.info("Ignite store was successfully initialized");
+ } catch (ClassNotFoundException | SQLException ex) {
+ LOG.error("Error while initializing Ignite store", ex);
+ throw new GoraException(ex);
+ }
+ }
+
+ private Connection acquiereConnection() throws ClassNotFoundException, SQLException {
+ Class.forName("org.apache.ignite.IgniteJdbcThinDriver");
+ StringBuilder urlBuilder = new StringBuilder();
+ urlBuilder.append("jdbc:ignite:thin://");
+ urlBuilder.append(igniteParameters.getHost());
+ if (igniteParameters.getPort() != null) {
+ urlBuilder.append(":" + igniteParameters.getPort());
+ }
+ if (igniteParameters.getSchema() != null) {
+ urlBuilder.append("/" + igniteParameters.getSchema());
+ }
+ if (igniteParameters.getUser() != null) {
+ urlBuilder.append(";" + igniteParameters.getUser());
+ }
+ if (igniteParameters.getPassword() != null) {
+ urlBuilder.append(";" + igniteParameters.getPassword());
+ }
+ if (igniteParameters.getAdditionalConfigurations() != null) {
+ urlBuilder.append(igniteParameters.getAdditionalConfigurations());
+ }
+ Connection conn = DriverManager.getConnection(urlBuilder.toString());
+ return conn;
+ }
+
+ @Override
+ public String getSchemaName() {
+ return igniteMapping.getTableName();
+ }
+
+ @Override
+ public String getSchemaName(final String mappingSchemaName,
+ final Class<?> persistentClass) {
+ return super.getSchemaName(mappingSchemaName, persistentClass);
+ }
+
+ @Override
+ public void createSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to create the schema as no connection has been initiated.");
+ }
+ if (schemaExists()) {
+ return;
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String createTableSQL = IgniteSQLBuilder.createTable(igniteMapping);
+ stmt.executeUpdate(createTableSQL);
+ LOG.info("Table {} has been created for Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public void deleteSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to delete the schema as no connection has been initiated.");
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String dropTableSQL = IgniteSQLBuilder.dropTable(igniteMapping.getTableName());
+ stmt.executeUpdate(dropTableSQL);
+ LOG.info("Table {} has been dropped from Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public boolean schemaExists() throws GoraException {
+ boolean exists = false;
+ try (Statement stmt = connection.createStatement()) {
+ String tableExistsSQL = IgniteSQLBuilder.tableExists(igniteMapping.getTableName());
+ ResultSet executeQuery = stmt.executeQuery(tableExistsSQL);
+ executeQuery.close();
+ exists = true;
+ } catch (SQLException ex) {
+ /**
+ * a 42000 error code is thrown by Ignite when a non-existent table
+ * queried. More details:
+ * https://apacheignite-sql.readme.io/docs/jdbc-error-codes
+ */
+ if (ex.getSQLState() != null && ex.getSQLState().equals("42000")) {
+ exists = false;
+ } else {
+ throw new GoraException(ex);
+ }
+ }
+ return exists;
+ }
+
+ @Override
+ public T get(K key, String[] fields) throws GoraException {
+ String[] avFields = getFieldsToQuery(fields);
+ Object[] keyl = null;
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ keyl = new Object[]{key};
+ } else {
+ //Composite key pending
+ }
+ //Avro fields to Ignite fields
+ List<String> dbFields = new ArrayList<>();
+ for (String af : avFields) {
+ dbFields.add(igniteMapping.getFields().get(af).getName());
+ }
+ String selectQuery = IgniteSQLBuilder.selectGet(igniteMapping, dbFields);
+ try (PreparedStatement stmt = connection.prepareStatement(selectQuery)) {
+ IgniteSQLBuilder.fillSelectStatement(stmt, igniteMapping, keyl);
+ ResultSet rs = stmt.executeQuery();
+ boolean data = rs.next();
+ T resp = null;
+ if (data) {
+ resp = newInstance(rs, fields);
+ if (rs.next()) {
+ LOG.warn("Multiple results for primary key {} in the schema {}, ignoring additional rows.", keyl, igniteMapping.getTableName());
+ }
+ }
+ rs.close();
+ return resp;
+ } catch (SQLException | IOException ex) {
+ throw new GoraException(ex);
+ }
+
+ }
+
+ public T newInstance(ResultSet rs, String[] fields) throws GoraException, SQLException, IOException {
+ fields = getFieldsToQuery(fields);
+ T persistent = newPersistent();
+ for (String f : fields) {
+ Schema.Field field = fieldMap.get(f);
+ Schema fieldSchema = field.schema();
+ String dbField = igniteMapping.getFields().get(f).getName();
+ Object sv = rs.getObject(dbField);
+ if (sv == null) {
+ continue;
+ }
+ Object v = deserializeFieldValue(field, fieldSchema, sv, persistent);
+ persistent.put(field.pos(), v);
+ persistent.setDirty(field.pos());
+ }
+ return persistent;
+ }
+
+ private Object deserializeFieldValue(Schema.Field field, Schema fieldSchema,
+ Object igniteValue, T persistent) throws IOException {
+ Object fieldValue = null;
+ switch (fieldSchema.getType()) {
+ case MAP:
+ case ARRAY:
+ case RECORD:
+ @SuppressWarnings("rawtypes") SpecificDatumReader reader = getDatumReader(fieldSchema);
+ fieldValue = IOUtils.deserialize((byte[]) igniteValue, reader,
+ persistent.get(field.pos()));
+ break;
+ case ENUM:
+ fieldValue = AvroUtils.getEnumValue(fieldSchema, igniteValue.toString());
+ break;
+ case FIXED:
+ break;
+ case BYTES:
+ fieldValue = ByteBuffer.wrap((byte[]) igniteValue);
+ break;
+ case STRING:
+ fieldValue = new Utf8(igniteValue.toString());
+ break;
+ case UNION:
+ if (fieldSchema.getTypes().size() == 2 && isNullable(fieldSchema)) {
+ int schemaPos = getUnionSchema(igniteValue, fieldSchema);
+ Schema unionSchema = fieldSchema.getTypes().get(schemaPos);
+ fieldValue = deserializeFieldValue(field, unionSchema, igniteValue, persistent);
+ } else {
+ reader = getDatumReader(fieldSchema);
+ fieldValue = IOUtils.deserialize((byte[]) igniteValue, reader,
+ persistent.get(field.pos()));
+ }
+ break;
+ default:
+ fieldValue = igniteValue;
+ }
+ return fieldValue;
+
+ }
+
+ @Override
+ public void put(K key, T obj) throws GoraException {
+ try {
+ if (obj.isDirty()) {
+ Schema schema = obj.getSchema();
+ List<Schema.Field> fields = schema.getFields();
+ Map<Column, Object> data = new HashMap<>();
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ Column getKey = igniteMapping.getPrimaryKey().get(0);
+ data.put(getKey, key);
+ } else {
+ //Composite keys pending..
+ }
+ for (Schema.Field field : fields) {
+ Column get = igniteMapping.getFields().get(field.name());
+ Object v = obj.get(field.pos());
+ if (get != null && v != null) {
+ Schema fieldSchema = field.schema();
+ Object serializedObj = serializeFieldValue(get, fieldSchema, v);
+ data.put(get, serializedObj);
+ }
+ }
+ String baseInsertStatement = IgniteSQLBuilder.baseInsertStatement(igniteMapping, data);
+ try (PreparedStatement stmt = connection.prepareStatement(baseInsertStatement)) {
+ IgniteSQLBuilder.fillInsertStatement(stmt, data);
+ stmt.executeUpdate();
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ } else {
+ LOG.info("Ignored putting object {} in the store as it is neither "
+ + "new, neither dirty.", new Object[]{obj});
+ }
+ } catch (Exception e) {
+ throw new GoraException(e);
+ }
+ }
+
+ @Override
+ public boolean delete(K key) throws GoraException {
+ String deleteQuery = null;
+ Object[] keyl = null;
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ deleteQuery = IgniteSQLBuilder.delete(igniteMapping);
+ keyl = new Object[]{key};
+ } else {
+ //Composite key pending
+ }
+ try (PreparedStatement stmt = connection.prepareStatement(deleteQuery)) {
+ IgniteSQLBuilder.fillDeleteStatement(stmt, igniteMapping, keyl);
+ stmt.executeUpdate();
+ return true;
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public long deleteByQuery(Query<K, T> query) throws GoraException {
+
+ String deleteQuery;
+ if (query.getFields() != null && query.getFields().length < igniteMapping.getFields().size()) {
+ List<String> dbFields = new ArrayList<>();
+ for (String af : query.getFields()) {
+ dbFields.add(igniteMapping.getFields().get(af).getName());
+ }
+ deleteQuery = IgniteSQLBuilder.deleteQueryFields(igniteMapping, dbFields);
+ } else {
+ deleteQuery = IgniteSQLBuilder.deleteQuery(igniteMapping);
+ }
+ String selectQueryWhere = IgniteSQLBuilder.selectQueryWhere(igniteMapping, query.getStartKey(), query.getEndKey(), query.getLimit());
+ try (PreparedStatement stmt = connection.prepareStatement(deleteQuery + selectQueryWhere)) {
+ IgniteSQLBuilder.fillSelectQuery(stmt, query.getStartKey(), query.getEndKey());
+ stmt.executeUpdate();
+ return 0;
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public Result<K, T> execute(Query<K, T> query) throws GoraException {
+ String[] fields = getFieldsToQuery(query.getFields());
+ //Avro fields to Ignite fields
+ List<String> dbFields = new ArrayList<>();
+ for (String af : fields) {
+ dbFields.add(igniteMapping.getFields().get(af).getName());
+ }
+ String selectQuery = IgniteSQLBuilder.selectQuery(igniteMapping, dbFields);
+ String selectQueryWhere = IgniteSQLBuilder.selectQueryWhere(igniteMapping, query.getStartKey(), query.getEndKey(), query.getLimit());
+ try {
+ PreparedStatement stmt = connection.prepareStatement(selectQuery + selectQueryWhere);
+ RowSetFactory factory = RowSetProvider.newFactory();
+ CachedRowSet rowset = factory.createCachedRowSet();
+ IgniteSQLBuilder.fillSelectQuery(stmt, query.getStartKey(), query.getEndKey());
+ ResultSet executeQuery = stmt.executeQuery();
+ rowset.populate(executeQuery);
+ IgniteResult<K, T> igniteResult = new IgniteResult<>(this, query);
+ igniteResult.setResultSet(rowset);
+ return igniteResult;
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ public K extractKey(ResultSet r) throws SQLException {
+ assert igniteMapping.getPrimaryKey().size() == 1;
+ return (K) r.getObject(igniteMapping.getPrimaryKey().get(0).getName());
+ }
+
+ @Override
+ public Query<K, T> newQuery() {
+ IgniteQuery<K, T> query = new IgniteQuery<>(this);
+ query.setFields(getFieldsToQuery(null));
+ return query;
+ }
+
+ @Override
+ public List<PartitionQuery<K, T>> getPartitions(Query<K, T> query) throws IOException {
+ List<PartitionQuery<K, T>> partitions = new ArrayList<>();
+ PartitionQueryImpl<K, T> partitionQuery = new PartitionQueryImpl<>(
+ query);
+ partitionQuery.setConf(getConf());
+ partitions.add(partitionQuery);
+ return partitions;
+ }
+
+ @Override
+ public void flush() throws GoraException {
+ //Auto-commit mode by default
+ }
+
+ @Override
+ public void close() {
+ try {
+ connection.close();
+ LOG.info("Ignite datastore destroyed successfully.");
+ } catch (Exception ex) {
+ LOG.error(ex.getMessage(), ex);
+ }
+ }
+
+ private Object serializeFieldValue(Column get, Schema fieldSchema, Object fieldValue) {
+ Object output = fieldValue;
+ switch (fieldSchema.getType()) {
+ case ARRAY:
+ case MAP:
+ case RECORD:
+ byte[] data = null;
+ try {
+ @SuppressWarnings("rawtypes")
+ SpecificDatumWriter writer = getDatumWriter(fieldSchema);
+ data = IOUtils.serialize(writer, fieldValue);
+ } catch (IOException e) {
+ LOG.error(e.getMessage(), e);
+ }
+ output = data;
+ break;
+ case UNION:
+ if (fieldSchema.getTypes().size() == 2 && isNullable(fieldSchema)) {
+ int schemaPos = getUnionSchema(fieldValue, fieldSchema);
+ Schema unionSchema = fieldSchema.getTypes().get(schemaPos);
+ output = serializeFieldValue(get, unionSchema, fieldValue);
+ } else {
+ data = null;
+ try {
+ @SuppressWarnings("rawtypes")
+ SpecificDatumWriter writer = getDatumWriter(fieldSchema);
+ data = IOUtils.serialize(writer, fieldValue);
--- End diff --
Type safety: Unchecked invocation serialize(SpecificDatumWriter, Object) of the generic method serialize(SpecificDatumWriter< T >, T) of type IOUtils
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202526771
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteMappingBuilder.java ---
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import com.google.inject.ConfigurationException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.jdom.Document;
+import org.jdom.Element;
+import org.jdom.JDOMException;
+import org.jdom.input.SAXBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Builder for Mapping definitions of Ignite.
+ */
+public class IgniteMappingBuilder<K, T extends PersistentBase> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IgniteMappingBuilder.class);
+ /**
+ * Mapping instance being built
+ */
+ private IgniteMapping igniteMapping;
+
+ private final IgniteStore<K, T> dataStore;
+
+ public IgniteMappingBuilder(final IgniteStore<K, T> store) {
+ this.igniteMapping = new IgniteMapping();
+ this.dataStore = store;
+ }
+
+ public IgniteMapping getIgniteMapping() {
+ return igniteMapping;
+ }
+
+ public void setIgniteMapping(IgniteMapping igniteMapping) {
+ this.igniteMapping = igniteMapping;
+ }
+
+ public void readMappingFile(String mappingFile) {
+ try {
+ SAXBuilder saxBuilder = new SAXBuilder();
+ InputStream inputStream = getClass().getClassLoader().getResourceAsStream(mappingFile);
+ if (inputStream == null) {
+ LOG.error("Mapping file '{}' could not be found!", mappingFile);
+ throw new IOException("Mapping file '" + mappingFile + "' could not be found!");
+ }
+ Document document = saxBuilder.build(inputStream);
+ if (document == null) {
+ LOG.error("Mapping file '{}' could not be found!", mappingFile);
+ throw new IOException("Mapping file '" + mappingFile + "' could not be found!");
+ }
+ List<Element> classes = document.getRootElement().getChildren("class");
+ for (Element classElement : classes) {
+ if (classElement.getAttributeValue("keyClass").equals(
+ dataStore.getKeyClass().getCanonicalName())
+ && classElement.getAttributeValue("name").equals(
+ dataStore.getPersistentClass().getCanonicalName())) {
+ final String tableNameFromMapping = classElement.getAttributeValue("table");
+ String tableName = dataStore.getSchemaName(tableNameFromMapping, dataStore.getPersistentClass());
+ igniteMapping.setTableName(tableName);
+ List<Element> prColumns = classElement.getChildren("primarykey");
+ List<Column> prFields = new ArrayList<>();
+ for (Element aPrimaryKey : prColumns) {
+ String name = aPrimaryKey.getAttributeValue("column");
+ String type = aPrimaryKey.getAttributeValue("type");
+ prFields.add(new Column(name, Column.FieldType.valueOf(type)));
+ }
+ igniteMapping.setPrimaryKey(prFields);
+ List<Element> fields = classElement.getChildren("field");
--- End diff --
Type safety: The expression of type List needs unchecked conversion to conform to List < Element >
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by nishadi <gi...@git.apache.org>.
Github user nishadi commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202521891
--- Diff: gora-ignite/src/examples/java/.gitignore ---
@@ -0,0 +1,15 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
--- End diff --
Is there any specific reason why we are adding this file?
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211546255
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/query/package-info.java ---
@@ -0,0 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+/**
+ * This package contains Ignite datastore related all classes.
--- End diff --
Please use suitable description for the query package.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/gora/pull/134
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211545479
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteStore.java ---
@@ -0,0 +1,565 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.sql.rowset.CachedRowSet;
+import javax.sql.rowset.RowSetFactory;
+import javax.sql.rowset.RowSetProvider;
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificDatumReader;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.ignite.query.IgniteQuery;
+import org.apache.gora.ignite.query.IgniteResult;
+import org.apache.gora.ignite.utils.IgniteBackendConstants;
+import org.apache.gora.ignite.utils.IgniteSQLBuilder;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.AvroUtils;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of a Ignite data store to be used by gora.
+ *
+ * @param <K> class to be used for the key
+ * @param <T> class to be persisted within the store
+ */
+public class IgniteStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IgniteStore.class);
+ private static final String PARSE_MAPPING_FILE_KEY = "gora.ignite.mapping.file";
+ private static final String DEFAULT_MAPPING_FILE = "gora-ignite-mapping.xml";
+ private IgniteParameters igniteParameters;
+ private IgniteMapping igniteMapping;
+ private Connection connection;
+ private static final ConcurrentHashMap<Schema, SpecificDatumReader<?>> readerMap = new ConcurrentHashMap<>();
+ private static final ConcurrentHashMap<Schema, SpecificDatumWriter<?>> writerMap = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) throws GoraException {
+ try {
+ super.initialize(keyClass, persistentClass, properties);
+ IgniteMappingBuilder<K, T> builder = new IgniteMappingBuilder<K, T>(this);
+ builder.readMappingFile(getConf().get(PARSE_MAPPING_FILE_KEY, DEFAULT_MAPPING_FILE));
+ igniteMapping = builder.getIgniteMapping();
+ igniteParameters = IgniteParameters.load(properties);
+ connection = acquireConnection();
+ LOG.info("Ignite store was successfully initialized");
+ } catch (ClassNotFoundException | SQLException ex) {
+ LOG.error("Error while initializing Ignite store", ex);
+ throw new GoraException(ex);
+ }
+ }
+
+ private Connection acquireConnection() throws ClassNotFoundException, SQLException {
+ Class.forName("org.apache.ignite.IgniteJdbcThinDriver");
--- End diff --
You may use String constant here, move to constant file and refer from there.
Eg:- DRIVER_NAME = org.apache.ignite.IgniteJdbcThinDriver;
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211221395
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/Column.java ---
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+public class Column {
+
+ private String name;
+ private FieldType dataType;
+
+ public Column(String name, FieldType dataType) {
+ this.name = name;
+ this.dataType = dataType;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public void setName(String name) {
+ this.name = name;
+ }
+
+ public FieldType getDataType() {
+ return dataType;
+ }
+
+ public void setDataType(FieldType dataType) {
+ this.dataType = dataType;
+ }
+
+
--- End diff --
Please remove these unnecessary new lines.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211539151
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteStore.java ---
@@ -0,0 +1,565 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.sql.rowset.CachedRowSet;
+import javax.sql.rowset.RowSetFactory;
+import javax.sql.rowset.RowSetProvider;
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificDatumReader;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.ignite.query.IgniteQuery;
+import org.apache.gora.ignite.query.IgniteResult;
+import org.apache.gora.ignite.utils.IgniteBackendConstants;
+import org.apache.gora.ignite.utils.IgniteSQLBuilder;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.AvroUtils;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of a Ignite data store to be used by gora.
+ *
+ * @param <K> class to be used for the key
+ * @param <T> class to be persisted within the store
+ */
+public class IgniteStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IgniteStore.class);
+ private static final String PARSE_MAPPING_FILE_KEY = "gora.ignite.mapping.file";
+ private static final String DEFAULT_MAPPING_FILE = "gora-ignite-mapping.xml";
+ private IgniteParameters igniteParameters;
+ private IgniteMapping igniteMapping;
+ private Connection connection;
+ private static final ConcurrentHashMap<Schema, SpecificDatumReader<?>> readerMap = new ConcurrentHashMap<>();
+ private static final ConcurrentHashMap<Schema, SpecificDatumWriter<?>> writerMap = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) throws GoraException {
+ try {
+ super.initialize(keyClass, persistentClass, properties);
+ IgniteMappingBuilder<K, T> builder = new IgniteMappingBuilder<K, T>(this);
+ builder.readMappingFile(getConf().get(PARSE_MAPPING_FILE_KEY, DEFAULT_MAPPING_FILE));
+ igniteMapping = builder.getIgniteMapping();
+ igniteParameters = IgniteParameters.load(properties);
+ connection = acquireConnection();
+ LOG.info("Ignite store was successfully initialized");
+ } catch (ClassNotFoundException | SQLException ex) {
+ LOG.error("Error while initializing Ignite store", ex);
+ throw new GoraException(ex);
+ }
+ }
+
+ private Connection acquireConnection() throws ClassNotFoundException, SQLException {
+ Class.forName("org.apache.ignite.IgniteJdbcThinDriver");
+ StringBuilder urlBuilder = new StringBuilder();
+ urlBuilder.append("jdbc:ignite:thin://");
+ urlBuilder.append(igniteParameters.getHost());
+ if (igniteParameters.getPort() != null) {
+ urlBuilder.append(":" + igniteParameters.getPort());
+ }
+ if (igniteParameters.getSchema() != null) {
+ urlBuilder.append("/" + igniteParameters.getSchema());
+ }
+ if (igniteParameters.getUser() != null) {
+ urlBuilder.append(";" + igniteParameters.getUser());
+ }
+ if (igniteParameters.getPassword() != null) {
+ urlBuilder.append(";" + igniteParameters.getPassword());
+ }
+ if (igniteParameters.getAdditionalConfigurations() != null) {
+ urlBuilder.append(igniteParameters.getAdditionalConfigurations());
+ }
+ return DriverManager.getConnection(urlBuilder.toString());
+ }
+
+ @Override
+ public String getSchemaName() {
+ return igniteMapping.getTableName();
+ }
+
+ @Override
+ public String getSchemaName(final String mappingSchemaName,
+ final Class<?> persistentClass) {
+ return super.getSchemaName(mappingSchemaName, persistentClass);
+ }
+
+ @Override
+ public void createSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to create the schema as no connection has been initiated.");
+ }
+ if (schemaExists()) {
+ return;
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String createTableSQL = IgniteSQLBuilder.createTable(igniteMapping);
+ stmt.executeUpdate(createTableSQL);
+ LOG.info("Table {} has been created for Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public void deleteSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to delete the schema as no connection has been initiated.");
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String dropTableSQL = IgniteSQLBuilder.dropTable(igniteMapping.getTableName());
+ stmt.executeUpdate(dropTableSQL);
+ LOG.info("Table {} has been dropped from Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public boolean schemaExists() throws GoraException {
+ try (Statement stmt = connection.createStatement()) {
+ String tableExistsSQL = IgniteSQLBuilder.tableExists(igniteMapping.getTableName());
+ ResultSet executeQuery = stmt.executeQuery(tableExistsSQL);
+ executeQuery.close();
+ return true;
+ } catch (SQLException ex) {
+ if (ex.getSQLState() != null
+ && ex.getSQLState().equals(IgniteBackendConstants.DEFAULT_IGNITE_TABLE_NOT_EXISTS_CODE)) {
+ return false;
+ } else {
+ throw new GoraException(ex);
+ }
+ }
+ }
+
+ @Override
+ public T get(K key, String[] fields) throws GoraException {
+ String[] avFields = getFieldsToQuery(fields);
+ Object[] keyl = null;
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ keyl = new Object[]{key};
+ } else {
+ //Composite key pending
+ }
+ //Avro fields to Ignite fields
+ List<String> dbFields = new ArrayList<>();
+ for (String af : avFields) {
+ dbFields.add(igniteMapping.getFields().get(af).getName());
+ }
+ String selectQuery = IgniteSQLBuilder.selectGet(igniteMapping, dbFields);
+ try (PreparedStatement stmt = connection.prepareStatement(selectQuery)) {
+ IgniteSQLBuilder.fillSelectStatement(stmt, igniteMapping, keyl);
+ ResultSet rs = stmt.executeQuery();
+ boolean data = rs.next();
+ T resp = null;
+ if (data) {
+ resp = newInstance(rs, fields);
+ if (rs.next()) {
+ LOG.warn("Multiple results for primary key {} in the schema {}, ignoring additional rows.", keyl, igniteMapping.getTableName());
+ }
+ }
+ rs.close();
+ return resp;
+ } catch (SQLException | IOException ex) {
+ throw new GoraException(ex);
+ }
+
+ }
+
+ public T newInstance(ResultSet rs, String[] fields) throws GoraException, SQLException, IOException {
+ fields = getFieldsToQuery(fields);
+ T persistent = newPersistent();
+ for (String f : fields) {
+ Schema.Field field = fieldMap.get(f);
+ Schema fieldSchema = field.schema();
+ String dbField = igniteMapping.getFields().get(f).getName();
+ Object sv = rs.getObject(dbField);
+ if (sv == null) {
+ continue;
+ }
+ Object v = deserializeFieldValue(field, fieldSchema, sv, persistent);
+ persistent.put(field.pos(), v);
+ persistent.setDirty(field.pos());
+ }
+ return persistent;
+ }
+
+ @SuppressWarnings("unchecked")
+ private Object deserializeFieldValue(Schema.Field field, Schema fieldSchema,
+ Object igniteValue, T persistent) throws IOException {
+ Object fieldValue = null;
+ switch (fieldSchema.getType()) {
+ case MAP:
+ case ARRAY:
+ case RECORD:
+ @SuppressWarnings("rawtypes") SpecificDatumReader reader = getDatumReader(fieldSchema);
+ fieldValue = IOUtils.deserialize((byte[]) igniteValue, reader,
+ persistent.get(field.pos()));
+ break;
+ case ENUM:
+ fieldValue = AvroUtils.getEnumValue(fieldSchema, igniteValue.toString());
+ break;
+ case FIXED:
+ break;
+ case BYTES:
+ fieldValue = ByteBuffer.wrap((byte[]) igniteValue);
+ break;
+ case STRING:
+ fieldValue = new Utf8(igniteValue.toString());
+ break;
+ case UNION:
+ if (fieldSchema.getTypes().size() == 2 && isNullable(fieldSchema)) {
+ int schemaPos = getUnionSchema(igniteValue, fieldSchema);
+ Schema unionSchema = fieldSchema.getTypes().get(schemaPos);
+ fieldValue = deserializeFieldValue(field, unionSchema, igniteValue, persistent);
+ } else {
+ reader = getDatumReader(fieldSchema);
+ fieldValue = IOUtils.deserialize((byte[]) igniteValue, reader,
+ persistent.get(field.pos()));
+ }
+ break;
+ default:
+ fieldValue = igniteValue;
+ }
+ return fieldValue;
+
+ }
+
+ @Override
+ public void put(K key, T obj) throws GoraException {
+ try {
+ if (obj.isDirty()) {
+ Schema schema = obj.getSchema();
+ List<Schema.Field> fields = schema.getFields();
+ Map<Column, Object> data = new HashMap<>();
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ Column getKey = igniteMapping.getPrimaryKey().get(0);
+ data.put(getKey, key);
+ } else {
+ //Composite keys pending..
+ }
+ for (Schema.Field field : fields) {
+ Column get = igniteMapping.getFields().get(field.name());
+ Object v = obj.get(field.pos());
--- End diff --
Same goes here. Please use intuitive names for variables.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by nishadi <gi...@git.apache.org>.
Github user nishadi commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202533833
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteStore.java ---
@@ -0,0 +1,578 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.sql.rowset.CachedRowSet;
+import javax.sql.rowset.RowSetFactory;
+import javax.sql.rowset.RowSetProvider;
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificDatumReader;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.ignite.query.IgniteQuery;
+import org.apache.gora.ignite.query.IgniteResult;
+import org.apache.gora.ignite.utils.IgniteSQLBuilder;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.AvroUtils;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of a Ignite data store to be used by gora.
+ *
+ * @param <K> class to be used for the key
+ * @param <T> class to be persisted within the store
+ */
+public class IgniteStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ public static final Logger LOG = LoggerFactory.getLogger(IgniteStore.class);
+ private static final String PARSE_MAPPING_FILE_KEY = "gora.ignite.mapping.file";
+ private static final String DEFAULT_MAPPING_FILE = "gora-ignite-mapping.xml";
+ private IgniteParameters igniteParameters;
+ private IgniteMapping igniteMapping;
+ private Connection connection;
+
+ /*
+ * Create a threadlocal map for the datum readers and writers, because they
+ * are not thread safe, at least not before Avro 1.4.0 (See AVRO-650). When
+ * they are thread safe, it is possible to maintain a single reader and writer
+ * pair for every schema, instead of one for every thread.
+ */
+ public static final ConcurrentHashMap<Schema, SpecificDatumReader<?>> readerMap = new ConcurrentHashMap<>();
+
+ public static final ConcurrentHashMap<Schema, SpecificDatumWriter<?>> writerMap = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) throws GoraException {
+
+ try {
+ super.initialize(keyClass, persistentClass, properties);
+ IgniteMappingBuilder builder = new IgniteMappingBuilder(this);
+ builder.readMappingFile(getConf().get(PARSE_MAPPING_FILE_KEY, DEFAULT_MAPPING_FILE));
+ igniteMapping = builder.getIgniteMapping();
+ igniteParameters = IgniteParameters.load(properties, conf);
+ connection = acquiereConnection();
+ LOG.info("Ignite store was successfully initialized");
+ } catch (ClassNotFoundException | SQLException ex) {
+ LOG.error("Error while initializing Ignite store", ex);
+ throw new GoraException(ex);
+ }
+ }
+
+ private Connection acquiereConnection() throws ClassNotFoundException, SQLException {
+ Class.forName("org.apache.ignite.IgniteJdbcThinDriver");
+ StringBuilder urlBuilder = new StringBuilder();
+ urlBuilder.append("jdbc:ignite:thin://");
+ urlBuilder.append(igniteParameters.getHost());
+ if (igniteParameters.getPort() != null) {
+ urlBuilder.append(":" + igniteParameters.getPort());
+ }
+ if (igniteParameters.getSchema() != null) {
+ urlBuilder.append("/" + igniteParameters.getSchema());
+ }
+ if (igniteParameters.getUser() != null) {
+ urlBuilder.append(";" + igniteParameters.getUser());
+ }
+ if (igniteParameters.getPassword() != null) {
+ urlBuilder.append(";" + igniteParameters.getPassword());
+ }
+ if (igniteParameters.getAdditionalConfigurations() != null) {
+ urlBuilder.append(igniteParameters.getAdditionalConfigurations());
+ }
+ Connection conn = DriverManager.getConnection(urlBuilder.toString());
+ return conn;
+ }
+
+ @Override
+ public String getSchemaName() {
+ return igniteMapping.getTableName();
+ }
+
+ @Override
+ public String getSchemaName(final String mappingSchemaName,
+ final Class<?> persistentClass) {
+ return super.getSchemaName(mappingSchemaName, persistentClass);
+ }
+
+ @Override
+ public void createSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to create the schema as no connection has been initiated.");
+ }
+ if (schemaExists()) {
+ return;
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String createTableSQL = IgniteSQLBuilder.createTable(igniteMapping);
+ stmt.executeUpdate(createTableSQL);
+ LOG.info("Table {} has been created for Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public void deleteSchema() throws GoraException {
--- End diff --
Don't we need to have a check to see if the schema exists or not?
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by nishadi <gi...@git.apache.org>.
Github user nishadi commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202533866
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteStore.java ---
@@ -0,0 +1,578 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.sql.rowset.CachedRowSet;
+import javax.sql.rowset.RowSetFactory;
+import javax.sql.rowset.RowSetProvider;
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificDatumReader;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.ignite.query.IgniteQuery;
+import org.apache.gora.ignite.query.IgniteResult;
+import org.apache.gora.ignite.utils.IgniteSQLBuilder;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.AvroUtils;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of a Ignite data store to be used by gora.
+ *
+ * @param <K> class to be used for the key
+ * @param <T> class to be persisted within the store
+ */
+public class IgniteStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ public static final Logger LOG = LoggerFactory.getLogger(IgniteStore.class);
+ private static final String PARSE_MAPPING_FILE_KEY = "gora.ignite.mapping.file";
+ private static final String DEFAULT_MAPPING_FILE = "gora-ignite-mapping.xml";
+ private IgniteParameters igniteParameters;
+ private IgniteMapping igniteMapping;
+ private Connection connection;
+
+ /*
+ * Create a threadlocal map for the datum readers and writers, because they
+ * are not thread safe, at least not before Avro 1.4.0 (See AVRO-650). When
+ * they are thread safe, it is possible to maintain a single reader and writer
+ * pair for every schema, instead of one for every thread.
+ */
+ public static final ConcurrentHashMap<Schema, SpecificDatumReader<?>> readerMap = new ConcurrentHashMap<>();
+
+ public static final ConcurrentHashMap<Schema, SpecificDatumWriter<?>> writerMap = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) throws GoraException {
+
+ try {
+ super.initialize(keyClass, persistentClass, properties);
+ IgniteMappingBuilder builder = new IgniteMappingBuilder(this);
+ builder.readMappingFile(getConf().get(PARSE_MAPPING_FILE_KEY, DEFAULT_MAPPING_FILE));
+ igniteMapping = builder.getIgniteMapping();
+ igniteParameters = IgniteParameters.load(properties, conf);
+ connection = acquiereConnection();
+ LOG.info("Ignite store was successfully initialized");
+ } catch (ClassNotFoundException | SQLException ex) {
+ LOG.error("Error while initializing Ignite store", ex);
+ throw new GoraException(ex);
+ }
+ }
+
+ private Connection acquiereConnection() throws ClassNotFoundException, SQLException {
+ Class.forName("org.apache.ignite.IgniteJdbcThinDriver");
+ StringBuilder urlBuilder = new StringBuilder();
+ urlBuilder.append("jdbc:ignite:thin://");
+ urlBuilder.append(igniteParameters.getHost());
+ if (igniteParameters.getPort() != null) {
+ urlBuilder.append(":" + igniteParameters.getPort());
+ }
+ if (igniteParameters.getSchema() != null) {
+ urlBuilder.append("/" + igniteParameters.getSchema());
+ }
+ if (igniteParameters.getUser() != null) {
+ urlBuilder.append(";" + igniteParameters.getUser());
+ }
+ if (igniteParameters.getPassword() != null) {
+ urlBuilder.append(";" + igniteParameters.getPassword());
+ }
+ if (igniteParameters.getAdditionalConfigurations() != null) {
+ urlBuilder.append(igniteParameters.getAdditionalConfigurations());
+ }
+ Connection conn = DriverManager.getConnection(urlBuilder.toString());
+ return conn;
+ }
+
+ @Override
+ public String getSchemaName() {
+ return igniteMapping.getTableName();
+ }
+
+ @Override
+ public String getSchemaName(final String mappingSchemaName,
+ final Class<?> persistentClass) {
+ return super.getSchemaName(mappingSchemaName, persistentClass);
+ }
+
+ @Override
+ public void createSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to create the schema as no connection has been initiated.");
+ }
+ if (schemaExists()) {
+ return;
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String createTableSQL = IgniteSQLBuilder.createTable(igniteMapping);
+ stmt.executeUpdate(createTableSQL);
+ LOG.info("Table {} has been created for Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public void deleteSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to delete the schema as no connection has been initiated.");
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String dropTableSQL = IgniteSQLBuilder.dropTable(igniteMapping.getTableName());
+ stmt.executeUpdate(dropTableSQL);
+ LOG.info("Table {} has been dropped from Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public boolean schemaExists() throws GoraException {
+ boolean exists = false;
--- End diff --
Rather than having a separate variable, we can directly return true or false below.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202523561
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteStore.java ---
@@ -0,0 +1,578 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.sql.rowset.CachedRowSet;
+import javax.sql.rowset.RowSetFactory;
+import javax.sql.rowset.RowSetProvider;
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificDatumReader;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.ignite.query.IgniteQuery;
+import org.apache.gora.ignite.query.IgniteResult;
+import org.apache.gora.ignite.utils.IgniteSQLBuilder;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.AvroUtils;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of a Ignite data store to be used by gora.
+ *
+ * @param <K> class to be used for the key
+ * @param <T> class to be persisted within the store
+ */
+public class IgniteStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ public static final Logger LOG = LoggerFactory.getLogger(IgniteStore.class);
+ private static final String PARSE_MAPPING_FILE_KEY = "gora.ignite.mapping.file";
+ private static final String DEFAULT_MAPPING_FILE = "gora-ignite-mapping.xml";
+ private IgniteParameters igniteParameters;
+ private IgniteMapping igniteMapping;
+ private Connection connection;
+
+ /*
+ * Create a threadlocal map for the datum readers and writers, because they
+ * are not thread safe, at least not before Avro 1.4.0 (See AVRO-650). When
--- End diff --
We use a much more recent version of AVRO. Is it necessary to include this comment?
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by nishadi <gi...@git.apache.org>.
Github user nishadi commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r203057892
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteStore.java ---
@@ -0,0 +1,578 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.sql.rowset.CachedRowSet;
+import javax.sql.rowset.RowSetFactory;
+import javax.sql.rowset.RowSetProvider;
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificDatumReader;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.ignite.query.IgniteQuery;
+import org.apache.gora.ignite.query.IgniteResult;
+import org.apache.gora.ignite.utils.IgniteSQLBuilder;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.AvroUtils;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of a Ignite data store to be used by gora.
+ *
+ * @param <K> class to be used for the key
+ * @param <T> class to be persisted within the store
+ */
+public class IgniteStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ public static final Logger LOG = LoggerFactory.getLogger(IgniteStore.class);
+ private static final String PARSE_MAPPING_FILE_KEY = "gora.ignite.mapping.file";
+ private static final String DEFAULT_MAPPING_FILE = "gora-ignite-mapping.xml";
+ private IgniteParameters igniteParameters;
+ private IgniteMapping igniteMapping;
+ private Connection connection;
+
+ /*
+ * Create a threadlocal map for the datum readers and writers, because they
+ * are not thread safe, at least not before Avro 1.4.0 (See AVRO-650). When
+ * they are thread safe, it is possible to maintain a single reader and writer
+ * pair for every schema, instead of one for every thread.
+ */
+ public static final ConcurrentHashMap<Schema, SpecificDatumReader<?>> readerMap = new ConcurrentHashMap<>();
+
+ public static final ConcurrentHashMap<Schema, SpecificDatumWriter<?>> writerMap = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) throws GoraException {
+
+ try {
+ super.initialize(keyClass, persistentClass, properties);
+ IgniteMappingBuilder builder = new IgniteMappingBuilder(this);
+ builder.readMappingFile(getConf().get(PARSE_MAPPING_FILE_KEY, DEFAULT_MAPPING_FILE));
+ igniteMapping = builder.getIgniteMapping();
+ igniteParameters = IgniteParameters.load(properties, conf);
+ connection = acquiereConnection();
+ LOG.info("Ignite store was successfully initialized");
+ } catch (ClassNotFoundException | SQLException ex) {
+ LOG.error("Error while initializing Ignite store", ex);
+ throw new GoraException(ex);
+ }
+ }
+
+ private Connection acquiereConnection() throws ClassNotFoundException, SQLException {
+ Class.forName("org.apache.ignite.IgniteJdbcThinDriver");
+ StringBuilder urlBuilder = new StringBuilder();
+ urlBuilder.append("jdbc:ignite:thin://");
+ urlBuilder.append(igniteParameters.getHost());
+ if (igniteParameters.getPort() != null) {
+ urlBuilder.append(":" + igniteParameters.getPort());
+ }
+ if (igniteParameters.getSchema() != null) {
+ urlBuilder.append("/" + igniteParameters.getSchema());
+ }
+ if (igniteParameters.getUser() != null) {
+ urlBuilder.append(";" + igniteParameters.getUser());
+ }
+ if (igniteParameters.getPassword() != null) {
+ urlBuilder.append(";" + igniteParameters.getPassword());
+ }
+ if (igniteParameters.getAdditionalConfigurations() != null) {
+ urlBuilder.append(igniteParameters.getAdditionalConfigurations());
+ }
+ Connection conn = DriverManager.getConnection(urlBuilder.toString());
+ return conn;
+ }
+
+ @Override
+ public String getSchemaName() {
+ return igniteMapping.getTableName();
+ }
+
+ @Override
+ public String getSchemaName(final String mappingSchemaName,
+ final Class<?> persistentClass) {
+ return super.getSchemaName(mappingSchemaName, persistentClass);
+ }
+
+ @Override
+ public void createSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to create the schema as no connection has been initiated.");
+ }
+ if (schemaExists()) {
+ return;
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String createTableSQL = IgniteSQLBuilder.createTable(igniteMapping);
+ stmt.executeUpdate(createTableSQL);
+ LOG.info("Table {} has been created for Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public void deleteSchema() throws GoraException {
--- End diff --
Good point Alfonso :)
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by nishadi <gi...@git.apache.org>.
Github user nishadi commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202521399
--- Diff: gora-ignite/pom.xml ---
@@ -0,0 +1,182 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora</artifactId>
+ <version>0.9-SNAPSHOT</version>
+ <relativePath>../</relativePath>
+ </parent>
+ <artifactId>gora-ignite</artifactId>
+ <packaging>bundle</packaging>
+
+ <name>Apache Gora :: Ignite</name>
+ <url>http://gora.apache.org</url>
+ <description>The Apache Gora open source framework provides an in-memory data model and
+ persistence for big data. Gora supports persisting to column stores, key value stores,
+ document stores and RDBMSs, and analyzing the data with extensive Apache Hadoop MapReduce
+ support.</description>
+ <inceptionYear>2010</inceptionYear>
+ <organization>
+ <name>The Apache Software Foundation</name>
+ <url>http://www.apache.org/</url>
+ </organization>
+ <issueManagement>
+ <system>JIRA</system>
+ <url>https://issues.apache.org/jira/browse/GORA</url>
+ </issueManagement>
+ <ciManagement>
+ <system>Jenkins</system>
+ <url>https://builds.apache.org/job/Gora-trunk/</url>
+ </ciManagement>
+
+ <properties>
+ <ignite.version>2.4.0</ignite.version>
+ <osgi.import>*</osgi.import>
+ <osgi.export>org.apache.gora.ignite*;version="${project.version}";-noimport:=true</osgi.export>
+ </properties>
+
+ <build>
+ <directory>target</directory>
+ <outputDirectory>target/classes</outputDirectory>
+ <finalName>${project.artifactId}-${project.version}</finalName>
+ <testOutputDirectory>target/test-classes</testOutputDirectory>
+ <testSourceDirectory>src/test/java</testSourceDirectory>
+ <sourceDirectory>src/main/java</sourceDirectory>
+ <testResources>
+ <testResource>
+ <directory>${project.basedir}/src/test/resources</directory>
+ <includes>
+ <include>**/*</include>
+ </includes>
+ <!--targetPath>${project.basedir}/target/classes/</targetPath -->
+ </testResource>
+ </testResources>
+ <plugins>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>build-helper-maven-plugin</artifactId>
+ <version>${build-helper-maven-plugin.version}</version>
+ <executions>
+ <execution>
+ <phase>generate-sources</phase>
+ <goals>
+ <goal>add-source</goal>
+ </goals>
+ <configuration>
+ <sources>
+ <source>src/examples/java</source>
+ </sources>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+
+ <dependencies>
+ <!-- Gora Internal Dependencies -->
+ <dependency>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora-core</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora-core</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+
+ <!--Ignite Dependency -->
+ <dependency>
+ <groupId>org.apache.ignite</groupId>
+ <artifactId>ignite-core</artifactId>
+ <version>${ignite.version}</version>
+ </dependency>
+
--- End diff --
Shall we move these dependencies to the dependency management section of the parent POM and refer them here?
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211553433
--- Diff: gora-ignite/src/test/java/org/apache/gora/ignite/store/TestIgniteStore.java ---
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import org.apache.gora.ignite.GoraIgniteTestDriver;
+import org.apache.gora.store.DataStoreTestBase;
+
+/**
+ * Test case for IgniteStore.
+ */
+public class TestIgniteStore extends DataStoreTestBase {
+
+ static {
+ setTestDriver(new GoraIgniteTestDriver());
+ }
+
+}
--- End diff --
Please add package-info.java files to test classes packages as well.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202523485
--- Diff: gora-ignite/pom.xml ---
@@ -0,0 +1,182 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora</artifactId>
+ <version>0.9-SNAPSHOT</version>
+ <relativePath>../</relativePath>
+ </parent>
+ <artifactId>gora-ignite</artifactId>
+ <packaging>bundle</packaging>
+
+ <name>Apache Gora :: Ignite</name>
+ <url>http://gora.apache.org</url>
+ <description>The Apache Gora open source framework provides an in-memory data model and
+ persistence for big data. Gora supports persisting to column stores, key value stores,
+ document stores and RDBMSs, and analyzing the data with extensive Apache Hadoop MapReduce
+ support.</description>
+ <inceptionYear>2010</inceptionYear>
+ <organization>
+ <name>The Apache Software Foundation</name>
+ <url>http://www.apache.org/</url>
+ </organization>
+ <issueManagement>
+ <system>JIRA</system>
+ <url>https://issues.apache.org/jira/browse/GORA</url>
+ </issueManagement>
+ <ciManagement>
+ <system>Jenkins</system>
+ <url>https://builds.apache.org/job/Gora-trunk/</url>
+ </ciManagement>
+
+ <properties>
+ <ignite.version>2.4.0</ignite.version>
--- End diff --
Please upgrade to 2.5.0
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202526769
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteMappingBuilder.java ---
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import com.google.inject.ConfigurationException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.jdom.Document;
+import org.jdom.Element;
+import org.jdom.JDOMException;
+import org.jdom.input.SAXBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Builder for Mapping definitions of Ignite.
+ */
+public class IgniteMappingBuilder<K, T extends PersistentBase> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IgniteMappingBuilder.class);
+ /**
+ * Mapping instance being built
+ */
+ private IgniteMapping igniteMapping;
+
+ private final IgniteStore<K, T> dataStore;
+
+ public IgniteMappingBuilder(final IgniteStore<K, T> store) {
+ this.igniteMapping = new IgniteMapping();
+ this.dataStore = store;
+ }
+
+ public IgniteMapping getIgniteMapping() {
+ return igniteMapping;
+ }
+
+ public void setIgniteMapping(IgniteMapping igniteMapping) {
+ this.igniteMapping = igniteMapping;
+ }
+
+ public void readMappingFile(String mappingFile) {
+ try {
+ SAXBuilder saxBuilder = new SAXBuilder();
+ InputStream inputStream = getClass().getClassLoader().getResourceAsStream(mappingFile);
+ if (inputStream == null) {
+ LOG.error("Mapping file '{}' could not be found!", mappingFile);
+ throw new IOException("Mapping file '" + mappingFile + "' could not be found!");
+ }
+ Document document = saxBuilder.build(inputStream);
+ if (document == null) {
+ LOG.error("Mapping file '{}' could not be found!", mappingFile);
+ throw new IOException("Mapping file '" + mappingFile + "' could not be found!");
+ }
+ List<Element> classes = document.getRootElement().getChildren("class");
+ for (Element classElement : classes) {
+ if (classElement.getAttributeValue("keyClass").equals(
+ dataStore.getKeyClass().getCanonicalName())
+ && classElement.getAttributeValue("name").equals(
+ dataStore.getPersistentClass().getCanonicalName())) {
+ final String tableNameFromMapping = classElement.getAttributeValue("table");
+ String tableName = dataStore.getSchemaName(tableNameFromMapping, dataStore.getPersistentClass());
+ igniteMapping.setTableName(tableName);
+ List<Element> prColumns = classElement.getChildren("primarykey");
--- End diff --
Type safety: The expression of type List needs unchecked conversion to conform to List < Element >
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211214861
--- Diff: gora-ignite/pom.xml ---
@@ -0,0 +1,172 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora</artifactId>
+ <version>0.9-SNAPSHOT</version>
+ <relativePath>../</relativePath>
+ </parent>
+ <artifactId>gora-ignite</artifactId>
+ <packaging>bundle</packaging>
+
+ <name>Apache Gora :: Ignite</name>
+ <url>http://gora.apache.org</url>
+ <description>The Apache Gora open source framework provides an in-memory data model and
+ persistence for big data. Gora supports persisting to column stores, key value stores,
+ document stores and RDBMSs, and analyzing the data with extensive Apache Hadoop MapReduce
+ support.</description>
+ <inceptionYear>2010</inceptionYear>
+ <organization>
+ <name>The Apache Software Foundation</name>
+ <url>http://www.apache.org/</url>
+ </organization>
+ <issueManagement>
+ <system>JIRA</system>
+ <url>https://issues.apache.org/jira/browse/GORA</url>
+ </issueManagement>
+ <ciManagement>
+ <system>Jenkins</system>
+ <url>https://builds.apache.org/job/Gora-trunk/</url>
+ </ciManagement>
+
+ <properties>
+ <osgi.import>*</osgi.import>
+ <osgi.export>org.apache.gora.ignite*;version="${project.version}";-noimport:=true</osgi.export>
+ </properties>
+
+ <build>
+ <directory>target</directory>
+ <outputDirectory>target/classes</outputDirectory>
+ <finalName>${project.artifactId}-${project.version}</finalName>
+ <testOutputDirectory>target/test-classes</testOutputDirectory>
+ <testSourceDirectory>src/test/java</testSourceDirectory>
+ <sourceDirectory>src/main/java</sourceDirectory>
+ <testResources>
+ <testResource>
+ <directory>${project.basedir}/src/test/resources</directory>
+ <includes>
+ <include>**/*</include>
+ </includes>
+ <!--targetPath>${project.basedir}/target/classes/</targetPath -->
+ </testResource>
+ </testResources>
+ <plugins>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>build-helper-maven-plugin</artifactId>
+ <version>${build-helper-maven-plugin.version}</version>
+ <executions>
+ <execution>
+ <phase>generate-sources</phase>
+ <goals>
+ <goal>add-source</goal>
+ </goals>
+ <configuration>
+ <sources>
+ <source>src/examples/java</source>
+ </sources>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+
+ <dependencies>
+ <!-- Gora Internal Dependencies -->
+ <dependency>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora-core</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.gora</groupId>
+ <artifactId>gora-core</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+
+ <!--Ignite Dependency -->
+ <dependency>
+ <groupId>org.apache.ignite</groupId>
+ <artifactId>ignite-core</artifactId>
+ <version>${ignite.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.zookeeper</groupId>
+ <artifactId>zookeeper</artifactId>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.avro</groupId>
+ <artifactId>avro</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.jdom</groupId>
+ <artifactId>jdom</artifactId>
+ <scope>compile</scope>
+ </dependency>
+
+ <!-- Logging Dependencies -->
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-log4j12</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>log4j</groupId>
+ <artifactId>log4j</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>javax.jms</groupId>
+ <artifactId>jms</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <!-- Testing Dependencies -->
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-minicluster</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.ignite</groupId>
+ <artifactId>ignite-indexing</artifactId>
+ <version>${ignite.version}</version>
+ </dependency>
--- End diff --
Please move this dependency to Parent POM, then you just need to refer without
```<version>${ignite.version}</version>``` tag.
Please check other dependencies here, which have been referred from parent pom.
---
[GitHub] gora issue #134: GORA-535 Add a data store for Apache Ignite
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on the issue:
https://github.com/apache/gora/pull/134
@djkevincr @carlosrmng is this ready to merge?
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211238621
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteParameters.java ---
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.util.Properties;
+import org.apache.gora.ignite.utils.IgniteBackendConstants;
+import org.apache.hadoop.conf.Configuration;
+
+/**
+ * Parameters definitions for Ignite.
+ */
+public class IgniteParameters {
+
+ /**
+ * Property indicating the Ignite Schema to be used
+ */
+ public static final String PROP_SCHEMA = "gora.datastore.ignite.schema";
+
+ /**
+ * Property indicating the Ignite Cluster Node IP
+ */
+ public static final String PROP_HOST = "gora.datastore.ignite.host";
+
+ /**
+ * Property indicating the port used by the Ignite Server
+ */
+ public static final String PROP_PORT = "gora.datastore.ignite.port";
+
+ /**
+ * Property indicating the username to connect to the server
+ */
+ public static final String PROP_USER = "gora.datastore.ignite.user";
+
+ /**
+ * Property indicating the password to connect to the server
+ */
+ public static final String PROP_PASSWORD = "gora.datastore.ignite.password";
+
+ /**
+ * Property indicating additional JDBC options
+ */
+ public static final String PROP_ADDITIONALS = "gora.datastore.ignite.additionalConfigurations";
+
+ private String host;
+ private String port;
+ private String schema;
+ private String user;
+ private String password;
+ private String additionalConfigurations;
+
+ /**
+ * @param host Hostname/IP of the Ignite Server
+ * @param port Optional port for Ignite Server
+ * @param user Optional username for Ignite
+ * @param password Optional password for Ignite
+ * @param additionalConfigurations Optional additional configurations for
+ * Ignite
+ */
+ private IgniteParameters(String host, String port, String schema, String user, String password, String additionalConfigurations) {
+ this.host = host;
+ this.port = port;
+ this.schema = schema;
+ this.user = user;
+ this.password = password;
+ this.additionalConfigurations = additionalConfigurations;
+ }
+
+ public String getHost() {
+ return host;
+ }
+
+ public void setHost(String host) {
+ this.host = host;
+ }
+
+ public String getPort() {
+ return port;
+ }
+
+ public void setPort(String port) {
--- End diff --
Please document these public methods, similarly for other cases.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211519945
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteStore.java ---
@@ -0,0 +1,565 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.sql.rowset.CachedRowSet;
+import javax.sql.rowset.RowSetFactory;
+import javax.sql.rowset.RowSetProvider;
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificDatumReader;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.ignite.query.IgniteQuery;
+import org.apache.gora.ignite.query.IgniteResult;
+import org.apache.gora.ignite.utils.IgniteBackendConstants;
+import org.apache.gora.ignite.utils.IgniteSQLBuilder;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.AvroUtils;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of a Ignite data store to be used by gora.
+ *
+ * @param <K> class to be used for the key
+ * @param <T> class to be persisted within the store
+ */
+public class IgniteStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IgniteStore.class);
+ private static final String PARSE_MAPPING_FILE_KEY = "gora.ignite.mapping.file";
+ private static final String DEFAULT_MAPPING_FILE = "gora-ignite-mapping.xml";
+ private IgniteParameters igniteParameters;
+ private IgniteMapping igniteMapping;
+ private Connection connection;
+ private static final ConcurrentHashMap<Schema, SpecificDatumReader<?>> readerMap = new ConcurrentHashMap<>();
+ private static final ConcurrentHashMap<Schema, SpecificDatumWriter<?>> writerMap = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) throws GoraException {
+ try {
+ super.initialize(keyClass, persistentClass, properties);
+ IgniteMappingBuilder<K, T> builder = new IgniteMappingBuilder<K, T>(this);
+ builder.readMappingFile(getConf().get(PARSE_MAPPING_FILE_KEY, DEFAULT_MAPPING_FILE));
+ igniteMapping = builder.getIgniteMapping();
+ igniteParameters = IgniteParameters.load(properties);
+ connection = acquireConnection();
+ LOG.info("Ignite store was successfully initialized");
+ } catch (ClassNotFoundException | SQLException ex) {
+ LOG.error("Error while initializing Ignite store", ex);
+ throw new GoraException(ex);
+ }
+ }
+
+ private Connection acquireConnection() throws ClassNotFoundException, SQLException {
+ Class.forName("org.apache.ignite.IgniteJdbcThinDriver");
+ StringBuilder urlBuilder = new StringBuilder();
+ urlBuilder.append("jdbc:ignite:thin://");
+ urlBuilder.append(igniteParameters.getHost());
+ if (igniteParameters.getPort() != null) {
+ urlBuilder.append(":" + igniteParameters.getPort());
+ }
+ if (igniteParameters.getSchema() != null) {
+ urlBuilder.append("/" + igniteParameters.getSchema());
+ }
+ if (igniteParameters.getUser() != null) {
+ urlBuilder.append(";" + igniteParameters.getUser());
+ }
+ if (igniteParameters.getPassword() != null) {
+ urlBuilder.append(";" + igniteParameters.getPassword());
+ }
+ if (igniteParameters.getAdditionalConfigurations() != null) {
+ urlBuilder.append(igniteParameters.getAdditionalConfigurations());
+ }
+ return DriverManager.getConnection(urlBuilder.toString());
+ }
+
+ @Override
+ public String getSchemaName() {
+ return igniteMapping.getTableName();
+ }
+
+ @Override
+ public String getSchemaName(final String mappingSchemaName,
+ final Class<?> persistentClass) {
+ return super.getSchemaName(mappingSchemaName, persistentClass);
+ }
+
+ @Override
+ public void createSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to create the schema as no connection has been initiated.");
+ }
+ if (schemaExists()) {
+ return;
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String createTableSQL = IgniteSQLBuilder.createTable(igniteMapping);
+ stmt.executeUpdate(createTableSQL);
+ LOG.info("Table {} has been created for Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public void deleteSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to delete the schema as no connection has been initiated.");
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String dropTableSQL = IgniteSQLBuilder.dropTable(igniteMapping.getTableName());
+ stmt.executeUpdate(dropTableSQL);
+ LOG.info("Table {} has been dropped from Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public boolean schemaExists() throws GoraException {
+ try (Statement stmt = connection.createStatement()) {
+ String tableExistsSQL = IgniteSQLBuilder.tableExists(igniteMapping.getTableName());
+ ResultSet executeQuery = stmt.executeQuery(tableExistsSQL);
+ executeQuery.close();
+ return true;
+ } catch (SQLException ex) {
+ if (ex.getSQLState() != null
+ && ex.getSQLState().equals(IgniteBackendConstants.DEFAULT_IGNITE_TABLE_NOT_EXISTS_CODE)) {
+ return false;
+ } else {
+ throw new GoraException(ex);
+ }
+ }
+ }
+
+ @Override
+ public T get(K key, String[] fields) throws GoraException {
+ String[] avFields = getFieldsToQuery(fields);
+ Object[] keyl = null;
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ keyl = new Object[]{key};
+ } else {
+ //Composite key pending
+ }
+ //Avro fields to Ignite fields
+ List<String> dbFields = new ArrayList<>();
+ for (String af : avFields) {
+ dbFields.add(igniteMapping.getFields().get(af).getName());
+ }
+ String selectQuery = IgniteSQLBuilder.selectGet(igniteMapping, dbFields);
+ try (PreparedStatement stmt = connection.prepareStatement(selectQuery)) {
+ IgniteSQLBuilder.fillSelectStatement(stmt, igniteMapping, keyl);
+ ResultSet rs = stmt.executeQuery();
+ boolean data = rs.next();
+ T resp = null;
+ if (data) {
+ resp = newInstance(rs, fields);
+ if (rs.next()) {
+ LOG.warn("Multiple results for primary key {} in the schema {}, ignoring additional rows.", keyl, igniteMapping.getTableName());
+ }
+ }
+ rs.close();
+ return resp;
+ } catch (SQLException | IOException ex) {
+ throw new GoraException(ex);
+ }
+
+ }
+
+ public T newInstance(ResultSet rs, String[] fields) throws GoraException, SQLException, IOException {
+ fields = getFieldsToQuery(fields);
+ T persistent = newPersistent();
+ for (String f : fields) {
+ Schema.Field field = fieldMap.get(f);
+ Schema fieldSchema = field.schema();
+ String dbField = igniteMapping.getFields().get(f).getName();
+ Object sv = rs.getObject(dbField);
+ if (sv == null) {
+ continue;
+ }
+ Object v = deserializeFieldValue(field, fieldSchema, sv, persistent);
+ persistent.put(field.pos(), v);
+ persistent.setDirty(field.pos());
+ }
+ return persistent;
+ }
+
+ @SuppressWarnings("unchecked")
+ private Object deserializeFieldValue(Schema.Field field, Schema fieldSchema,
+ Object igniteValue, T persistent) throws IOException {
+ Object fieldValue = null;
+ switch (fieldSchema.getType()) {
+ case MAP:
+ case ARRAY:
+ case RECORD:
+ @SuppressWarnings("rawtypes") SpecificDatumReader reader = getDatumReader(fieldSchema);
+ fieldValue = IOUtils.deserialize((byte[]) igniteValue, reader,
+ persistent.get(field.pos()));
+ break;
+ case ENUM:
+ fieldValue = AvroUtils.getEnumValue(fieldSchema, igniteValue.toString());
+ break;
+ case FIXED:
+ break;
+ case BYTES:
+ fieldValue = ByteBuffer.wrap((byte[]) igniteValue);
+ break;
+ case STRING:
+ fieldValue = new Utf8(igniteValue.toString());
+ break;
+ case UNION:
+ if (fieldSchema.getTypes().size() == 2 && isNullable(fieldSchema)) {
+ int schemaPos = getUnionSchema(igniteValue, fieldSchema);
+ Schema unionSchema = fieldSchema.getTypes().get(schemaPos);
+ fieldValue = deserializeFieldValue(field, unionSchema, igniteValue, persistent);
+ } else {
+ reader = getDatumReader(fieldSchema);
+ fieldValue = IOUtils.deserialize((byte[]) igniteValue, reader,
+ persistent.get(field.pos()));
+ }
+ break;
+ default:
+ fieldValue = igniteValue;
+ }
+ return fieldValue;
+
+ }
+
+ @Override
+ public void put(K key, T obj) throws GoraException {
+ try {
+ if (obj.isDirty()) {
+ Schema schema = obj.getSchema();
+ List<Schema.Field> fields = schema.getFields();
+ Map<Column, Object> data = new HashMap<>();
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ Column getKey = igniteMapping.getPrimaryKey().get(0);
+ data.put(getKey, key);
+ } else {
+ //Composite keys pending..
--- End diff --
You may create JIRA ticket for the composite key feature as it is not supported at the moment.
You may refer that JIRA ticket ID from comment it self.
---
[GitHub] gora issue #134: GORA-535 Add a data store for Apache Ignite
Posted by carlosrmng <gi...@git.apache.org>.
Github user carlosrmng commented on the issue:
https://github.com/apache/gora/pull/134
Hi @lewismc @nishadi @alfonsonishikawa
Thanks for your feedback I will start working on the issues.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211216655
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/package-info.java ---
@@ -0,0 +1,20 @@
+/**
--- End diff --
Yes Please. @carlosrmng
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211221573
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteMapping.java ---
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Mapping definitions for Ignite.
+ */
+public class IgniteMapping {
+
+ private String tableName;
+ private Map<String, Column> fields;
+ private List<Column> primaryKey;
+
+ public IgniteMapping() {
+ fields = new HashMap<>();
+ }
+
+ public String getTableName() {
--- End diff --
Please document these public methods.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211546620
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/query/package-info.java ---
@@ -0,0 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+/**
+ * This package contains Ignite datastore related all classes.
+ */
+package org.apache.gora.ignite.query;
--- End diff --
You can omit this by ending with new lines for these java class files.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211529975
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteStore.java ---
@@ -0,0 +1,565 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.sql.rowset.CachedRowSet;
+import javax.sql.rowset.RowSetFactory;
+import javax.sql.rowset.RowSetProvider;
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificDatumReader;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.ignite.query.IgniteQuery;
+import org.apache.gora.ignite.query.IgniteResult;
+import org.apache.gora.ignite.utils.IgniteBackendConstants;
+import org.apache.gora.ignite.utils.IgniteSQLBuilder;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.AvroUtils;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of a Ignite data store to be used by gora.
+ *
+ * @param <K> class to be used for the key
+ * @param <T> class to be persisted within the store
+ */
+public class IgniteStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IgniteStore.class);
+ private static final String PARSE_MAPPING_FILE_KEY = "gora.ignite.mapping.file";
+ private static final String DEFAULT_MAPPING_FILE = "gora-ignite-mapping.xml";
+ private IgniteParameters igniteParameters;
+ private IgniteMapping igniteMapping;
+ private Connection connection;
+ private static final ConcurrentHashMap<Schema, SpecificDatumReader<?>> readerMap = new ConcurrentHashMap<>();
+ private static final ConcurrentHashMap<Schema, SpecificDatumWriter<?>> writerMap = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) throws GoraException {
+ try {
+ super.initialize(keyClass, persistentClass, properties);
+ IgniteMappingBuilder<K, T> builder = new IgniteMappingBuilder<K, T>(this);
+ builder.readMappingFile(getConf().get(PARSE_MAPPING_FILE_KEY, DEFAULT_MAPPING_FILE));
+ igniteMapping = builder.getIgniteMapping();
+ igniteParameters = IgniteParameters.load(properties);
+ connection = acquireConnection();
+ LOG.info("Ignite store was successfully initialized");
+ } catch (ClassNotFoundException | SQLException ex) {
+ LOG.error("Error while initializing Ignite store", ex);
+ throw new GoraException(ex);
+ }
+ }
+
+ private Connection acquireConnection() throws ClassNotFoundException, SQLException {
+ Class.forName("org.apache.ignite.IgniteJdbcThinDriver");
+ StringBuilder urlBuilder = new StringBuilder();
+ urlBuilder.append("jdbc:ignite:thin://");
+ urlBuilder.append(igniteParameters.getHost());
+ if (igniteParameters.getPort() != null) {
+ urlBuilder.append(":" + igniteParameters.getPort());
+ }
+ if (igniteParameters.getSchema() != null) {
+ urlBuilder.append("/" + igniteParameters.getSchema());
+ }
+ if (igniteParameters.getUser() != null) {
+ urlBuilder.append(";" + igniteParameters.getUser());
+ }
+ if (igniteParameters.getPassword() != null) {
+ urlBuilder.append(";" + igniteParameters.getPassword());
+ }
+ if (igniteParameters.getAdditionalConfigurations() != null) {
+ urlBuilder.append(igniteParameters.getAdditionalConfigurations());
+ }
+ return DriverManager.getConnection(urlBuilder.toString());
+ }
+
+ @Override
+ public String getSchemaName() {
+ return igniteMapping.getTableName();
+ }
+
+ @Override
+ public String getSchemaName(final String mappingSchemaName,
+ final Class<?> persistentClass) {
+ return super.getSchemaName(mappingSchemaName, persistentClass);
+ }
+
+ @Override
+ public void createSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to create the schema as no connection has been initiated.");
+ }
+ if (schemaExists()) {
+ return;
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String createTableSQL = IgniteSQLBuilder.createTable(igniteMapping);
+ stmt.executeUpdate(createTableSQL);
+ LOG.info("Table {} has been created for Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public void deleteSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to delete the schema as no connection has been initiated.");
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String dropTableSQL = IgniteSQLBuilder.dropTable(igniteMapping.getTableName());
+ stmt.executeUpdate(dropTableSQL);
+ LOG.info("Table {} has been dropped from Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public boolean schemaExists() throws GoraException {
+ try (Statement stmt = connection.createStatement()) {
+ String tableExistsSQL = IgniteSQLBuilder.tableExists(igniteMapping.getTableName());
+ ResultSet executeQuery = stmt.executeQuery(tableExistsSQL);
+ executeQuery.close();
+ return true;
+ } catch (SQLException ex) {
+ if (ex.getSQLState() != null
+ && ex.getSQLState().equals(IgniteBackendConstants.DEFAULT_IGNITE_TABLE_NOT_EXISTS_CODE)) {
+ return false;
+ } else {
+ throw new GoraException(ex);
+ }
+ }
+ }
+
+ @Override
+ public T get(K key, String[] fields) throws GoraException {
+ String[] avFields = getFieldsToQuery(fields);
+ Object[] keyl = null;
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ keyl = new Object[]{key};
+ } else {
+ //Composite key pending
+ }
+ //Avro fields to Ignite fields
+ List<String> dbFields = new ArrayList<>();
+ for (String af : avFields) {
+ dbFields.add(igniteMapping.getFields().get(af).getName());
+ }
+ String selectQuery = IgniteSQLBuilder.selectGet(igniteMapping, dbFields);
+ try (PreparedStatement stmt = connection.prepareStatement(selectQuery)) {
+ IgniteSQLBuilder.fillSelectStatement(stmt, igniteMapping, keyl);
+ ResultSet rs = stmt.executeQuery();
+ boolean data = rs.next();
+ T resp = null;
+ if (data) {
+ resp = newInstance(rs, fields);
+ if (rs.next()) {
+ LOG.warn("Multiple results for primary key {} in the schema {}, ignoring additional rows.", keyl, igniteMapping.getTableName());
+ }
+ }
+ rs.close();
+ return resp;
+ } catch (SQLException | IOException ex) {
+ throw new GoraException(ex);
+ }
+
+ }
+
+ public T newInstance(ResultSet rs, String[] fields) throws GoraException, SQLException, IOException {
+ fields = getFieldsToQuery(fields);
+ T persistent = newPersistent();
+ for (String f : fields) {
+ Schema.Field field = fieldMap.get(f);
+ Schema fieldSchema = field.schema();
+ String dbField = igniteMapping.getFields().get(f).getName();
+ Object sv = rs.getObject(dbField);
+ if (sv == null) {
+ continue;
+ }
+ Object v = deserializeFieldValue(field, fieldSchema, sv, persistent);
+ persistent.put(field.pos(), v);
+ persistent.setDirty(field.pos());
+ }
+ return persistent;
+ }
+
+ @SuppressWarnings("unchecked")
+ private Object deserializeFieldValue(Schema.Field field, Schema fieldSchema,
+ Object igniteValue, T persistent) throws IOException {
+ Object fieldValue = null;
+ switch (fieldSchema.getType()) {
+ case MAP:
+ case ARRAY:
+ case RECORD:
+ @SuppressWarnings("rawtypes") SpecificDatumReader reader = getDatumReader(fieldSchema);
+ fieldValue = IOUtils.deserialize((byte[]) igniteValue, reader,
+ persistent.get(field.pos()));
+ break;
+ case ENUM:
+ fieldValue = AvroUtils.getEnumValue(fieldSchema, igniteValue.toString());
+ break;
+ case FIXED:
+ break;
+ case BYTES:
+ fieldValue = ByteBuffer.wrap((byte[]) igniteValue);
+ break;
+ case STRING:
+ fieldValue = new Utf8(igniteValue.toString());
+ break;
+ case UNION:
+ if (fieldSchema.getTypes().size() == 2 && isNullable(fieldSchema)) {
+ int schemaPos = getUnionSchema(igniteValue, fieldSchema);
+ Schema unionSchema = fieldSchema.getTypes().get(schemaPos);
+ fieldValue = deserializeFieldValue(field, unionSchema, igniteValue, persistent);
+ } else {
+ reader = getDatumReader(fieldSchema);
+ fieldValue = IOUtils.deserialize((byte[]) igniteValue, reader,
+ persistent.get(field.pos()));
+ }
+ break;
+ default:
+ fieldValue = igniteValue;
+ }
+ return fieldValue;
+
+ }
+
+ @Override
+ public void put(K key, T obj) throws GoraException {
+ try {
+ if (obj.isDirty()) {
+ Schema schema = obj.getSchema();
+ List<Schema.Field> fields = schema.getFields();
+ Map<Column, Object> data = new HashMap<>();
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ Column getKey = igniteMapping.getPrimaryKey().get(0);
+ data.put(getKey, key);
+ } else {
+ //Composite keys pending..
+ }
+ for (Schema.Field field : fields) {
+ Column get = igniteMapping.getFields().get(field.name());
+ Object v = obj.get(field.pos());
+ if (get != null && v != null) {
+ Schema fieldSchema = field.schema();
+ Object serializedObj = serializeFieldValue(fieldSchema, v);
+ data.put(get, serializedObj);
+ }
+ }
+ String baseInsertStatement = IgniteSQLBuilder.baseInsertStatement(igniteMapping, data);
+ try (PreparedStatement stmt = connection.prepareStatement(baseInsertStatement)) {
+ IgniteSQLBuilder.fillInsertStatement(stmt, data);
+ stmt.executeUpdate();
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ } else {
+ LOG.info("Ignored putting object {} in the store as it is neither "
+ + "new, neither dirty.", new Object[]{obj});
+ }
+ } catch (Exception e) {
+ throw new GoraException(e);
+ }
+ }
+
+ @Override
+ public boolean delete(K key) throws GoraException {
+ String deleteQuery = null;
+ Object[] keyl = null;
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ deleteQuery = IgniteSQLBuilder.delete(igniteMapping);
+ keyl = new Object[]{key};
+ } else {
+ //Composite key pending
+ }
+ try (PreparedStatement stmt = connection.prepareStatement(deleteQuery)) {
+ IgniteSQLBuilder.fillDeleteStatement(stmt, igniteMapping, keyl);
+ stmt.executeUpdate();
+ return true;
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public long deleteByQuery(Query<K, T> query) throws GoraException {
+
+ String deleteQuery;
+ if (query.getFields() != null && query.getFields().length < igniteMapping.getFields().size()) {
+ List<String> dbFields = new ArrayList<>();
+ for (String af : query.getFields()) {
+ dbFields.add(igniteMapping.getFields().get(af).getName());
+ }
+ deleteQuery = IgniteSQLBuilder.deleteQueryFields(igniteMapping, dbFields);
+ } else {
+ deleteQuery = IgniteSQLBuilder.deleteQuery(igniteMapping);
+ }
+ String selectQueryWhere = IgniteSQLBuilder.queryWhere(igniteMapping, query.getStartKey(), query.getEndKey(), query.getLimit());
+ try (PreparedStatement stmt = connection.prepareStatement(deleteQuery + selectQueryWhere)) {
+ IgniteSQLBuilder.fillSelectQuery(stmt, query.getStartKey(), query.getEndKey());
+ stmt.executeUpdate();
+ return 0;
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public Result<K, T> execute(Query<K, T> query) throws GoraException {
+ String[] fields = getFieldsToQuery(query.getFields());
+ //Avro fields to Ignite fields
+ List<String> dbFields = new ArrayList<>();
+ for (String af : fields) {
+ dbFields.add(igniteMapping.getFields().get(af).getName());
+ }
+ String selectQuery = IgniteSQLBuilder.selectQuery(igniteMapping, dbFields);
+ String selectQueryWhere = IgniteSQLBuilder.queryWhere(igniteMapping, query.getStartKey(), query.getEndKey(), query.getLimit());
+ try (PreparedStatement stmt = connection.prepareStatement(selectQuery + selectQueryWhere)) {
+ RowSetFactory factory = RowSetProvider.newFactory();
+ CachedRowSet rowset = factory.createCachedRowSet();
+ IgniteSQLBuilder.fillSelectQuery(stmt, query.getStartKey(), query.getEndKey());
+ ResultSet executeQuery = stmt.executeQuery();
+ rowset.populate(executeQuery);
+ IgniteResult<K, T> igniteResult = new IgniteResult<>(this, query);
+ igniteResult.setResultSet(rowset);
+ return igniteResult;
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ public K extractKey(ResultSet r) throws SQLException {
+ assert igniteMapping.getPrimaryKey().size() == 1;
+ return (K) r.getObject(igniteMapping.getPrimaryKey().get(0).getName());
+ }
+
+ @Override
+ public Query<K, T> newQuery() {
+ IgniteQuery<K, T> query = new IgniteQuery<>(this);
+ query.setFields(getFieldsToQuery(null));
+ return query;
+ }
+
+ @Override
+ public List<PartitionQuery<K, T>> getPartitions(Query<K, T> query) throws IOException {
+ List<PartitionQuery<K, T>> partitions = new ArrayList<>();
+ PartitionQueryImpl<K, T> partitionQuery = new PartitionQueryImpl<>(
+ query);
+ partitionQuery.setConf(getConf());
+ partitions.add(partitionQuery);
+ return partitions;
+ }
+
+ @Override
+ public void flush() throws GoraException {
+ //Auto-commit mode by default
+ }
+
+ @Override
+ public void close() {
+ try {
+ connection.close();
+ LOG.info("Ignite datastore destroyed successfully.");
+ } catch (Exception ex) {
+ LOG.error(ex.getMessage(), ex);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ private Object serializeFieldValue(Schema fieldSchema, Object fieldValue) {
+ Object output = fieldValue;
+ switch (fieldSchema.getType()) {
+ case ARRAY:
+ case MAP:
+ case RECORD:
+ byte[] data = null;
+ try {
+ @SuppressWarnings("rawtypes")
+ SpecificDatumWriter writer = getDatumWriter(fieldSchema);
+ data = IOUtils.serialize(writer, fieldValue);
+ } catch (IOException e) {
+ LOG.error(e.getMessage(), e);
+ }
+ output = data;
+ break;
+ case UNION:
+ if (fieldSchema.getTypes().size() == 2 && isNullable(fieldSchema)) {
+ int schemaPos = getUnionSchema(fieldValue, fieldSchema);
+ Schema unionSchema = fieldSchema.getTypes().get(schemaPos);
+ output = serializeFieldValue(unionSchema, fieldValue);
+ } else {
+ data = null;
+ try {
+ @SuppressWarnings("rawtypes")
+ SpecificDatumWriter writer = getDatumWriter(fieldSchema);
+ data = IOUtils.serialize(writer, fieldValue);
+ } catch (IOException e) {
+ LOG.error(e.getMessage(), e);
+ }
+ output = data;
+ }
+ break;
+ case FIXED:
+ break;
+ case ENUM:
+ case STRING:
+ output = fieldValue.toString();
+ break;
+ case BYTES:
+ output = ((ByteBuffer) fieldValue).array();
+ break;
+ case INT:
+ case LONG:
+ case FLOAT:
+ case DOUBLE:
+ case BOOLEAN:
+ output = fieldValue;
+ break;
+ case NULL:
+ break;
+ default:
+ throw new AssertionError(fieldSchema.getType().name());
+
--- End diff --
Please remove unnecessary new lines.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202523567
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteStore.java ---
@@ -0,0 +1,578 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.sql.rowset.CachedRowSet;
+import javax.sql.rowset.RowSetFactory;
+import javax.sql.rowset.RowSetProvider;
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificDatumReader;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.ignite.query.IgniteQuery;
+import org.apache.gora.ignite.query.IgniteResult;
+import org.apache.gora.ignite.utils.IgniteSQLBuilder;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.AvroUtils;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of a Ignite data store to be used by gora.
+ *
+ * @param <K> class to be used for the key
+ * @param <T> class to be persisted within the store
+ */
+public class IgniteStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ public static final Logger LOG = LoggerFactory.getLogger(IgniteStore.class);
+ private static final String PARSE_MAPPING_FILE_KEY = "gora.ignite.mapping.file";
+ private static final String DEFAULT_MAPPING_FILE = "gora-ignite-mapping.xml";
+ private IgniteParameters igniteParameters;
+ private IgniteMapping igniteMapping;
+ private Connection connection;
+
+ /*
+ * Create a threadlocal map for the datum readers and writers, because they
+ * are not thread safe, at least not before Avro 1.4.0 (See AVRO-650). When
+ * they are thread safe, it is possible to maintain a single reader and writer
+ * pair for every schema, instead of one for every thread.
+ */
+ public static final ConcurrentHashMap<Schema, SpecificDatumReader<?>> readerMap = new ConcurrentHashMap<>();
+
+ public static final ConcurrentHashMap<Schema, SpecificDatumWriter<?>> writerMap = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) throws GoraException {
+
+ try {
+ super.initialize(keyClass, persistentClass, properties);
+ IgniteMappingBuilder builder = new IgniteMappingBuilder(this);
+ builder.readMappingFile(getConf().get(PARSE_MAPPING_FILE_KEY, DEFAULT_MAPPING_FILE));
+ igniteMapping = builder.getIgniteMapping();
+ igniteParameters = IgniteParameters.load(properties, conf);
+ connection = acquiereConnection();
+ LOG.info("Ignite store was successfully initialized");
+ } catch (ClassNotFoundException | SQLException ex) {
+ LOG.error("Error while initializing Ignite store", ex);
+ throw new GoraException(ex);
+ }
+ }
+
+ private Connection acquiereConnection() throws ClassNotFoundException, SQLException {
--- End diff --
Please spell ```acquire``` correctly.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202526790
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteStore.java ---
@@ -0,0 +1,578 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.sql.rowset.CachedRowSet;
+import javax.sql.rowset.RowSetFactory;
+import javax.sql.rowset.RowSetProvider;
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificDatumReader;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.ignite.query.IgniteQuery;
+import org.apache.gora.ignite.query.IgniteResult;
+import org.apache.gora.ignite.utils.IgniteSQLBuilder;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.AvroUtils;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of a Ignite data store to be used by gora.
+ *
+ * @param <K> class to be used for the key
+ * @param <T> class to be persisted within the store
+ */
+public class IgniteStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ public static final Logger LOG = LoggerFactory.getLogger(IgniteStore.class);
+ private static final String PARSE_MAPPING_FILE_KEY = "gora.ignite.mapping.file";
+ private static final String DEFAULT_MAPPING_FILE = "gora-ignite-mapping.xml";
+ private IgniteParameters igniteParameters;
+ private IgniteMapping igniteMapping;
+ private Connection connection;
+
+ /*
+ * Create a threadlocal map for the datum readers and writers, because they
+ * are not thread safe, at least not before Avro 1.4.0 (See AVRO-650). When
+ * they are thread safe, it is possible to maintain a single reader and writer
+ * pair for every schema, instead of one for every thread.
+ */
+ public static final ConcurrentHashMap<Schema, SpecificDatumReader<?>> readerMap = new ConcurrentHashMap<>();
+
+ public static final ConcurrentHashMap<Schema, SpecificDatumWriter<?>> writerMap = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) throws GoraException {
+
+ try {
+ super.initialize(keyClass, persistentClass, properties);
+ IgniteMappingBuilder builder = new IgniteMappingBuilder(this);
+ builder.readMappingFile(getConf().get(PARSE_MAPPING_FILE_KEY, DEFAULT_MAPPING_FILE));
+ igniteMapping = builder.getIgniteMapping();
+ igniteParameters = IgniteParameters.load(properties, conf);
+ connection = acquiereConnection();
+ LOG.info("Ignite store was successfully initialized");
+ } catch (ClassNotFoundException | SQLException ex) {
+ LOG.error("Error while initializing Ignite store", ex);
+ throw new GoraException(ex);
+ }
+ }
+
+ private Connection acquiereConnection() throws ClassNotFoundException, SQLException {
+ Class.forName("org.apache.ignite.IgniteJdbcThinDriver");
+ StringBuilder urlBuilder = new StringBuilder();
+ urlBuilder.append("jdbc:ignite:thin://");
+ urlBuilder.append(igniteParameters.getHost());
+ if (igniteParameters.getPort() != null) {
+ urlBuilder.append(":" + igniteParameters.getPort());
+ }
+ if (igniteParameters.getSchema() != null) {
+ urlBuilder.append("/" + igniteParameters.getSchema());
+ }
+ if (igniteParameters.getUser() != null) {
+ urlBuilder.append(";" + igniteParameters.getUser());
+ }
+ if (igniteParameters.getPassword() != null) {
+ urlBuilder.append(";" + igniteParameters.getPassword());
+ }
+ if (igniteParameters.getAdditionalConfigurations() != null) {
+ urlBuilder.append(igniteParameters.getAdditionalConfigurations());
+ }
+ Connection conn = DriverManager.getConnection(urlBuilder.toString());
+ return conn;
+ }
+
+ @Override
+ public String getSchemaName() {
+ return igniteMapping.getTableName();
+ }
+
+ @Override
+ public String getSchemaName(final String mappingSchemaName,
+ final Class<?> persistentClass) {
+ return super.getSchemaName(mappingSchemaName, persistentClass);
+ }
+
+ @Override
+ public void createSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to create the schema as no connection has been initiated.");
+ }
+ if (schemaExists()) {
+ return;
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String createTableSQL = IgniteSQLBuilder.createTable(igniteMapping);
+ stmt.executeUpdate(createTableSQL);
+ LOG.info("Table {} has been created for Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public void deleteSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to delete the schema as no connection has been initiated.");
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String dropTableSQL = IgniteSQLBuilder.dropTable(igniteMapping.getTableName());
+ stmt.executeUpdate(dropTableSQL);
+ LOG.info("Table {} has been dropped from Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public boolean schemaExists() throws GoraException {
+ boolean exists = false;
+ try (Statement stmt = connection.createStatement()) {
+ String tableExistsSQL = IgniteSQLBuilder.tableExists(igniteMapping.getTableName());
+ ResultSet executeQuery = stmt.executeQuery(tableExistsSQL);
+ executeQuery.close();
+ exists = true;
+ } catch (SQLException ex) {
+ /**
+ * a 42000 error code is thrown by Ignite when a non-existent table
+ * queried. More details:
+ * https://apacheignite-sql.readme.io/docs/jdbc-error-codes
+ */
+ if (ex.getSQLState() != null && ex.getSQLState().equals("42000")) {
+ exists = false;
+ } else {
+ throw new GoraException(ex);
+ }
+ }
+ return exists;
+ }
+
+ @Override
+ public T get(K key, String[] fields) throws GoraException {
+ String[] avFields = getFieldsToQuery(fields);
+ Object[] keyl = null;
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ keyl = new Object[]{key};
+ } else {
+ //Composite key pending
+ }
+ //Avro fields to Ignite fields
+ List<String> dbFields = new ArrayList<>();
+ for (String af : avFields) {
+ dbFields.add(igniteMapping.getFields().get(af).getName());
+ }
+ String selectQuery = IgniteSQLBuilder.selectGet(igniteMapping, dbFields);
+ try (PreparedStatement stmt = connection.prepareStatement(selectQuery)) {
+ IgniteSQLBuilder.fillSelectStatement(stmt, igniteMapping, keyl);
+ ResultSet rs = stmt.executeQuery();
+ boolean data = rs.next();
+ T resp = null;
+ if (data) {
+ resp = newInstance(rs, fields);
+ if (rs.next()) {
+ LOG.warn("Multiple results for primary key {} in the schema {}, ignoring additional rows.", keyl, igniteMapping.getTableName());
+ }
+ }
+ rs.close();
+ return resp;
+ } catch (SQLException | IOException ex) {
+ throw new GoraException(ex);
+ }
+
+ }
+
+ public T newInstance(ResultSet rs, String[] fields) throws GoraException, SQLException, IOException {
+ fields = getFieldsToQuery(fields);
+ T persistent = newPersistent();
+ for (String f : fields) {
+ Schema.Field field = fieldMap.get(f);
+ Schema fieldSchema = field.schema();
+ String dbField = igniteMapping.getFields().get(f).getName();
+ Object sv = rs.getObject(dbField);
+ if (sv == null) {
+ continue;
+ }
+ Object v = deserializeFieldValue(field, fieldSchema, sv, persistent);
+ persistent.put(field.pos(), v);
+ persistent.setDirty(field.pos());
+ }
+ return persistent;
+ }
+
+ private Object deserializeFieldValue(Schema.Field field, Schema fieldSchema,
+ Object igniteValue, T persistent) throws IOException {
+ Object fieldValue = null;
+ switch (fieldSchema.getType()) {
+ case MAP:
+ case ARRAY:
+ case RECORD:
+ @SuppressWarnings("rawtypes") SpecificDatumReader reader = getDatumReader(fieldSchema);
+ fieldValue = IOUtils.deserialize((byte[]) igniteValue, reader,
--- End diff --
Type safety: Unchecked invocation deserialize(byte[], SpecificDatumReader, Object) of the generic method deserialize(byte[], SpecificDatumReader< T >, T) of type IOUtils
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by nishadi <gi...@git.apache.org>.
Github user nishadi commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202521701
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/utils/IgniteSQLBuilder.java ---
@@ -0,0 +1,238 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.utils;
+
+import avro.shaded.com.google.common.collect.Lists;
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+import java.text.MessageFormat;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import org.apache.gora.ignite.store.Column;
+import org.apache.gora.ignite.store.IgniteMapping;
+
+/**
+ *
+ * SQL Builder utility for Ignite.
+ */
+public class IgniteSQLBuilder {
+
+ private static String format(String pattern, Object... args) {
+ MessageFormat messageFormat = new MessageFormat(pattern, Locale.getDefault());
+ return messageFormat.format(args);
+ }
+
+ public static String tableExists(String tableName) {
+ return format("SELECT * FROM {0} LIMIT 0", tableName);
+ }
+
+ public static String createTable(IgniteMapping mapping) {
+ StringBuilder sqlBuilder = new StringBuilder();
+ sqlBuilder.append("CREATE TABLE ");
+ sqlBuilder.append(mapping.getTableName());
+ sqlBuilder.append("(");
+ ArrayList<Column> fieldsList = Lists.newArrayList(mapping.getPrimaryKey());
+ fieldsList.addAll(Lists.newArrayList(mapping.getFields().values()));
+ for (Column aColumn : fieldsList) {
+ String name = aColumn.getName();
+ Column.FieldType dataType = aColumn.getDataType();
+ sqlBuilder.append(name).append(" ").append(dataType.toString()).append(",");
+ }
+ sqlBuilder.append("PRIMARY KEY ");
+ sqlBuilder.append("(");
+ for (int i = 0; i < mapping.getPrimaryKey().size(); i++) {
+ sqlBuilder.append(mapping.getPrimaryKey().get(i).getName());
+ sqlBuilder.append(i == mapping.getPrimaryKey().size() - 1 ? "" : ",");
+ }
+ sqlBuilder.append(")");
+ sqlBuilder.append(");");
+ return sqlBuilder.toString();
+ }
+
+ public static String dropTable(String tableName) {
+ return format("DROP TABLE IF EXISTS {0} ;", tableName);
+ }
+
+ public static String baseInsertStatement(IgniteMapping mapping, Map<Column, Object> data) {
+ StringBuilder sqlBuilder = new StringBuilder();
+ sqlBuilder.append("MERGE INTO ");
+ sqlBuilder.append(mapping.getTableName());
+ sqlBuilder.append(" (");
+ List<Entry<Column, Object>> list = new ArrayList<>(data.entrySet());
+ for (int i = 0; i < list.size(); i++) {
+ sqlBuilder.append(list.get(i).getKey().getName());
+ sqlBuilder.append(i == list.size() - 1 ? "" : ",");
+ }
+ sqlBuilder.append(")");
+ sqlBuilder.append(" VALUES ");
+ sqlBuilder.append(" (");
+ for (int i = 0; i < list.size(); i++) {
+ sqlBuilder.append("?");
+ sqlBuilder.append(i == list.size() - 1 ? "" : ",");
+ }
+ sqlBuilder.append(" )");
+ return sqlBuilder.toString();
+ }
+
+ public static void fillInsertStatement(PreparedStatement st, Map<Column, Object> data) throws SQLException {
+ List<Entry<Column, Object>> list = new ArrayList<>(data.entrySet());
+ for (int i = 0; i < list.size(); i++) {
+ int j = i + 1;
+ st.setObject(j, list.get(i).getValue());
+ }
+ }
+
+ public static String delete(IgniteMapping mapping) {
+ StringBuilder sqlBuilder = new StringBuilder();
+ sqlBuilder.append("DELETE FROM ");
+ sqlBuilder.append(mapping.getTableName());
+ sqlBuilder.append(" WHERE ");
+ for (int i = 0; i < mapping.getPrimaryKey().size(); i++) {
+ sqlBuilder.append(mapping.getPrimaryKey().get(i).getName());
+ sqlBuilder.append("= ? ");
+ sqlBuilder.append(i == mapping.getPrimaryKey().size() - 1 ? "" : " AND ");
+ }
+ return sqlBuilder.toString();
+ }
+
+ public static void fillDeleteStatement(PreparedStatement st, IgniteMapping mapping, Object... data) throws SQLException {
+ assert mapping.getPrimaryKey().size() == data.length;
+ for (int i = 0; i < mapping.getPrimaryKey().size(); i++) {
+ int j = i + 1;
+ st.setObject(j, data[i]);
+ }
+ }
+
+ public static String selectGet(IgniteMapping mapping, List<String> fields) {
+ StringBuilder sqlBuilder = new StringBuilder();
+ sqlBuilder.append("SELECT ");
+ for (int i = 0; i < fields.size(); i++) {
+ sqlBuilder.append(fields.get(i));
+ sqlBuilder.append(i == fields.size() - 1 ? "" : " , ");
+ }
+ sqlBuilder.append(" FROM ");
+ sqlBuilder.append(mapping.getTableName());
+ sqlBuilder.append(" WHERE ");
+ for (int i = 0; i < mapping.getPrimaryKey().size(); i++) {
+ sqlBuilder.append(mapping.getPrimaryKey().get(i).getName());
+ sqlBuilder.append("= ? ");
+ sqlBuilder.append(i == mapping.getPrimaryKey().size() - 1 ? "" : " AND ");
+ }
+ return sqlBuilder.toString();
+ }
+
+ public static void fillSelectStatement(PreparedStatement st, IgniteMapping mapping, Object... data) throws SQLException {
+ assert mapping.getPrimaryKey().size() == data.length;
+ for (int i = 0; i < mapping.getPrimaryKey().size(); i++) {
+ int j = i + 1;
+ st.setObject(j, data[i]);
+ }
+ }
+
+ public static String selectQuery(IgniteMapping mapping, List<String> ifields) {
+ List<String> fields = new ArrayList<>();
+ for (Column c : mapping.getPrimaryKey()) {
+ fields.add(c.getName());
+ }
+ fields.addAll(ifields);
+ StringBuilder sqlBuilder = new StringBuilder();
+ sqlBuilder.append("SELECT ");
+ for (int i = 0; i < fields.size(); i++) {
+ sqlBuilder.append(fields.get(i));
+ sqlBuilder.append(i == fields.size() - 1 ? "" : " , ");
+ }
+ sqlBuilder.append(" FROM ");
+ sqlBuilder.append(mapping.getTableName());
+ return sqlBuilder.toString();
+ }
+
+ public static String deleteQuery(IgniteMapping mapping) {
+ StringBuilder sqlBuilder = new StringBuilder();
+ sqlBuilder.append("DELETE FROM ");
+ sqlBuilder.append(mapping.getTableName());
+ return sqlBuilder.toString();
+ }
+
+ public static String deleteQueryFields(IgniteMapping mapping, List<String> lsFields) {
+ StringBuilder sqlBuilder = new StringBuilder();
+ sqlBuilder.append("UPDATE ");
+ sqlBuilder.append(mapping.getTableName());
+ if (!lsFields.isEmpty()) {
+ sqlBuilder.append(" SET ");
+ }
+ for (int i = 0; i < lsFields.size(); i++) {
+ sqlBuilder.append(lsFields.get(i));
+ sqlBuilder.append(" = null");
+ sqlBuilder.append(i == lsFields.size() - 1 ? "" : " , ");
+ }
+ return sqlBuilder.toString();
+ }
+
+ public static String selectQueryWhere(IgniteMapping mapping, Object sk, Object ek, long limit) {
+ //composite keys pending
+ assert mapping.getPrimaryKey().size() == 1;
+ String keycolumn = mapping.getPrimaryKey().get(0).getName();
+ StringBuilder sqlBuilder = new StringBuilder();
+ if (sk != null || ek != null) {
+ sqlBuilder.append(" WHERE ");
+ if (sk != null && ek != null && sk.equals(ek)) {
+ sqlBuilder.append(keycolumn);
+ sqlBuilder.append("= ?");
+ } else {
+ if (sk != null) {
+ sqlBuilder.append(keycolumn);
+ sqlBuilder.append(">= ?");
+ }
+ if (sk != null && ek != null) {
+ sqlBuilder.append(" AND ");
+ }
+ if (ek != null) {
+ sqlBuilder.append(keycolumn);
+ sqlBuilder.append("<= ?");
+ }
+ }
+ }
+ if (limit > 0) {
+ sqlBuilder.append(" LIMIT " + limit);
+ }
+ return sqlBuilder.toString();
+ }
+
+ public static void fillSelectQuery(PreparedStatement st, Object sk, Object ek) throws SQLException {
--- End diff --
Shall we add Java docs to all the public methods? It would be better if the variable names can be meaningful.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211238005
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteMappingBuilder.java ---
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import com.google.inject.ConfigurationException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.jdom.Document;
+import org.jdom.Element;
+import org.jdom.JDOMException;
+import org.jdom.input.SAXBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Builder for Mapping definitions of Ignite.
+ */
+public class IgniteMappingBuilder<K, T extends PersistentBase> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IgniteMappingBuilder.class);
+ /**
+ * Mapping instance being built
+ */
+ private IgniteMapping igniteMapping;
+
+ private final IgniteStore<K, T> dataStore;
+
+ public IgniteMappingBuilder(final IgniteStore<K, T> store) {
+ this.igniteMapping = new IgniteMapping();
+ this.dataStore = store;
+ }
+
+ public IgniteMapping getIgniteMapping() {
--- End diff --
Please document these public methods.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by nishadi <gi...@git.apache.org>.
Github user nishadi commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202534472
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteStore.java ---
@@ -0,0 +1,578 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.sql.rowset.CachedRowSet;
+import javax.sql.rowset.RowSetFactory;
+import javax.sql.rowset.RowSetProvider;
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificDatumReader;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.ignite.query.IgniteQuery;
+import org.apache.gora.ignite.query.IgniteResult;
+import org.apache.gora.ignite.utils.IgniteSQLBuilder;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.AvroUtils;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of a Ignite data store to be used by gora.
+ *
+ * @param <K> class to be used for the key
+ * @param <T> class to be persisted within the store
+ */
+public class IgniteStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ public static final Logger LOG = LoggerFactory.getLogger(IgniteStore.class);
+ private static final String PARSE_MAPPING_FILE_KEY = "gora.ignite.mapping.file";
+ private static final String DEFAULT_MAPPING_FILE = "gora-ignite-mapping.xml";
+ private IgniteParameters igniteParameters;
+ private IgniteMapping igniteMapping;
+ private Connection connection;
+
+ /*
+ * Create a threadlocal map for the datum readers and writers, because they
+ * are not thread safe, at least not before Avro 1.4.0 (See AVRO-650). When
+ * they are thread safe, it is possible to maintain a single reader and writer
+ * pair for every schema, instead of one for every thread.
+ */
+ public static final ConcurrentHashMap<Schema, SpecificDatumReader<?>> readerMap = new ConcurrentHashMap<>();
+
+ public static final ConcurrentHashMap<Schema, SpecificDatumWriter<?>> writerMap = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) throws GoraException {
+
+ try {
+ super.initialize(keyClass, persistentClass, properties);
+ IgniteMappingBuilder builder = new IgniteMappingBuilder(this);
+ builder.readMappingFile(getConf().get(PARSE_MAPPING_FILE_KEY, DEFAULT_MAPPING_FILE));
+ igniteMapping = builder.getIgniteMapping();
+ igniteParameters = IgniteParameters.load(properties, conf);
+ connection = acquiereConnection();
+ LOG.info("Ignite store was successfully initialized");
+ } catch (ClassNotFoundException | SQLException ex) {
+ LOG.error("Error while initializing Ignite store", ex);
+ throw new GoraException(ex);
+ }
+ }
+
+ private Connection acquiereConnection() throws ClassNotFoundException, SQLException {
+ Class.forName("org.apache.ignite.IgniteJdbcThinDriver");
+ StringBuilder urlBuilder = new StringBuilder();
+ urlBuilder.append("jdbc:ignite:thin://");
+ urlBuilder.append(igniteParameters.getHost());
+ if (igniteParameters.getPort() != null) {
+ urlBuilder.append(":" + igniteParameters.getPort());
+ }
+ if (igniteParameters.getSchema() != null) {
+ urlBuilder.append("/" + igniteParameters.getSchema());
+ }
+ if (igniteParameters.getUser() != null) {
+ urlBuilder.append(";" + igniteParameters.getUser());
+ }
+ if (igniteParameters.getPassword() != null) {
+ urlBuilder.append(";" + igniteParameters.getPassword());
+ }
+ if (igniteParameters.getAdditionalConfigurations() != null) {
+ urlBuilder.append(igniteParameters.getAdditionalConfigurations());
+ }
+ Connection conn = DriverManager.getConnection(urlBuilder.toString());
+ return conn;
+ }
+
+ @Override
+ public String getSchemaName() {
+ return igniteMapping.getTableName();
+ }
+
+ @Override
+ public String getSchemaName(final String mappingSchemaName,
+ final Class<?> persistentClass) {
+ return super.getSchemaName(mappingSchemaName, persistentClass);
+ }
+
+ @Override
+ public void createSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to create the schema as no connection has been initiated.");
+ }
+ if (schemaExists()) {
+ return;
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String createTableSQL = IgniteSQLBuilder.createTable(igniteMapping);
+ stmt.executeUpdate(createTableSQL);
+ LOG.info("Table {} has been created for Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public void deleteSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to delete the schema as no connection has been initiated.");
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String dropTableSQL = IgniteSQLBuilder.dropTable(igniteMapping.getTableName());
+ stmt.executeUpdate(dropTableSQL);
+ LOG.info("Table {} has been dropped from Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public boolean schemaExists() throws GoraException {
+ boolean exists = false;
+ try (Statement stmt = connection.createStatement()) {
+ String tableExistsSQL = IgniteSQLBuilder.tableExists(igniteMapping.getTableName());
+ ResultSet executeQuery = stmt.executeQuery(tableExistsSQL);
+ executeQuery.close();
+ exists = true;
+ } catch (SQLException ex) {
+ /**
+ * a 42000 error code is thrown by Ignite when a non-existent table
+ * queried. More details:
+ * https://apacheignite-sql.readme.io/docs/jdbc-error-codes
+ */
+ if (ex.getSQLState() != null && ex.getSQLState().equals("42000")) {
+ exists = false;
+ } else {
+ throw new GoraException(ex);
+ }
+ }
+ return exists;
+ }
+
+ @Override
+ public T get(K key, String[] fields) throws GoraException {
+ String[] avFields = getFieldsToQuery(fields);
+ Object[] keyl = null;
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ keyl = new Object[]{key};
+ } else {
+ //Composite key pending
+ }
+ //Avro fields to Ignite fields
+ List<String> dbFields = new ArrayList<>();
+ for (String af : avFields) {
+ dbFields.add(igniteMapping.getFields().get(af).getName());
+ }
+ String selectQuery = IgniteSQLBuilder.selectGet(igniteMapping, dbFields);
+ try (PreparedStatement stmt = connection.prepareStatement(selectQuery)) {
+ IgniteSQLBuilder.fillSelectStatement(stmt, igniteMapping, keyl);
+ ResultSet rs = stmt.executeQuery();
+ boolean data = rs.next();
+ T resp = null;
+ if (data) {
+ resp = newInstance(rs, fields);
+ if (rs.next()) {
+ LOG.warn("Multiple results for primary key {} in the schema {}, ignoring additional rows.", keyl, igniteMapping.getTableName());
+ }
+ }
+ rs.close();
+ return resp;
+ } catch (SQLException | IOException ex) {
+ throw new GoraException(ex);
+ }
+
+ }
+
+ public T newInstance(ResultSet rs, String[] fields) throws GoraException, SQLException, IOException {
+ fields = getFieldsToQuery(fields);
+ T persistent = newPersistent();
+ for (String f : fields) {
+ Schema.Field field = fieldMap.get(f);
+ Schema fieldSchema = field.schema();
+ String dbField = igniteMapping.getFields().get(f).getName();
+ Object sv = rs.getObject(dbField);
+ if (sv == null) {
+ continue;
+ }
+ Object v = deserializeFieldValue(field, fieldSchema, sv, persistent);
+ persistent.put(field.pos(), v);
+ persistent.setDirty(field.pos());
+ }
+ return persistent;
+ }
+
+ private Object deserializeFieldValue(Schema.Field field, Schema fieldSchema,
+ Object igniteValue, T persistent) throws IOException {
+ Object fieldValue = null;
+ switch (fieldSchema.getType()) {
+ case MAP:
+ case ARRAY:
+ case RECORD:
+ @SuppressWarnings("rawtypes") SpecificDatumReader reader = getDatumReader(fieldSchema);
+ fieldValue = IOUtils.deserialize((byte[]) igniteValue, reader,
+ persistent.get(field.pos()));
+ break;
+ case ENUM:
+ fieldValue = AvroUtils.getEnumValue(fieldSchema, igniteValue.toString());
+ break;
+ case FIXED:
+ break;
+ case BYTES:
+ fieldValue = ByteBuffer.wrap((byte[]) igniteValue);
+ break;
+ case STRING:
+ fieldValue = new Utf8(igniteValue.toString());
+ break;
+ case UNION:
+ if (fieldSchema.getTypes().size() == 2 && isNullable(fieldSchema)) {
+ int schemaPos = getUnionSchema(igniteValue, fieldSchema);
+ Schema unionSchema = fieldSchema.getTypes().get(schemaPos);
+ fieldValue = deserializeFieldValue(field, unionSchema, igniteValue, persistent);
+ } else {
+ reader = getDatumReader(fieldSchema);
+ fieldValue = IOUtils.deserialize((byte[]) igniteValue, reader,
+ persistent.get(field.pos()));
+ }
+ break;
+ default:
+ fieldValue = igniteValue;
+ }
+ return fieldValue;
+
+ }
+
+ @Override
+ public void put(K key, T obj) throws GoraException {
+ try {
+ if (obj.isDirty()) {
+ Schema schema = obj.getSchema();
+ List<Schema.Field> fields = schema.getFields();
+ Map<Column, Object> data = new HashMap<>();
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ Column getKey = igniteMapping.getPrimaryKey().get(0);
+ data.put(getKey, key);
+ } else {
+ //Composite keys pending..
+ }
+ for (Schema.Field field : fields) {
+ Column get = igniteMapping.getFields().get(field.name());
+ Object v = obj.get(field.pos());
+ if (get != null && v != null) {
+ Schema fieldSchema = field.schema();
+ Object serializedObj = serializeFieldValue(get, fieldSchema, v);
+ data.put(get, serializedObj);
+ }
+ }
+ String baseInsertStatement = IgniteSQLBuilder.baseInsertStatement(igniteMapping, data);
+ try (PreparedStatement stmt = connection.prepareStatement(baseInsertStatement)) {
+ IgniteSQLBuilder.fillInsertStatement(stmt, data);
+ stmt.executeUpdate();
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ } else {
+ LOG.info("Ignored putting object {} in the store as it is neither "
+ + "new, neither dirty.", new Object[]{obj});
+ }
+ } catch (Exception e) {
+ throw new GoraException(e);
+ }
+ }
+
+ @Override
+ public boolean delete(K key) throws GoraException {
+ String deleteQuery = null;
+ Object[] keyl = null;
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ deleteQuery = IgniteSQLBuilder.delete(igniteMapping);
+ keyl = new Object[]{key};
+ } else {
+ //Composite key pending
+ }
+ try (PreparedStatement stmt = connection.prepareStatement(deleteQuery)) {
+ IgniteSQLBuilder.fillDeleteStatement(stmt, igniteMapping, keyl);
+ stmt.executeUpdate();
+ return true;
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public long deleteByQuery(Query<K, T> query) throws GoraException {
+
+ String deleteQuery;
+ if (query.getFields() != null && query.getFields().length < igniteMapping.getFields().size()) {
+ List<String> dbFields = new ArrayList<>();
+ for (String af : query.getFields()) {
+ dbFields.add(igniteMapping.getFields().get(af).getName());
+ }
+ deleteQuery = IgniteSQLBuilder.deleteQueryFields(igniteMapping, dbFields);
+ } else {
+ deleteQuery = IgniteSQLBuilder.deleteQuery(igniteMapping);
+ }
+ String selectQueryWhere = IgniteSQLBuilder.selectQueryWhere(igniteMapping, query.getStartKey(), query.getEndKey(), query.getLimit());
+ try (PreparedStatement stmt = connection.prepareStatement(deleteQuery + selectQueryWhere)) {
+ IgniteSQLBuilder.fillSelectQuery(stmt, query.getStartKey(), query.getEndKey());
+ stmt.executeUpdate();
+ return 0;
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public Result<K, T> execute(Query<K, T> query) throws GoraException {
+ String[] fields = getFieldsToQuery(query.getFields());
+ //Avro fields to Ignite fields
+ List<String> dbFields = new ArrayList<>();
+ for (String af : fields) {
+ dbFields.add(igniteMapping.getFields().get(af).getName());
+ }
+ String selectQuery = IgniteSQLBuilder.selectQuery(igniteMapping, dbFields);
+ String selectQueryWhere = IgniteSQLBuilder.selectQueryWhere(igniteMapping, query.getStartKey(), query.getEndKey(), query.getLimit());
+ try {
+ PreparedStatement stmt = connection.prepareStatement(selectQuery + selectQueryWhere);
+ RowSetFactory factory = RowSetProvider.newFactory();
+ CachedRowSet rowset = factory.createCachedRowSet();
+ IgniteSQLBuilder.fillSelectQuery(stmt, query.getStartKey(), query.getEndKey());
+ ResultSet executeQuery = stmt.executeQuery();
+ rowset.populate(executeQuery);
+ IgniteResult<K, T> igniteResult = new IgniteResult<>(this, query);
+ igniteResult.setResultSet(rowset);
+ return igniteResult;
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ public K extractKey(ResultSet r) throws SQLException {
+ assert igniteMapping.getPrimaryKey().size() == 1;
+ return (K) r.getObject(igniteMapping.getPrimaryKey().get(0).getName());
+ }
+
+ @Override
+ public Query<K, T> newQuery() {
+ IgniteQuery<K, T> query = new IgniteQuery<>(this);
+ query.setFields(getFieldsToQuery(null));
+ return query;
+ }
+
+ @Override
+ public List<PartitionQuery<K, T>> getPartitions(Query<K, T> query) throws IOException {
+ List<PartitionQuery<K, T>> partitions = new ArrayList<>();
+ PartitionQueryImpl<K, T> partitionQuery = new PartitionQueryImpl<>(
+ query);
+ partitionQuery.setConf(getConf());
+ partitions.add(partitionQuery);
+ return partitions;
+ }
+
+ @Override
+ public void flush() throws GoraException {
+ //Auto-commit mode by default
+ }
+
+ @Override
+ public void close() {
+ try {
+ connection.close();
+ LOG.info("Ignite datastore destroyed successfully.");
+ } catch (Exception ex) {
+ LOG.error(ex.getMessage(), ex);
+ }
+ }
+
+ private Object serializeFieldValue(Column get, Schema fieldSchema, Object fieldValue) {
--- End diff --
Is there any specific reason why we are accepting the variable 'get' in here, as it is not being used?
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by djkevincr <gi...@git.apache.org>.
Github user djkevincr commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r211541050
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteStore.java ---
@@ -0,0 +1,565 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.sql.rowset.CachedRowSet;
+import javax.sql.rowset.RowSetFactory;
+import javax.sql.rowset.RowSetProvider;
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificDatumReader;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.ignite.query.IgniteQuery;
+import org.apache.gora.ignite.query.IgniteResult;
+import org.apache.gora.ignite.utils.IgniteBackendConstants;
+import org.apache.gora.ignite.utils.IgniteSQLBuilder;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.AvroUtils;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of a Ignite data store to be used by gora.
+ *
+ * @param <K> class to be used for the key
+ * @param <T> class to be persisted within the store
+ */
+public class IgniteStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(IgniteStore.class);
+ private static final String PARSE_MAPPING_FILE_KEY = "gora.ignite.mapping.file";
+ private static final String DEFAULT_MAPPING_FILE = "gora-ignite-mapping.xml";
+ private IgniteParameters igniteParameters;
+ private IgniteMapping igniteMapping;
+ private Connection connection;
+ private static final ConcurrentHashMap<Schema, SpecificDatumReader<?>> readerMap = new ConcurrentHashMap<>();
+ private static final ConcurrentHashMap<Schema, SpecificDatumWriter<?>> writerMap = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) throws GoraException {
+ try {
+ super.initialize(keyClass, persistentClass, properties);
+ IgniteMappingBuilder<K, T> builder = new IgniteMappingBuilder<K, T>(this);
+ builder.readMappingFile(getConf().get(PARSE_MAPPING_FILE_KEY, DEFAULT_MAPPING_FILE));
+ igniteMapping = builder.getIgniteMapping();
+ igniteParameters = IgniteParameters.load(properties);
+ connection = acquireConnection();
+ LOG.info("Ignite store was successfully initialized");
+ } catch (ClassNotFoundException | SQLException ex) {
+ LOG.error("Error while initializing Ignite store", ex);
+ throw new GoraException(ex);
+ }
+ }
+
+ private Connection acquireConnection() throws ClassNotFoundException, SQLException {
+ Class.forName("org.apache.ignite.IgniteJdbcThinDriver");
+ StringBuilder urlBuilder = new StringBuilder();
+ urlBuilder.append("jdbc:ignite:thin://");
+ urlBuilder.append(igniteParameters.getHost());
+ if (igniteParameters.getPort() != null) {
+ urlBuilder.append(":" + igniteParameters.getPort());
+ }
+ if (igniteParameters.getSchema() != null) {
+ urlBuilder.append("/" + igniteParameters.getSchema());
+ }
+ if (igniteParameters.getUser() != null) {
+ urlBuilder.append(";" + igniteParameters.getUser());
+ }
+ if (igniteParameters.getPassword() != null) {
+ urlBuilder.append(";" + igniteParameters.getPassword());
+ }
+ if (igniteParameters.getAdditionalConfigurations() != null) {
+ urlBuilder.append(igniteParameters.getAdditionalConfigurations());
+ }
+ return DriverManager.getConnection(urlBuilder.toString());
+ }
+
+ @Override
+ public String getSchemaName() {
+ return igniteMapping.getTableName();
+ }
+
+ @Override
+ public String getSchemaName(final String mappingSchemaName,
+ final Class<?> persistentClass) {
+ return super.getSchemaName(mappingSchemaName, persistentClass);
+ }
+
+ @Override
+ public void createSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to create the schema as no connection has been initiated.");
+ }
+ if (schemaExists()) {
+ return;
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String createTableSQL = IgniteSQLBuilder.createTable(igniteMapping);
+ stmt.executeUpdate(createTableSQL);
+ LOG.info("Table {} has been created for Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public void deleteSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to delete the schema as no connection has been initiated.");
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String dropTableSQL = IgniteSQLBuilder.dropTable(igniteMapping.getTableName());
+ stmt.executeUpdate(dropTableSQL);
+ LOG.info("Table {} has been dropped from Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public boolean schemaExists() throws GoraException {
+ try (Statement stmt = connection.createStatement()) {
+ String tableExistsSQL = IgniteSQLBuilder.tableExists(igniteMapping.getTableName());
+ ResultSet executeQuery = stmt.executeQuery(tableExistsSQL);
+ executeQuery.close();
+ return true;
+ } catch (SQLException ex) {
+ if (ex.getSQLState() != null
+ && ex.getSQLState().equals(IgniteBackendConstants.DEFAULT_IGNITE_TABLE_NOT_EXISTS_CODE)) {
+ return false;
+ } else {
+ throw new GoraException(ex);
+ }
+ }
+ }
+
+ @Override
+ public T get(K key, String[] fields) throws GoraException {
+ String[] avFields = getFieldsToQuery(fields);
+ Object[] keyl = null;
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ keyl = new Object[]{key};
+ } else {
+ //Composite key pending
+ }
+ //Avro fields to Ignite fields
+ List<String> dbFields = new ArrayList<>();
+ for (String af : avFields) {
+ dbFields.add(igniteMapping.getFields().get(af).getName());
+ }
+ String selectQuery = IgniteSQLBuilder.selectGet(igniteMapping, dbFields);
+ try (PreparedStatement stmt = connection.prepareStatement(selectQuery)) {
+ IgniteSQLBuilder.fillSelectStatement(stmt, igniteMapping, keyl);
+ ResultSet rs = stmt.executeQuery();
+ boolean data = rs.next();
+ T resp = null;
+ if (data) {
+ resp = newInstance(rs, fields);
+ if (rs.next()) {
+ LOG.warn("Multiple results for primary key {} in the schema {}, ignoring additional rows.", keyl, igniteMapping.getTableName());
+ }
+ }
+ rs.close();
+ return resp;
+ } catch (SQLException | IOException ex) {
+ throw new GoraException(ex);
+ }
+
--- End diff --
Please remove unnecessary new lines.
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by nishadi <gi...@git.apache.org>.
Github user nishadi commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202522577
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteParameters.java ---
@@ -0,0 +1,139 @@
+/*
+ * Copyright 2018 The Apache Software Foundation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.util.Properties;
+import org.apache.hadoop.conf.Configuration;
+
+/**
+ * Parameters definitions for Ignite.
+ */
+public class IgniteParameters {
+
+ /**
+ * Property indicating the Ignite Schema to be used
+ */
+ public static final String PROP_SCHEMA = "gora.datastore.ignite.schema";
+
+ /**
+ * Property indicating the Ignite Cluster Node IP
+ */
+ public static final String PROP_HOST = "gora.datastore.ignite.host";
+
+ /**
+ * Property indicating the port used by the Ignite Server
+ */
+ public static final String PROP_PORT = "gora.datastore.ignite.port";
+
+ /**
+ * Property indicating the username to connect to the server
+ */
+ public static final String PROP_USER = "gora.datastore.ignite.user";
+
+ /**
+ * Property indicating the password to connect to the server
+ */
+ public static final String PROP_PASSWORD = "gora.datastore.ignite.password";
+
+ /**
+ * Property indicating additional JDBC options
+ */
+ public static final String PROP_ADDITIONALS = "gora.datastore.ignite.additionalConfigurations";
+
+ private String host;
+ private String port;
+ private String schema;
+ private String user;
+ private String password;
+ private String additionalConfigurations;
+
+ /**
+ *
+ * @param host
+ * @param port Optional port for Ignite Server
+ * @param user Optional username for Ignite
+ * @param password Optional password for Ignite
+ * @param additionalConfigurations Optional additional configurations for
+ * Ignite
+ */
+ private IgniteParameters(String host, String port, String schema, String user, String password, String additionalConfigurations) {
+ this.host = host;
+ this.port = port;
+ this.schema = schema;
+ this.user = user;
+ this.password = password;
+ this.additionalConfigurations = additionalConfigurations;
+ }
+
+ public String getHost() {
+ return host;
+ }
+
+ public void setHost(String host) {
+ this.host = host;
+ }
+
+ public String getPort() {
+ return port;
+ }
+
+ public void setPort(String port) {
+ this.port = port;
+ }
+
+ public String getUser() {
+ return user;
+ }
+
+ public void setUser(String user) {
+ this.user = user;
+ }
+
+ public String getPassword() {
+ return password;
+ }
+
+ public void setPassword(String password) {
+ this.password = password;
+ }
+
+ public String getAdditionalConfigurations() {
+ return additionalConfigurations;
+ }
+
+ public void setAdditionalConfigurations(String additionalConfigurations) {
+ this.additionalConfigurations = additionalConfigurations;
+ }
+
+ public String getSchema() {
+ return schema;
+ }
+
+ public void setSchema(String schema) {
+ this.schema = schema;
+ }
+
+ public static IgniteParameters load(Properties properties, Configuration conf) {
+ return new IgniteParameters(
+ properties.getProperty(PROP_HOST, "localhost"),
+ properties.getProperty(PROP_PORT, "10800"),
--- End diff --
Shall we move the constants to a separate constants file?
---
[GitHub] gora pull request #134: GORA-535 Add a data store for Apache Ignite
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on a diff in the pull request:
https://github.com/apache/gora/pull/134#discussion_r202526802
--- Diff: gora-ignite/src/main/java/org/apache/gora/ignite/store/IgniteStore.java ---
@@ -0,0 +1,578 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gora.ignite.store;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import javax.sql.rowset.CachedRowSet;
+import javax.sql.rowset.RowSetFactory;
+import javax.sql.rowset.RowSetProvider;
+import org.apache.avro.Schema;
+import org.apache.avro.specific.SpecificDatumReader;
+import org.apache.avro.specific.SpecificDatumWriter;
+import org.apache.avro.util.Utf8;
+import org.apache.gora.ignite.query.IgniteQuery;
+import org.apache.gora.ignite.query.IgniteResult;
+import org.apache.gora.ignite.utils.IgniteSQLBuilder;
+import org.apache.gora.persistency.Persistent;
+import org.apache.gora.persistency.impl.PersistentBase;
+import org.apache.gora.query.PartitionQuery;
+import org.apache.gora.query.Query;
+import org.apache.gora.query.Result;
+import org.apache.gora.query.impl.PartitionQueryImpl;
+import org.apache.gora.store.impl.DataStoreBase;
+import org.apache.gora.util.AvroUtils;
+import org.apache.gora.util.GoraException;
+import org.apache.gora.util.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of a Ignite data store to be used by gora.
+ *
+ * @param <K> class to be used for the key
+ * @param <T> class to be persisted within the store
+ */
+public class IgniteStore<K, T extends PersistentBase> extends DataStoreBase<K, T> {
+
+ public static final Logger LOG = LoggerFactory.getLogger(IgniteStore.class);
+ private static final String PARSE_MAPPING_FILE_KEY = "gora.ignite.mapping.file";
+ private static final String DEFAULT_MAPPING_FILE = "gora-ignite-mapping.xml";
+ private IgniteParameters igniteParameters;
+ private IgniteMapping igniteMapping;
+ private Connection connection;
+
+ /*
+ * Create a threadlocal map for the datum readers and writers, because they
+ * are not thread safe, at least not before Avro 1.4.0 (See AVRO-650). When
+ * they are thread safe, it is possible to maintain a single reader and writer
+ * pair for every schema, instead of one for every thread.
+ */
+ public static final ConcurrentHashMap<Schema, SpecificDatumReader<?>> readerMap = new ConcurrentHashMap<>();
+
+ public static final ConcurrentHashMap<Schema, SpecificDatumWriter<?>> writerMap = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(Class<K> keyClass, Class<T> persistentClass, Properties properties) throws GoraException {
+
+ try {
+ super.initialize(keyClass, persistentClass, properties);
+ IgniteMappingBuilder builder = new IgniteMappingBuilder(this);
+ builder.readMappingFile(getConf().get(PARSE_MAPPING_FILE_KEY, DEFAULT_MAPPING_FILE));
+ igniteMapping = builder.getIgniteMapping();
+ igniteParameters = IgniteParameters.load(properties, conf);
+ connection = acquiereConnection();
+ LOG.info("Ignite store was successfully initialized");
+ } catch (ClassNotFoundException | SQLException ex) {
+ LOG.error("Error while initializing Ignite store", ex);
+ throw new GoraException(ex);
+ }
+ }
+
+ private Connection acquiereConnection() throws ClassNotFoundException, SQLException {
+ Class.forName("org.apache.ignite.IgniteJdbcThinDriver");
+ StringBuilder urlBuilder = new StringBuilder();
+ urlBuilder.append("jdbc:ignite:thin://");
+ urlBuilder.append(igniteParameters.getHost());
+ if (igniteParameters.getPort() != null) {
+ urlBuilder.append(":" + igniteParameters.getPort());
+ }
+ if (igniteParameters.getSchema() != null) {
+ urlBuilder.append("/" + igniteParameters.getSchema());
+ }
+ if (igniteParameters.getUser() != null) {
+ urlBuilder.append(";" + igniteParameters.getUser());
+ }
+ if (igniteParameters.getPassword() != null) {
+ urlBuilder.append(";" + igniteParameters.getPassword());
+ }
+ if (igniteParameters.getAdditionalConfigurations() != null) {
+ urlBuilder.append(igniteParameters.getAdditionalConfigurations());
+ }
+ Connection conn = DriverManager.getConnection(urlBuilder.toString());
+ return conn;
+ }
+
+ @Override
+ public String getSchemaName() {
+ return igniteMapping.getTableName();
+ }
+
+ @Override
+ public String getSchemaName(final String mappingSchemaName,
+ final Class<?> persistentClass) {
+ return super.getSchemaName(mappingSchemaName, persistentClass);
+ }
+
+ @Override
+ public void createSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to create the schema as no connection has been initiated.");
+ }
+ if (schemaExists()) {
+ return;
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String createTableSQL = IgniteSQLBuilder.createTable(igniteMapping);
+ stmt.executeUpdate(createTableSQL);
+ LOG.info("Table {} has been created for Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public void deleteSchema() throws GoraException {
+ if (connection == null) {
+ throw new GoraException(
+ "Impossible to delete the schema as no connection has been initiated.");
+ }
+ try (Statement stmt = connection.createStatement()) {
+ String dropTableSQL = IgniteSQLBuilder.dropTable(igniteMapping.getTableName());
+ stmt.executeUpdate(dropTableSQL);
+ LOG.info("Table {} has been dropped from Ignite instance.",
+ igniteMapping.getTableName());
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public boolean schemaExists() throws GoraException {
+ boolean exists = false;
+ try (Statement stmt = connection.createStatement()) {
+ String tableExistsSQL = IgniteSQLBuilder.tableExists(igniteMapping.getTableName());
+ ResultSet executeQuery = stmt.executeQuery(tableExistsSQL);
+ executeQuery.close();
+ exists = true;
+ } catch (SQLException ex) {
+ /**
+ * a 42000 error code is thrown by Ignite when a non-existent table
+ * queried. More details:
+ * https://apacheignite-sql.readme.io/docs/jdbc-error-codes
+ */
+ if (ex.getSQLState() != null && ex.getSQLState().equals("42000")) {
+ exists = false;
+ } else {
+ throw new GoraException(ex);
+ }
+ }
+ return exists;
+ }
+
+ @Override
+ public T get(K key, String[] fields) throws GoraException {
+ String[] avFields = getFieldsToQuery(fields);
+ Object[] keyl = null;
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ keyl = new Object[]{key};
+ } else {
+ //Composite key pending
+ }
+ //Avro fields to Ignite fields
+ List<String> dbFields = new ArrayList<>();
+ for (String af : avFields) {
+ dbFields.add(igniteMapping.getFields().get(af).getName());
+ }
+ String selectQuery = IgniteSQLBuilder.selectGet(igniteMapping, dbFields);
+ try (PreparedStatement stmt = connection.prepareStatement(selectQuery)) {
+ IgniteSQLBuilder.fillSelectStatement(stmt, igniteMapping, keyl);
+ ResultSet rs = stmt.executeQuery();
+ boolean data = rs.next();
+ T resp = null;
+ if (data) {
+ resp = newInstance(rs, fields);
+ if (rs.next()) {
+ LOG.warn("Multiple results for primary key {} in the schema {}, ignoring additional rows.", keyl, igniteMapping.getTableName());
+ }
+ }
+ rs.close();
+ return resp;
+ } catch (SQLException | IOException ex) {
+ throw new GoraException(ex);
+ }
+
+ }
+
+ public T newInstance(ResultSet rs, String[] fields) throws GoraException, SQLException, IOException {
+ fields = getFieldsToQuery(fields);
+ T persistent = newPersistent();
+ for (String f : fields) {
+ Schema.Field field = fieldMap.get(f);
+ Schema fieldSchema = field.schema();
+ String dbField = igniteMapping.getFields().get(f).getName();
+ Object sv = rs.getObject(dbField);
+ if (sv == null) {
+ continue;
+ }
+ Object v = deserializeFieldValue(field, fieldSchema, sv, persistent);
+ persistent.put(field.pos(), v);
+ persistent.setDirty(field.pos());
+ }
+ return persistent;
+ }
+
+ private Object deserializeFieldValue(Schema.Field field, Schema fieldSchema,
+ Object igniteValue, T persistent) throws IOException {
+ Object fieldValue = null;
+ switch (fieldSchema.getType()) {
+ case MAP:
+ case ARRAY:
+ case RECORD:
+ @SuppressWarnings("rawtypes") SpecificDatumReader reader = getDatumReader(fieldSchema);
+ fieldValue = IOUtils.deserialize((byte[]) igniteValue, reader,
+ persistent.get(field.pos()));
+ break;
+ case ENUM:
+ fieldValue = AvroUtils.getEnumValue(fieldSchema, igniteValue.toString());
+ break;
+ case FIXED:
+ break;
+ case BYTES:
+ fieldValue = ByteBuffer.wrap((byte[]) igniteValue);
+ break;
+ case STRING:
+ fieldValue = new Utf8(igniteValue.toString());
+ break;
+ case UNION:
+ if (fieldSchema.getTypes().size() == 2 && isNullable(fieldSchema)) {
+ int schemaPos = getUnionSchema(igniteValue, fieldSchema);
+ Schema unionSchema = fieldSchema.getTypes().get(schemaPos);
+ fieldValue = deserializeFieldValue(field, unionSchema, igniteValue, persistent);
+ } else {
+ reader = getDatumReader(fieldSchema);
+ fieldValue = IOUtils.deserialize((byte[]) igniteValue, reader,
+ persistent.get(field.pos()));
+ }
+ break;
+ default:
+ fieldValue = igniteValue;
+ }
+ return fieldValue;
+
+ }
+
+ @Override
+ public void put(K key, T obj) throws GoraException {
+ try {
+ if (obj.isDirty()) {
+ Schema schema = obj.getSchema();
+ List<Schema.Field> fields = schema.getFields();
+ Map<Column, Object> data = new HashMap<>();
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ Column getKey = igniteMapping.getPrimaryKey().get(0);
+ data.put(getKey, key);
+ } else {
+ //Composite keys pending..
+ }
+ for (Schema.Field field : fields) {
+ Column get = igniteMapping.getFields().get(field.name());
+ Object v = obj.get(field.pos());
+ if (get != null && v != null) {
+ Schema fieldSchema = field.schema();
+ Object serializedObj = serializeFieldValue(get, fieldSchema, v);
+ data.put(get, serializedObj);
+ }
+ }
+ String baseInsertStatement = IgniteSQLBuilder.baseInsertStatement(igniteMapping, data);
+ try (PreparedStatement stmt = connection.prepareStatement(baseInsertStatement)) {
+ IgniteSQLBuilder.fillInsertStatement(stmt, data);
+ stmt.executeUpdate();
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ } else {
+ LOG.info("Ignored putting object {} in the store as it is neither "
+ + "new, neither dirty.", new Object[]{obj});
+ }
+ } catch (Exception e) {
+ throw new GoraException(e);
+ }
+ }
+
+ @Override
+ public boolean delete(K key) throws GoraException {
+ String deleteQuery = null;
+ Object[] keyl = null;
+ if (igniteMapping.getPrimaryKey().size() == 1) {
+ deleteQuery = IgniteSQLBuilder.delete(igniteMapping);
+ keyl = new Object[]{key};
+ } else {
+ //Composite key pending
+ }
+ try (PreparedStatement stmt = connection.prepareStatement(deleteQuery)) {
+ IgniteSQLBuilder.fillDeleteStatement(stmt, igniteMapping, keyl);
+ stmt.executeUpdate();
+ return true;
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public long deleteByQuery(Query<K, T> query) throws GoraException {
+
+ String deleteQuery;
+ if (query.getFields() != null && query.getFields().length < igniteMapping.getFields().size()) {
+ List<String> dbFields = new ArrayList<>();
+ for (String af : query.getFields()) {
+ dbFields.add(igniteMapping.getFields().get(af).getName());
+ }
+ deleteQuery = IgniteSQLBuilder.deleteQueryFields(igniteMapping, dbFields);
+ } else {
+ deleteQuery = IgniteSQLBuilder.deleteQuery(igniteMapping);
+ }
+ String selectQueryWhere = IgniteSQLBuilder.selectQueryWhere(igniteMapping, query.getStartKey(), query.getEndKey(), query.getLimit());
+ try (PreparedStatement stmt = connection.prepareStatement(deleteQuery + selectQueryWhere)) {
+ IgniteSQLBuilder.fillSelectQuery(stmt, query.getStartKey(), query.getEndKey());
+ stmt.executeUpdate();
+ return 0;
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ @Override
+ public Result<K, T> execute(Query<K, T> query) throws GoraException {
+ String[] fields = getFieldsToQuery(query.getFields());
+ //Avro fields to Ignite fields
+ List<String> dbFields = new ArrayList<>();
+ for (String af : fields) {
+ dbFields.add(igniteMapping.getFields().get(af).getName());
+ }
+ String selectQuery = IgniteSQLBuilder.selectQuery(igniteMapping, dbFields);
+ String selectQueryWhere = IgniteSQLBuilder.selectQueryWhere(igniteMapping, query.getStartKey(), query.getEndKey(), query.getLimit());
+ try {
+ PreparedStatement stmt = connection.prepareStatement(selectQuery + selectQueryWhere);
+ RowSetFactory factory = RowSetProvider.newFactory();
+ CachedRowSet rowset = factory.createCachedRowSet();
+ IgniteSQLBuilder.fillSelectQuery(stmt, query.getStartKey(), query.getEndKey());
+ ResultSet executeQuery = stmt.executeQuery();
+ rowset.populate(executeQuery);
+ IgniteResult<K, T> igniteResult = new IgniteResult<>(this, query);
+ igniteResult.setResultSet(rowset);
+ return igniteResult;
+ } catch (SQLException ex) {
+ throw new GoraException(ex);
+ }
+ }
+
+ public K extractKey(ResultSet r) throws SQLException {
+ assert igniteMapping.getPrimaryKey().size() == 1;
+ return (K) r.getObject(igniteMapping.getPrimaryKey().get(0).getName());
+ }
+
+ @Override
+ public Query<K, T> newQuery() {
+ IgniteQuery<K, T> query = new IgniteQuery<>(this);
+ query.setFields(getFieldsToQuery(null));
+ return query;
+ }
+
+ @Override
+ public List<PartitionQuery<K, T>> getPartitions(Query<K, T> query) throws IOException {
+ List<PartitionQuery<K, T>> partitions = new ArrayList<>();
+ PartitionQueryImpl<K, T> partitionQuery = new PartitionQueryImpl<>(
+ query);
+ partitionQuery.setConf(getConf());
+ partitions.add(partitionQuery);
+ return partitions;
+ }
+
+ @Override
+ public void flush() throws GoraException {
+ //Auto-commit mode by default
+ }
+
+ @Override
+ public void close() {
+ try {
+ connection.close();
+ LOG.info("Ignite datastore destroyed successfully.");
+ } catch (Exception ex) {
+ LOG.error(ex.getMessage(), ex);
+ }
+ }
+
+ private Object serializeFieldValue(Column get, Schema fieldSchema, Object fieldValue) {
+ Object output = fieldValue;
+ switch (fieldSchema.getType()) {
+ case ARRAY:
+ case MAP:
+ case RECORD:
+ byte[] data = null;
+ try {
+ @SuppressWarnings("rawtypes")
+ SpecificDatumWriter writer = getDatumWriter(fieldSchema);
+ data = IOUtils.serialize(writer, fieldValue);
--- End diff --
Type safety: Unchecked invocation serialize(SpecificDatumWriter, Object) of the generic method serialize(SpecificDatumWriter< T >, T) of type IOUtils
---