You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metamodel.apache.org by kaspersorensen <gi...@git.apache.org> on 2017/01/24 05:30:58 UTC
[GitHub] metamodel pull request #140: Dynamo DB support
GitHub user kaspersorensen opened a pull request:
https://github.com/apache/metamodel/pull/140
Dynamo DB support
This PR adds Amazon DynamoDB support to MetaModel. A few notes on the implementation:
* I'm personally fairly happy with the read/discovery/query part of things. As much as possible is leveraged from Amazon's SDK, but since DynamoDB is essentially schemaless, there's also the usual option of supplying `SimpleTableDef` objects.
* Creating tables is a bit funky. There are options in creating a DynamoDB table that I haven't seen elsewhere, such as the throughput capacity flags. I added system properties for those. There is also in the SDK options for creating secondary indices, but I didn't find a good way to map that to MetaModel during CREATE TABLE. So I left it out.
* The testing is pretty sparse right now, and you need a set of Amazon credentials to make what is there work. Not sure if there is a mock service available somehow, or an embedded version of DynamoDB.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kaspersorensen/metamodel dynamo-db
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metamodel/pull/140.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #140
----
commit 4bd981dfaf0cdd4731d3878701249ac2faf73d92
Author: Kasper S�rensen <i....@gmail.com>
Date: 2017-01-20T02:59:20Z
Added DynamoDB module project structure
commit 432faefd652f2b73975763863de8bcc16e0e9585
Author: Kasper S�rensen <i....@gmail.com>
Date: 2017-01-20T03:49:24Z
Created the initial basic DataContext implementation
commit 774ec721beb759ae2b992bc6153fc358e81f3cf3
Author: Kasper S�rensen <i....@gmail.com>
Date: 2017-01-21T18:47:31Z
Enhanced DynamoDbDataContext with support for count queries
commit 79ad973a09886c237d7392dc1ed0fe30574e7369
Author: Kasper S�rensen <i....@gmail.com>
Date: 2017-01-22T18:24:31Z
Added integration test
commit 2bbc2af5a1cd3060c49ec78cffd7f5e0e17d20a3
Author: Kasper S�rensen <i....@gmail.com>
Date: 2017-01-22T18:55:04Z
Improved integration test by having a check on table creation status
commit 16805c6b085282560d790e47eb36b897b70406a4
Author: Kasper S�rensen <i....@gmail.com>
Date: 2017-01-24T04:39:34Z
Greatly improved discovery and support for custom attributes
commit b5e68d17ca2bbb9031f25c0a1abb4fcf1ba8207d
Author: Kasper S�rensen <i....@gmail.com>
Date: 2017-01-24T05:23:54Z
Added update callback for DynamoDB
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel issue #140: Dynamo DB support
Posted by LosD <gi...@git.apache.org>.
Github user LosD commented on the issue:
https://github.com/apache/metamodel/pull/140
Hmmm... Maybe it's all line-ending changes, GitHub just only shows the change for .txt files?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request #140: Dynamo DB support
Posted by kaspersorensen <gi...@git.apache.org>.
Github user kaspersorensen commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/140#discussion_r97700078
--- Diff: dynamodb/src/main/java/org/apache/metamodel/dynamodb/DynamoDbDataContext.java ---
@@ -0,0 +1,320 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.dynamodb;
+
+import java.io.Closeable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.metamodel.MetaModelException;
+import org.apache.metamodel.QueryPostprocessDataContext;
+import org.apache.metamodel.UpdateScript;
+import org.apache.metamodel.UpdateableDataContext;
+import org.apache.metamodel.data.DataSet;
+import org.apache.metamodel.data.DefaultRow;
+import org.apache.metamodel.data.Row;
+import org.apache.metamodel.data.SimpleDataSetHeader;
+import org.apache.metamodel.query.FilterItem;
+import org.apache.metamodel.query.SelectItem;
+import org.apache.metamodel.schema.Column;
+import org.apache.metamodel.schema.ColumnType;
+import org.apache.metamodel.schema.MutableColumn;
+import org.apache.metamodel.schema.MutableSchema;
+import org.apache.metamodel.schema.MutableTable;
+import org.apache.metamodel.schema.Schema;
+import org.apache.metamodel.schema.Table;
+import org.apache.metamodel.util.SimpleTableDef;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.AttributeValue;
+import com.amazonaws.services.dynamodbv2.model.DescribeTableResult;
+import com.amazonaws.services.dynamodbv2.model.GetItemRequest;
+import com.amazonaws.services.dynamodbv2.model.GetItemResult;
+import com.amazonaws.services.dynamodbv2.model.GlobalSecondaryIndexDescription;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.ListTablesResult;
+import com.amazonaws.services.dynamodbv2.model.LocalSecondaryIndexDescription;
+import com.amazonaws.services.dynamodbv2.model.ScanRequest;
+import com.amazonaws.services.dynamodbv2.model.ScanResult;
+import com.amazonaws.services.dynamodbv2.model.TableDescription;
+
+/**
+ * DataContext implementation for Amazon DynamoDB.
+ */
+public class DynamoDbDataContext extends QueryPostprocessDataContext implements UpdateableDataContext, Closeable {
+
+ /**
+ * System property key used for getting the read throughput capacity when
+ * creating new tables. Defaults to 5.
+ */
+ public static final String SYSTEM_PROPERTY_THROUGHPUT_READ_CAPACITY = "metamodel.dynamodb.throughput.capacity.read";
+
+ /**
+ * System property key used for getting the write throughput capacity when
+ * creating new tables. Defaults to 5.
+ */
+ public static final String SYSTEM_PROPERTY_THROUGHPUT_WRITE_CAPACITY = "metamodel.dynamodb.throughput.capacity.write";
+
+ private static final Logger logger = LoggerFactory.getLogger(DynamoDbDataContext.class);
+
+ /**
+ * The artificial schema name used by this DataContext.
+ */
+ public static final String SCHEMA_NAME = "public";
+
+ private final AmazonDynamoDB _dynamoDb;
+ private final boolean _shutdownOnClose;
+ private final SimpleTableDef[] _tableDefs;
+
+ public DynamoDbDataContext() {
+ this(AmazonDynamoDBClientBuilder.defaultClient(), null, true);
+ }
+
+ public DynamoDbDataContext(SimpleTableDef[] tableDefs) {
+ this(AmazonDynamoDBClientBuilder.defaultClient(), tableDefs, true);
+ }
+
+ public DynamoDbDataContext(AmazonDynamoDB client) {
+ this(client, null, false);
+ }
+
+ public DynamoDbDataContext(AmazonDynamoDB client, SimpleTableDef[] tableDefs) {
+ this(client, tableDefs, false);
+ }
+
+ private DynamoDbDataContext(AmazonDynamoDB client, SimpleTableDef[] tableDefs, boolean shutdownOnClose) {
+ _dynamoDb = client;
+ _tableDefs = (tableDefs == null ? new SimpleTableDef[0] : tableDefs);
+ _shutdownOnClose = shutdownOnClose;
+ }
+
+ public AmazonDynamoDB getDynamoDb() {
+ return _dynamoDb;
+ }
+
+ @Override
+ public void close() {
+ if (_shutdownOnClose) {
+ _dynamoDb.shutdown();
+ }
+ }
+
+ @Override
+ protected Schema getMainSchema() throws MetaModelException {
+ final Map<String, SimpleTableDef> tableDefs = new HashMap<>();
+ for (SimpleTableDef tableDef : _tableDefs) {
+ tableDefs.put(tableDef.getName(), tableDef);
+ }
+
+ final MutableSchema schema = new MutableSchema(getMainSchemaName());
+ final ListTablesResult tables = _dynamoDb.listTables();
+ final List<String> tableNames = tables.getTableNames();
+ for (String tableName : tableNames) {
+ final MutableTable table = new MutableTable(tableName, schema);
+ schema.addTable(table);
+
+ final DescribeTableResult descripeTableResult = _dynamoDb.describeTable(tableName);
+ final TableDescription tableDescription = descripeTableResult.getTable();
+
+ // add primary keys
+ addColumnFromKeySchema("Primary index", tableDescription.getKeySchema(), table, true);
+
+ // add attributes from global and local indices
+ final List<GlobalSecondaryIndexDescription> globalSecondaryIndexes = tableDescription
+ .getGlobalSecondaryIndexes();
+ if (globalSecondaryIndexes != null) {
+ for (GlobalSecondaryIndexDescription globalSecondaryIndex : globalSecondaryIndexes) {
+ addColumnFromKeySchema(globalSecondaryIndex.getIndexName(), globalSecondaryIndex.getKeySchema(),
--- End diff --
Unfortunately the GlobalSecondaryIndexDescription and LocalSecondaryIndexDescription classes don't share a common super-class, so I'm not sure if it can be made more general.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request #140: Dynamo DB support
Posted by LosD <gi...@git.apache.org>.
Github user LosD commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/140#discussion_r97731934
--- Diff: dynamodb/src/main/java/org/apache/metamodel/dynamodb/DynamoDbTableCreationBuilder.java ---
@@ -0,0 +1,111 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.dynamodb;
+
+import java.util.ArrayList;
+import java.util.Collection;
+
+import org.apache.metamodel.MetaModelException;
+import org.apache.metamodel.create.AbstractTableCreationBuilder;
+import org.apache.metamodel.schema.Column;
+import org.apache.metamodel.schema.MutableTable;
+import org.apache.metamodel.schema.Schema;
+import org.apache.metamodel.schema.Table;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.CreateTableResult;
+import com.amazonaws.services.dynamodbv2.model.GlobalSecondaryIndex;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.TableStatus;
+
+class DynamoDbTableCreationBuilder extends AbstractTableCreationBuilder<DynamoDbUpdateCallback> {
+
+ private static final Logger logger = LoggerFactory.getLogger(DynamoDbTableCreationBuilder.class);
+
+ public DynamoDbTableCreationBuilder(DynamoDbUpdateCallback updateCallback, Schema schema, String name) {
+ super(updateCallback, schema, name);
+ }
+
+ @Override
+ public Table execute() throws MetaModelException {
+ final MutableTable table = getTable();
+ final String tableName = table.getName();
+
+ final Collection<AttributeDefinition> attributes = new ArrayList<>();
+ final Collection<KeySchemaElement> keySchema = new ArrayList<>();
+ final Collection<GlobalSecondaryIndex> globalSecondaryIndices = new ArrayList<>();
+
+ final long readCapacity = Long.parseLong(System.getProperty(
+ DynamoDbDataContext.SYSTEM_PROPERTY_THROUGHPUT_READ_CAPACITY, "5"));
+ final long writeCapacity = Long.parseLong(System.getProperty(
+ DynamoDbDataContext.SYSTEM_PROPERTY_THROUGHPUT_WRITE_CAPACITY, "5"));
+ final ProvisionedThroughput provisionedThroughput = new ProvisionedThroughput(readCapacity, writeCapacity);
+
+ final Column[] columns = table.getColumns();
+ for (Column column : columns) {
+ if (column.isPrimaryKey()) {
+ final KeyType keyType = getKeyType(column.getRemarks());
+ keySchema.add(new KeySchemaElement(column.getName(), keyType));
+ attributes.add(new AttributeDefinition(column.getName(), DynamoDbUtils.toAttributeType(column
+ .getType())));
+ }
+ }
+
+ final CreateTableRequest createTableRequest = new CreateTableRequest();
+ createTableRequest.setTableName(tableName);
+ createTableRequest.setAttributeDefinitions(attributes);
+ createTableRequest.setGlobalSecondaryIndexes(globalSecondaryIndices);
+ createTableRequest.setKeySchema(keySchema);
+ createTableRequest.setProvisionedThroughput(provisionedThroughput);
+
+ final AmazonDynamoDB client = getUpdateCallback().getDataContext().getDynamoDb();
+
+ final CreateTableResult createTableResult = client.createTable(createTableRequest);
+
+ // await the table creation to be "done".
+ {
+ String tableStatus = createTableResult.getTableDescription().getTableStatus();
+ while (TableStatus.CREATING.name().equals(tableStatus)) {
+ logger.debug("Waiting for table status to be ACTIVE. Currently: {}", tableStatus);
+ try {
+ Thread.sleep(300);
+ } catch (InterruptedException e) {
+ }
--- End diff --
Whoops, just saw this: This seems to the wrong way to handle InterruptedException? Someone is trying to stop the thread, so you should exit the operation, not ignore and press on.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request #140: Dynamo DB support
Posted by LosD <gi...@git.apache.org>.
Github user LosD commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/140#discussion_r97941969
--- Diff: dynamodb/src/main/java/org/apache/metamodel/dynamodb/DynamoDbTableCreationBuilder.java ---
@@ -0,0 +1,111 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.dynamodb;
+
+import java.util.ArrayList;
+import java.util.Collection;
+
+import org.apache.metamodel.MetaModelException;
+import org.apache.metamodel.create.AbstractTableCreationBuilder;
+import org.apache.metamodel.schema.Column;
+import org.apache.metamodel.schema.MutableTable;
+import org.apache.metamodel.schema.Schema;
+import org.apache.metamodel.schema.Table;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.CreateTableResult;
+import com.amazonaws.services.dynamodbv2.model.GlobalSecondaryIndex;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.TableStatus;
+
+class DynamoDbTableCreationBuilder extends AbstractTableCreationBuilder<DynamoDbUpdateCallback> {
+
+ private static final Logger logger = LoggerFactory.getLogger(DynamoDbTableCreationBuilder.class);
+
+ public DynamoDbTableCreationBuilder(DynamoDbUpdateCallback updateCallback, Schema schema, String name) {
+ super(updateCallback, schema, name);
+ }
+
+ @Override
+ public Table execute() throws MetaModelException {
+ final MutableTable table = getTable();
+ final String tableName = table.getName();
+
+ final Collection<AttributeDefinition> attributes = new ArrayList<>();
+ final Collection<KeySchemaElement> keySchema = new ArrayList<>();
+ final Collection<GlobalSecondaryIndex> globalSecondaryIndices = new ArrayList<>();
+
+ final long readCapacity = Long.parseLong(System.getProperty(
+ DynamoDbDataContext.SYSTEM_PROPERTY_THROUGHPUT_READ_CAPACITY, "5"));
+ final long writeCapacity = Long.parseLong(System.getProperty(
+ DynamoDbDataContext.SYSTEM_PROPERTY_THROUGHPUT_WRITE_CAPACITY, "5"));
+ final ProvisionedThroughput provisionedThroughput = new ProvisionedThroughput(readCapacity, writeCapacity);
+
+ final Column[] columns = table.getColumns();
+ for (Column column : columns) {
+ if (column.isPrimaryKey()) {
+ final KeyType keyType = getKeyType(column.getRemarks());
+ keySchema.add(new KeySchemaElement(column.getName(), keyType));
+ attributes.add(new AttributeDefinition(column.getName(), DynamoDbUtils.toAttributeType(column
+ .getType())));
+ }
+ }
+
+ final CreateTableRequest createTableRequest = new CreateTableRequest();
+ createTableRequest.setTableName(tableName);
+ createTableRequest.setAttributeDefinitions(attributes);
+ createTableRequest.setGlobalSecondaryIndexes(globalSecondaryIndices);
+ createTableRequest.setKeySchema(keySchema);
+ createTableRequest.setProvisionedThroughput(provisionedThroughput);
+
+ final AmazonDynamoDB client = getUpdateCallback().getDataContext().getDynamoDb();
+
+ final CreateTableResult createTableResult = client.createTable(createTableRequest);
+
+ // await the table creation to be "done".
+ {
+ String tableStatus = createTableResult.getTableDescription().getTableStatus();
+ while (TableStatus.CREATING.name().equals(tableStatus)) {
+ logger.debug("Waiting for table status to be ACTIVE. Currently: {}", tableStatus);
+ try {
+ Thread.sleep(300);
+ } catch (InterruptedException e) {
+ }
--- End diff --
The best thing you can do if you can't stop executing, e.g. because of atomicity concerns is something like:
```java
String tableStatus = createTableResult.getTableDescription().getTableStatus();
boolean interrupted = false
while (TableStatus.CREATING.name().equals(tableStatus)) {
logger.debug("Waiting for table status to be ACTIVE. Currently: {}", tableStatus);
try {
Thread.sleep(300);
} catch (InterruptedException e) {
interrupted = true;
}
tableStatus = client.describeTable(tableName).getTable().getTableStatus();
}
if (interrupted) {
Thread.currentThread().interrupt();
}
```
(Approach taken from https://www.ibm.com/developerworks/library/j-jtp05236/#N10186)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel issue #140: Dynamo DB support
Posted by kaspersorensen <gi...@git.apache.org>.
Github user kaspersorensen commented on the issue:
https://github.com/apache/metamodel/pull/140
Argh I can't get it to not do it. But if you add `?w=1` to the diff URL then it shows the patch without whitespace changes.
I'd like to just add the PR based on a diff download and then not worry about this since the "merging" is going to happen with a separate/squased commit to master anyway.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel issue #140: Dynamo DB support
Posted by kaspersorensen <gi...@git.apache.org>.
Github user kaspersorensen commented on the issue:
https://github.com/apache/metamodel/pull/140
Addressed your remarks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request #140: Dynamo DB support
Posted by kaspersorensen <gi...@git.apache.org>.
Github user kaspersorensen commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/140#discussion_r98075190
--- Diff: dynamodb/src/main/java/org/apache/metamodel/dynamodb/DynamoDbTableCreationBuilder.java ---
@@ -0,0 +1,111 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.dynamodb;
+
+import java.util.ArrayList;
+import java.util.Collection;
+
+import org.apache.metamodel.MetaModelException;
+import org.apache.metamodel.create.AbstractTableCreationBuilder;
+import org.apache.metamodel.schema.Column;
+import org.apache.metamodel.schema.MutableTable;
+import org.apache.metamodel.schema.Schema;
+import org.apache.metamodel.schema.Table;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.CreateTableResult;
+import com.amazonaws.services.dynamodbv2.model.GlobalSecondaryIndex;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.TableStatus;
+
+class DynamoDbTableCreationBuilder extends AbstractTableCreationBuilder<DynamoDbUpdateCallback> {
+
+ private static final Logger logger = LoggerFactory.getLogger(DynamoDbTableCreationBuilder.class);
+
+ public DynamoDbTableCreationBuilder(DynamoDbUpdateCallback updateCallback, Schema schema, String name) {
+ super(updateCallback, schema, name);
+ }
+
+ @Override
+ public Table execute() throws MetaModelException {
+ final MutableTable table = getTable();
+ final String tableName = table.getName();
+
+ final Collection<AttributeDefinition> attributes = new ArrayList<>();
+ final Collection<KeySchemaElement> keySchema = new ArrayList<>();
+ final Collection<GlobalSecondaryIndex> globalSecondaryIndices = new ArrayList<>();
+
+ final long readCapacity = Long.parseLong(System.getProperty(
+ DynamoDbDataContext.SYSTEM_PROPERTY_THROUGHPUT_READ_CAPACITY, "5"));
+ final long writeCapacity = Long.parseLong(System.getProperty(
+ DynamoDbDataContext.SYSTEM_PROPERTY_THROUGHPUT_WRITE_CAPACITY, "5"));
+ final ProvisionedThroughput provisionedThroughput = new ProvisionedThroughput(readCapacity, writeCapacity);
+
+ final Column[] columns = table.getColumns();
+ for (Column column : columns) {
+ if (column.isPrimaryKey()) {
+ final KeyType keyType = getKeyType(column.getRemarks());
+ keySchema.add(new KeySchemaElement(column.getName(), keyType));
+ attributes.add(new AttributeDefinition(column.getName(), DynamoDbUtils.toAttributeType(column
+ .getType())));
+ }
+ }
+
+ final CreateTableRequest createTableRequest = new CreateTableRequest();
+ createTableRequest.setTableName(tableName);
+ createTableRequest.setAttributeDefinitions(attributes);
+ createTableRequest.setGlobalSecondaryIndexes(globalSecondaryIndices);
+ createTableRequest.setKeySchema(keySchema);
+ createTableRequest.setProvisionedThroughput(provisionedThroughput);
+
+ final AmazonDynamoDB client = getUpdateCallback().getDataContext().getDynamoDb();
+
+ final CreateTableResult createTableResult = client.createTable(createTableRequest);
+
+ // await the table creation to be "done".
+ {
+ String tableStatus = createTableResult.getTableDescription().getTableStatus();
+ while (TableStatus.CREATING.name().equals(tableStatus)) {
+ logger.debug("Waiting for table status to be ACTIVE. Currently: {}", tableStatus);
+ try {
+ Thread.sleep(300);
+ } catch (InterruptedException e) {
+ }
--- End diff --
Got it ... So I implemented it a bit like that, except I did it at the UpdateCallback level, allowing for the complete update script to finish (not just the "create table" part of it).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel issue #140: Dynamo DB support
Posted by kaspersorensen <gi...@git.apache.org>.
Github user kaspersorensen commented on the issue:
https://github.com/apache/metamodel/pull/140
Hmm that's pretty odd. The .gitattributes file should force it to keep the same line endings.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel issue #140: Dynamo DB support
Posted by LosD <gi...@git.apache.org>.
Github user LosD commented on the issue:
https://github.com/apache/metamodel/pull/140
Yeah let's go ahead. LGTM!
Unfortunately you'll have to do the merge yourself, or wait until Monday. My home computer and the Apache Git repo has some weird hate for each other.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request #140: Dynamo DB support
Posted by kaspersorensen <gi...@git.apache.org>.
Github user kaspersorensen commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/140#discussion_r97700714
--- Diff: dynamodb/src/main/java/org/apache/metamodel/dynamodb/DynamoDbDataContext.java ---
@@ -0,0 +1,320 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.dynamodb;
+
+import java.io.Closeable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.metamodel.MetaModelException;
+import org.apache.metamodel.QueryPostprocessDataContext;
+import org.apache.metamodel.UpdateScript;
+import org.apache.metamodel.UpdateableDataContext;
+import org.apache.metamodel.data.DataSet;
+import org.apache.metamodel.data.DefaultRow;
+import org.apache.metamodel.data.Row;
+import org.apache.metamodel.data.SimpleDataSetHeader;
+import org.apache.metamodel.query.FilterItem;
+import org.apache.metamodel.query.SelectItem;
+import org.apache.metamodel.schema.Column;
+import org.apache.metamodel.schema.ColumnType;
+import org.apache.metamodel.schema.MutableColumn;
+import org.apache.metamodel.schema.MutableSchema;
+import org.apache.metamodel.schema.MutableTable;
+import org.apache.metamodel.schema.Schema;
+import org.apache.metamodel.schema.Table;
+import org.apache.metamodel.util.SimpleTableDef;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.AttributeValue;
+import com.amazonaws.services.dynamodbv2.model.DescribeTableResult;
+import com.amazonaws.services.dynamodbv2.model.GetItemRequest;
+import com.amazonaws.services.dynamodbv2.model.GetItemResult;
+import com.amazonaws.services.dynamodbv2.model.GlobalSecondaryIndexDescription;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.ListTablesResult;
+import com.amazonaws.services.dynamodbv2.model.LocalSecondaryIndexDescription;
+import com.amazonaws.services.dynamodbv2.model.ScanRequest;
+import com.amazonaws.services.dynamodbv2.model.ScanResult;
+import com.amazonaws.services.dynamodbv2.model.TableDescription;
+
+/**
+ * DataContext implementation for Amazon DynamoDB.
+ */
+public class DynamoDbDataContext extends QueryPostprocessDataContext implements UpdateableDataContext, Closeable {
+
+ /**
+ * System property key used for getting the read throughput capacity when
+ * creating new tables. Defaults to 5.
+ */
+ public static final String SYSTEM_PROPERTY_THROUGHPUT_READ_CAPACITY = "metamodel.dynamodb.throughput.capacity.read";
+
+ /**
+ * System property key used for getting the write throughput capacity when
+ * creating new tables. Defaults to 5.
+ */
+ public static final String SYSTEM_PROPERTY_THROUGHPUT_WRITE_CAPACITY = "metamodel.dynamodb.throughput.capacity.write";
+
+ private static final Logger logger = LoggerFactory.getLogger(DynamoDbDataContext.class);
+
+ /**
+ * The artificial schema name used by this DataContext.
+ */
+ public static final String SCHEMA_NAME = "public";
+
+ private final AmazonDynamoDB _dynamoDb;
+ private final boolean _shutdownOnClose;
+ private final SimpleTableDef[] _tableDefs;
+
+ public DynamoDbDataContext() {
+ this(AmazonDynamoDBClientBuilder.defaultClient(), null, true);
+ }
+
+ public DynamoDbDataContext(SimpleTableDef[] tableDefs) {
+ this(AmazonDynamoDBClientBuilder.defaultClient(), tableDefs, true);
+ }
+
+ public DynamoDbDataContext(AmazonDynamoDB client) {
+ this(client, null, false);
+ }
+
+ public DynamoDbDataContext(AmazonDynamoDB client, SimpleTableDef[] tableDefs) {
+ this(client, tableDefs, false);
+ }
+
+ private DynamoDbDataContext(AmazonDynamoDB client, SimpleTableDef[] tableDefs, boolean shutdownOnClose) {
--- End diff --
I guess I agree ... This is actually there to prevent that we will close the client inside the class. It seems fairly similar to what other MM connectors do though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request #140: Dynamo DB support
Posted by kaspersorensen <gi...@git.apache.org>.
Github user kaspersorensen commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/140#discussion_r97906184
--- Diff: dynamodb/src/main/java/org/apache/metamodel/dynamodb/DynamoDbTableCreationBuilder.java ---
@@ -0,0 +1,111 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.dynamodb;
+
+import java.util.ArrayList;
+import java.util.Collection;
+
+import org.apache.metamodel.MetaModelException;
+import org.apache.metamodel.create.AbstractTableCreationBuilder;
+import org.apache.metamodel.schema.Column;
+import org.apache.metamodel.schema.MutableTable;
+import org.apache.metamodel.schema.Schema;
+import org.apache.metamodel.schema.Table;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.CreateTableResult;
+import com.amazonaws.services.dynamodbv2.model.GlobalSecondaryIndex;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.TableStatus;
+
+class DynamoDbTableCreationBuilder extends AbstractTableCreationBuilder<DynamoDbUpdateCallback> {
+
+ private static final Logger logger = LoggerFactory.getLogger(DynamoDbTableCreationBuilder.class);
+
+ public DynamoDbTableCreationBuilder(DynamoDbUpdateCallback updateCallback, Schema schema, String name) {
+ super(updateCallback, schema, name);
+ }
+
+ @Override
+ public Table execute() throws MetaModelException {
+ final MutableTable table = getTable();
+ final String tableName = table.getName();
+
+ final Collection<AttributeDefinition> attributes = new ArrayList<>();
+ final Collection<KeySchemaElement> keySchema = new ArrayList<>();
+ final Collection<GlobalSecondaryIndex> globalSecondaryIndices = new ArrayList<>();
+
+ final long readCapacity = Long.parseLong(System.getProperty(
+ DynamoDbDataContext.SYSTEM_PROPERTY_THROUGHPUT_READ_CAPACITY, "5"));
+ final long writeCapacity = Long.parseLong(System.getProperty(
+ DynamoDbDataContext.SYSTEM_PROPERTY_THROUGHPUT_WRITE_CAPACITY, "5"));
+ final ProvisionedThroughput provisionedThroughput = new ProvisionedThroughput(readCapacity, writeCapacity);
+
+ final Column[] columns = table.getColumns();
+ for (Column column : columns) {
+ if (column.isPrimaryKey()) {
+ final KeyType keyType = getKeyType(column.getRemarks());
+ keySchema.add(new KeySchemaElement(column.getName(), keyType));
+ attributes.add(new AttributeDefinition(column.getName(), DynamoDbUtils.toAttributeType(column
+ .getType())));
+ }
+ }
+
+ final CreateTableRequest createTableRequest = new CreateTableRequest();
+ createTableRequest.setTableName(tableName);
+ createTableRequest.setAttributeDefinitions(attributes);
+ createTableRequest.setGlobalSecondaryIndexes(globalSecondaryIndices);
+ createTableRequest.setKeySchema(keySchema);
+ createTableRequest.setProvisionedThroughput(provisionedThroughput);
+
+ final AmazonDynamoDB client = getUpdateCallback().getDataContext().getDynamoDb();
+
+ final CreateTableResult createTableResult = client.createTable(createTableRequest);
+
+ // await the table creation to be "done".
+ {
+ String tableStatus = createTableResult.getTableDescription().getTableStatus();
+ while (TableStatus.CREATING.name().equals(tableStatus)) {
+ logger.debug("Waiting for table status to be ACTIVE. Currently: {}", tableStatus);
+ try {
+ Thread.sleep(300);
+ } catch (InterruptedException e) {
+ }
--- End diff --
But it would be someone "interrupting our sleep", not interrupting the thread in general, would it? Basically the table creation on Amazon takes some time - I've seen it taking from 1 second to 10 seconds. But since you can't do anything with the table in that timespan, I wanted a sleeping checker to wait before we return. An interruption of that ... Hmm I can't even see what that would mean :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel issue #140: Dynamo DB support
Posted by LosD <gi...@git.apache.org>.
Github user LosD commented on the issue:
https://github.com/apache/metamodel/pull/140
Code LGTM now, but for some files it seems like Git suddenly considers them changed (since commit e76cd23), even though I can't see any change, not even white space changes. Not entirely sure what's going on.
For some of the text files, line endings has also suddenly changed.
I guess it doesn't matter for mergeability, but it would sure be nice to find out what causes weird stuff like that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request #140: Dynamo DB support
Posted by LosD <gi...@git.apache.org>.
Github user LosD commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/140#discussion_r97908427
--- Diff: dynamodb/src/main/java/org/apache/metamodel/dynamodb/DynamoDbTableCreationBuilder.java ---
@@ -0,0 +1,111 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.dynamodb;
+
+import java.util.ArrayList;
+import java.util.Collection;
+
+import org.apache.metamodel.MetaModelException;
+import org.apache.metamodel.create.AbstractTableCreationBuilder;
+import org.apache.metamodel.schema.Column;
+import org.apache.metamodel.schema.MutableTable;
+import org.apache.metamodel.schema.Schema;
+import org.apache.metamodel.schema.Table;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.CreateTableResult;
+import com.amazonaws.services.dynamodbv2.model.GlobalSecondaryIndex;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.TableStatus;
+
+class DynamoDbTableCreationBuilder extends AbstractTableCreationBuilder<DynamoDbUpdateCallback> {
+
+ private static final Logger logger = LoggerFactory.getLogger(DynamoDbTableCreationBuilder.class);
+
+ public DynamoDbTableCreationBuilder(DynamoDbUpdateCallback updateCallback, Schema schema, String name) {
+ super(updateCallback, schema, name);
+ }
+
+ @Override
+ public Table execute() throws MetaModelException {
+ final MutableTable table = getTable();
+ final String tableName = table.getName();
+
+ final Collection<AttributeDefinition> attributes = new ArrayList<>();
+ final Collection<KeySchemaElement> keySchema = new ArrayList<>();
+ final Collection<GlobalSecondaryIndex> globalSecondaryIndices = new ArrayList<>();
+
+ final long readCapacity = Long.parseLong(System.getProperty(
+ DynamoDbDataContext.SYSTEM_PROPERTY_THROUGHPUT_READ_CAPACITY, "5"));
+ final long writeCapacity = Long.parseLong(System.getProperty(
+ DynamoDbDataContext.SYSTEM_PROPERTY_THROUGHPUT_WRITE_CAPACITY, "5"));
+ final ProvisionedThroughput provisionedThroughput = new ProvisionedThroughput(readCapacity, writeCapacity);
+
+ final Column[] columns = table.getColumns();
+ for (Column column : columns) {
+ if (column.isPrimaryKey()) {
+ final KeyType keyType = getKeyType(column.getRemarks());
+ keySchema.add(new KeySchemaElement(column.getName(), keyType));
+ attributes.add(new AttributeDefinition(column.getName(), DynamoDbUtils.toAttributeType(column
+ .getType())));
+ }
+ }
+
+ final CreateTableRequest createTableRequest = new CreateTableRequest();
+ createTableRequest.setTableName(tableName);
+ createTableRequest.setAttributeDefinitions(attributes);
+ createTableRequest.setGlobalSecondaryIndexes(globalSecondaryIndices);
+ createTableRequest.setKeySchema(keySchema);
+ createTableRequest.setProvisionedThroughput(provisionedThroughput);
+
+ final AmazonDynamoDB client = getUpdateCallback().getDataContext().getDynamoDb();
+
+ final CreateTableResult createTableResult = client.createTable(createTableRequest);
+
+ // await the table creation to be "done".
+ {
+ String tableStatus = createTableResult.getTableDescription().getTableStatus();
+ while (TableStatus.CREATING.name().equals(tableStatus)) {
+ logger.debug("Waiting for table status to be ACTIVE. Currently: {}", tableStatus);
+ try {
+ Thread.sleep(300);
+ } catch (InterruptedException e) {
+ }
--- End diff --
No, it means "stop what you are doing". http://www.nurkiewicz.com/2014/05/interruptedexception-and-interrupting.html?m=1
http://daniel.mitterdorfer.name/articles/2015/handling-interruptedexception/
When you see Java programs stalling when you press cancel (hello, Eclipse)? That's often because someone mistreated the InterruptException.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request #140: Dynamo DB support
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/metamodel/pull/140
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---