You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by cdfleischmann <cd...@gmail.com> on 2014/03/11 01:36:23 UTC
AWS + CAMEL + DynamoDB - Please Help
Hello folks, I have tried to use the AWS Camel Component for DynamoDB as
discussed here:http://camel.apache.org/aws-ddb.html... I am following the
snippet of code in the MOCK Services testing area, as listed here:
https://github.com/apache/camel/blob/master/components/camel-aws/src/test/java/org/apache/camel/component/aws/ddb/integration/DdbComponentIntegrationTest.java
I see in the example that you are meant to replace the Access Key and Secret
Key, but otherwise the example should be ok??
However, I received the following error (and this was after I added the
readCapacity and writeCapacity variables:
/*org.apache.camel.ResolveEndpointFailedException: Failed to resolve
endpoint:
aws-ddb://table?amazonDDBClient=%23ddbClient&amazonDdbEndpoint=ap-southeast-2&readCapacity=10&writeCapacity=10
due to: Status Code: 400, AWS Service: AmazonDynamoDB, AWS Request ID:
KHA79STK78SHC2BG2R8HLPF7RJVV4KQNSO5AEMVJF66Q9ASUAAJG, AWS Error Code:
ValidationException, AWS Error Message: 2 validation errors detected: Value
null at 'keySchema.hashKeyElement.attributeName' failed to satisfy
constraint: Member must not be null; Value null at
'keySchema.hashKeyElement.attributeType' failed to satisfy constraint:
Member must not be null*/
How do you add the keySchema to the URI? I note that it needs to be of type
KeySchema? I have tried the following:
//**
* Send the array map to the DynamoDB routes.
*
* @param route The route to send it to.
* @param exchange The MSG Exchange.
* @param producer The Producer template to use.
* @param attributeMap The Attribute Map to send off to DyanamoDB.
*/
private void sendMessage(String route, Exchange exchange, final Map<String,
AttributeValue> attributeMap, final Integer id) {
Exchange exchange2 = producer.send(route, new Processor() {
public void process(Exchange exchange) throws Exception {
exchange.getIn().setHeader(DdbConstants.OPERATION,
DdbOperations.PutItem);
exchange.getIn().setHeader(DdbConstants.RETURN_VALUES,
"ALL_OLD");
exchange.getIn().setHeader(DdbConstants.ITEM, attributeMap);
exchange.getIn().setHeader(DdbConstants.KEY, id);
ArrayList<KeySchemaElement> tableKeySchema = new
ArrayList<KeySchemaElement>();
tableKeySchema.add(new
KeySchemaElement().withAttributeName("id").withKeyType(KeyType.HASH));
exchange.getIn().setHeader(DdbConstants.KEY_SCHEMA,
tableKeySchema.get(0).);
}
});
System.out.println("LIST: " +
exchange2.getIn().getHeader(DdbConstants.ITEMS, List.class));
}/
But it doesn't work... Any ideas or thoughts to progress?
--
View this message in context: http://camel.465427.n5.nabble.com/AWS-CAMEL-DynamoDB-Please-Help-tp5748581.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: AWS + CAMEL + DynamoDB - Please Help
Posted by Willem Jiang <wi...@gmail.com>.
OK, I think I just found out the key by checking the url.
You may need to update ddbClient’s endpoint address before using the camel-aws-ddb.
As amazonDdbEndpoint option didn’t work when you using amazonDDBlClient option, I just fixed this issue[1] few days ago.
[1]https://issues.apache.org/jira/browse/CAMEL-7286
--
Willem Jiang
Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On March 13, 2014 at 5:08:14 PM, cdfleischmann (cdfleischmann@gmail.com) wrote:
> What do you mean? The route I was trying to use was:
> /*"aws-ddb://SourceItems?amazonDDBClient=#ddbClient&amazonDdbEndpoint=ap-southeast-2"
> + "&writeCapacity=10&readCapacity=10"*/, newSourceItem(sourceItem)
>
> You will see the table name straight after "aws-ddb://" called "SourceItems"
> ???
>
> I am not sure I follow...
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/AWS-CAMEL-DynamoDB-Please-Help-tp5748581p5748739.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: AWS + CAMEL + DynamoDB - Please Help
Posted by cdfleischmann <cd...@gmail.com>.
What do you mean? The route I was trying to use was:
/*"aws-ddb://SourceItems?amazonDDBClient=#ddbClient&amazonDdbEndpoint=ap-southeast-2"
+ "&writeCapacity=10&readCapacity=10"*/, newSourceItem(sourceItem)
You will see the table name straight after "aws-ddb://" called "SourceItems"
???
I am not sure I follow...
--
View this message in context: http://camel.465427.n5.nabble.com/AWS-CAMEL-DynamoDB-Please-Help-tp5748581p5748739.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: AWS + CAMEL + DynamoDB - Please Help
Posted by Willem Jiang <wi...@gmail.com>.
Hi,
I just found something interesting, you add the tableName into the PutReqeuest, but you don’t specify tableName on the url.
--
Willem Jiang
Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On March 11, 2014 at 3:34:20 PM, cdfleischmann (cdfleischmann@gmail.com) wrote:
> Hi...So the solution I created (which seems to work), is as
> follows:Imports:/*import
> com.amazonaws.auth.ClasspathPropertiesFileCredentialsProvider;import
> com.amazonaws.regions.Region;import com.amazonaws.regions.Regions;import
> com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient;import
> com.amazonaws.services.dynamodbv2.model.AttributeValue;import
> com.amazonaws.services.dynamodbv2.model.PutItemRequest;import
> com.amazonaws.services.dynamodbv2.model.PutItemResult;*/I used a mapping
> function for the attributes I want to add to DynamoDB like so;/*private
> static Map newSourceItem(Item sourceItem) {
> Map item = new HashMap();
> item.put("xxx", new AttributeValue(sourceItem.getXXX()));
> item.put("yyy", new AttributeValue(sourceItem.getYYY()));
> item.put("zzz", new
> AttributeValue().withN(String.valueOf(sourceItem.getZZZ())));
> item.put("id", new
> AttributeValue().withN(String.valueOf(sourceItem.getId())));
> return item; }*/In my case, the id is the hash number I used. I did not
> add a Hash Range.Now the code to initialise the dbClient follows:/*/* * This
> credentials provider implementation loads your AWS credentials * from a
> properties file at the root of your classpath. */ dynamoDB = new
> AmazonDynamoDBClient(new ClasspathPropertiesFileCredentialsProvider());
> Region apSouthEast2 = Region.getRegion(Regions.AP_SOUTHEAST_2);
> dynamoDB.setRegion(apSouthEast2);*/And lastly the Put method to add it to
> DynamoDB:/*PutItemRequest putItemRequest = new PutItemRequest(tableName,
> attributeMap); PutItemResult putItemResult =
> dynamoDB.putItem(putItemRequest); System.out.println("Result: " +
> putItemResult);*/Whereas my last Camel attempt was as follows:/*/** * Send
> the array map to the DynamoDB routes. * * @param route The route to send
> it to. * @param exchange The MSG Exchange. * @param attributeMap The
> Attribute Map to send off to DyanamoDB. */ private void sendMessage(String
> route, Exchange exchange, final Map attributeMap)
> { System.out.println("Add item to DynamoDB"); Exchange
> exchange2 = producer.send(route, new Processor() { public void
> process(Exchange exchange) throws Exception {
> exchange.getIn().setHeader(DdbConstants.OPERATION, DdbOperations.PutItem);
> exchange.getIn().setHeader(DdbConstants.RETURN_VALUES, "ALL_OLD");
> exchange.getIn().setHeader(DdbConstants.ITEM, attributeMap); }
> }); }*/with the "route" string
> being:/*sendMessage("aws-ddb://SourceItems?amazonDDBClient=#ddbClient&amazonDdbEndpoint=ap-southeast-2"
> + "&writeCapacity=10&readCapacity=10", newSourceItem(sourceItem));*/I am no
> Camel expert, but it seems the exchange.getIn().setHeader(...) isn't working
> as planned?
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/AWS-CAMEL-DynamoDB-Please-Help-tp5748581p5748605.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
Re: AWS + CAMEL + DynamoDB - Please Help
Posted by cdfleischmann <cd...@gmail.com>.
Hi...So the solution I created (which seems to work), is as
follows:Imports:/*import
com.amazonaws.auth.ClasspathPropertiesFileCredentialsProvider;import
com.amazonaws.regions.Region;import com.amazonaws.regions.Regions;import
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient;import
com.amazonaws.services.dynamodbv2.model.AttributeValue;import
com.amazonaws.services.dynamodbv2.model.PutItemRequest;import
com.amazonaws.services.dynamodbv2.model.PutItemResult;*/I used a mapping
function for the attributes I want to add to DynamoDB like so;/*private
static Map<String, AttributeValue> newSourceItem(Item sourceItem) {
Map<String, AttributeValue> item = new HashMap<String, AttributeValue>();
item.put("xxx", new AttributeValue(sourceItem.getXXX()));
item.put("yyy", new AttributeValue(sourceItem.getYYY()));
item.put("zzz", new
AttributeValue().withN(String.valueOf(sourceItem.getZZZ())));
item.put("id", new
AttributeValue().withN(String.valueOf(sourceItem.getId())));
return item; }*/In my case, the id is the hash number I used. I did not
add a Hash Range.Now the code to initialise the dbClient follows:/*/* * This
credentials provider implementation loads your AWS credentials * from a
properties file at the root of your classpath. */ dynamoDB = new
AmazonDynamoDBClient(new ClasspathPropertiesFileCredentialsProvider());
Region apSouthEast2 = Region.getRegion(Regions.AP_SOUTHEAST_2);
dynamoDB.setRegion(apSouthEast2);*/And lastly the Put method to add it to
DynamoDB:/*PutItemRequest putItemRequest = new PutItemRequest(tableName,
attributeMap); PutItemResult putItemResult =
dynamoDB.putItem(putItemRequest); System.out.println("Result: " +
putItemResult);*/Whereas my last Camel attempt was as follows:/*/** * Send
the array map to the DynamoDB routes. * * @param route The route to send
it to. * @param exchange The MSG Exchange. * @param attributeMap The
Attribute Map to send off to DyanamoDB. */ private void sendMessage(String
route, Exchange exchange, final Map<String, AttributeValue> attributeMap)
{ System.out.println("Add item to DynamoDB"); Exchange
exchange2 = producer.send(route, new Processor() { public void
process(Exchange exchange) throws Exception {
exchange.getIn().setHeader(DdbConstants.OPERATION, DdbOperations.PutItem);
exchange.getIn().setHeader(DdbConstants.RETURN_VALUES, "ALL_OLD");
exchange.getIn().setHeader(DdbConstants.ITEM, attributeMap); }
}); }*/with the "route" string
being:/*sendMessage("aws-ddb://SourceItems?amazonDDBClient=#ddbClient&amazonDdbEndpoint=ap-southeast-2"
+ "&writeCapacity=10&readCapacity=10", newSourceItem(sourceItem));*/I am no
Camel expert, but it seems the exchange.getIn().setHeader(...) isn't working
as planned?
--
View this message in context: http://camel.465427.n5.nabble.com/AWS-CAMEL-DynamoDB-Please-Help-tp5748581p5748605.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: AWS + CAMEL + DynamoDB - Please Help
Posted by Willem Jiang <wi...@gmail.com>.
Hi Chris,
Please share us the solution that you find with the AWS DynamoDB API.
I will check the current camel-aws code, to see if I can apply the patch for it.
--
Willem Jiang
Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On March 11, 2014 at 1:41:21 PM, cdfleischmann (cdfleischmann@gmail.com) wrote:
> Thanks... Yeah, I will move away from the ProducerTemplate of Camel over to
> the direct API calls from the DynamoDB API.
>
> Thanks for your help, was just trying to leverage Camel (framework) where
> possible, not move to too much code where I could.
>
> thanks muchly,
>
> Chris
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/AWS-CAMEL-DynamoDB-Please-Help-tp5748581p5748597.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: AWS + CAMEL + DynamoDB - Please Help
Posted by cdfleischmann <cd...@gmail.com>.
Thanks... Yeah, I will move away from the ProducerTemplate of Camel over to
the direct API calls from the DynamoDB API.
Thanks for your help, was just trying to leverage Camel (framework) where
possible, not move to too much code where I could.
thanks muchly,
Chris
--
View this message in context: http://camel.465427.n5.nabble.com/AWS-CAMEL-DynamoDB-Please-Help-tp5748581p5748597.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: AWS + CAMEL + DynamoDB - Please Help
Posted by Willem Jiang <wi...@gmail.com>.
Hi,
Can you use AmazonDynamoDBClient to send the request ?
I just have a quick look at the developer document[1], I didn’t find a place which need the KeySchema.
[1]http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/LowLevelJavaItemCRUD.html
--
Willem Jiang
Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On March 11, 2014 at 11:59:42 AM, cdfleischmann (cdfleischmann@gmail.com) wrote:
> I have tried the following...:
>
> /*exchange.getIn().setHeader(DdbConstants.KEY_SCHEMA,
> new
> KeySchemaElement().withAttributeName("id").withKeyType(KeyType.HASH));*/
>
> Which also doesn't work. I get this error when attempting to execute the
> send.... It seems to ignore the schema header.
>
> */Caused by: com.amazonaws.AmazonServiceException: Status Code: 400, AWS
> Service: AmazonDynamoDB, AWS Request ID:
> NRK75RLT68K7HSQ915THMTVDTBVV4KQNSO5AEMVJF66Q9ASUAAJG, AWS Error Code:
> ValidationException, AWS Error Message: 2 validation errors detected: Value
> null at 'keySchema.hashKeyElement.attributeName' failed to satisfy
> constraint: Member must not be null; Value null at
> 'keySchema.hashKeyElement.attributeType' failed to satisfy constraint:
> Member must not be null/*
> at
> com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:753)
> at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:414)
> at com.amazonaws.http.AmazonHttpClient.execute0(AmazonHttpClient.java:266)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:236)
> at
> com.amazonaws.services.dynamodb.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:998)
> at
> com.amazonaws.services.dynamodb.AmazonDynamoDBClient.createTable(AmazonDynamoDBClient.java:620)
> at
> org.apache.camel.component.aws.ddb.DdbEndpoint.createTable(DdbEndpoint.java:111)
> at
> org.apache.camel.component.aws.ddb.DdbEndpoint.doStart(DdbEndpoint.java:93)
> at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61)
> at
> org.apache.camel.impl.DefaultCamelContext.startService(DefaultCamelContext.java:1902)
> at
> org.apache.camel.impl.DefaultCamelContext.doAddService(DefaultCamelContext.java:985)
> at
> org.apache.camel.impl.DefaultCamelContext.addService(DefaultCamelContext.java:946)
> at
> org.apache.camel.impl.DefaultCamelContext.getEndpoint(DefaultCamelContext.java:526)
> ... 62 more
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/AWS-CAMEL-DynamoDB-Please-Help-tp5748581p5748592.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: AWS + CAMEL + DynamoDB - Please Help
Posted by cdfleischmann <cd...@gmail.com>.
I have tried the following...:
/*exchange.getIn().setHeader(DdbConstants.KEY_SCHEMA,
new
KeySchemaElement().withAttributeName("id").withKeyType(KeyType.HASH));*/
Which also doesn't work. I get this error when attempting to execute the
send.... It seems to ignore the schema header.
*/Caused by: com.amazonaws.AmazonServiceException: Status Code: 400, AWS
Service: AmazonDynamoDB, AWS Request ID:
NRK75RLT68K7HSQ915THMTVDTBVV4KQNSO5AEMVJF66Q9ASUAAJG, AWS Error Code:
ValidationException, AWS Error Message: 2 validation errors detected: Value
null at 'keySchema.hashKeyElement.attributeName' failed to satisfy
constraint: Member must not be null; Value null at
'keySchema.hashKeyElement.attributeType' failed to satisfy constraint:
Member must not be null/*
at
com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:753)
at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:414)
at com.amazonaws.http.AmazonHttpClient.execute0(AmazonHttpClient.java:266)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:236)
at
com.amazonaws.services.dynamodb.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:998)
at
com.amazonaws.services.dynamodb.AmazonDynamoDBClient.createTable(AmazonDynamoDBClient.java:620)
at
org.apache.camel.component.aws.ddb.DdbEndpoint.createTable(DdbEndpoint.java:111)
at
org.apache.camel.component.aws.ddb.DdbEndpoint.doStart(DdbEndpoint.java:93)
at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61)
at
org.apache.camel.impl.DefaultCamelContext.startService(DefaultCamelContext.java:1902)
at
org.apache.camel.impl.DefaultCamelContext.doAddService(DefaultCamelContext.java:985)
at
org.apache.camel.impl.DefaultCamelContext.addService(DefaultCamelContext.java:946)
at
org.apache.camel.impl.DefaultCamelContext.getEndpoint(DefaultCamelContext.java:526)
... 62 more
--
View this message in context: http://camel.465427.n5.nabble.com/AWS-CAMEL-DynamoDB-Please-Help-tp5748581p5748592.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: AWS + CAMEL + DynamoDB - Please Help
Posted by Willem Jiang <wi...@gmail.com>.
No, you cannot add the the keySchema into the uri. Current camel-aws-ddb just takes the KeySchema from the message header.
I just checked the code of PutItemCommand, it put any KeySchema into request item. The interesting part is aws SDK has a dynamodbv2 package. It looks like AWS has the plan to move to new dynamo db API. I’m not sure if it relates to your issue.
Please let us know if you found out the old dynamodb API is retired.
--
Willem Jiang
Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On March 11, 2014 at 8:36:51 AM, cdfleischmann (cdfleischmann@gmail.com) wrote:
> Hello folks, I have tried to use the AWS Camel Component for DynamoDB as
> discussed here:http://camel.apache.org/aws-ddb.html... I am following the
> snippet of code in the MOCK Services testing area, as listed here:
> https://github.com/apache/camel/blob/master/components/camel-aws/src/test/java/org/apache/camel/component/aws/ddb/integration/DdbComponentIntegrationTest.java
>
> I see in the example that you are meant to replace the Access Key and Secret
> Key, but otherwise the example should be ok??
>
> However, I received the following error (and this was after I added the
> readCapacity and writeCapacity variables:
>
> /*org.apache.camel.ResolveEndpointFailedException: Failed to resolve
> endpoint:
> aws-ddb://table?amazonDDBClient=%23ddbClient&amazonDdbEndpoint=ap-southeast-2&readCapacity=10&writeCapacity=10
> due to: Status Code: 400, AWS Service: AmazonDynamoDB, AWS Request ID:
> KHA79STK78SHC2BG2R8HLPF7RJVV4KQNSO5AEMVJF66Q9ASUAAJG, AWS Error Code:
> ValidationException, AWS Error Message: 2 validation errors detected: Value
> null at 'keySchema.hashKeyElement.attributeName' failed to satisfy
> constraint: Member must not be null; Value null at
> 'keySchema.hashKeyElement.attributeType' failed to satisfy constraint:
> Member must not be null*/
>
> How do you add the keySchema to the URI? I note that it needs to be of type
> KeySchema? I have tried the following:
>
> //**
> * Send the array map to the DynamoDB routes.
> *
> * @param route The route to send it to.
> * @param exchange The MSG Exchange.
> * @param producer The Producer template to use.
> * @param attributeMap The Attribute Map to send off to DyanamoDB.
> */
> private void sendMessage(String route, Exchange exchange, final Map> AttributeValue> attributeMap, final Integer id) {
> Exchange exchange2 = producer.send(route, new Processor() {
> public void process(Exchange exchange) throws Exception {
> exchange.getIn().setHeader(DdbConstants.OPERATION,
> DdbOperations.PutItem);
> exchange.getIn().setHeader(DdbConstants.RETURN_VALUES,
> "ALL_OLD");
> exchange.getIn().setHeader(DdbConstants.ITEM, attributeMap);
> exchange.getIn().setHeader(DdbConstants.KEY, id);
>
> ArrayList tableKeySchema = new
> ArrayList();
> tableKeySchema.add(new
> KeySchemaElement().withAttributeName("id").withKeyType(KeyType.HASH));
>
> exchange.getIn().setHeader(DdbConstants.KEY_SCHEMA,
> tableKeySchema.get(0).);
> }
> });
>
> System.out.println("LIST: " +
> exchange2.getIn().getHeader(DdbConstants.ITEMS, List.class));
> }/
>
> But it doesn't work... Any ideas or thoughts to progress?
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/AWS-CAMEL-DynamoDB-Please-Help-tp5748581.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>